Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Parse JSON in Python: API responses, strings, files and nested data, with the real errors

Parsing JSON in Python from an API called through a mobile proxy: r.json() turns the response into a dict with keys such as city, region and org, nested quote data flattens into a table of 10 rows and 5 columns with pandas json_normalize, and the card lists the three errors met in the test, an HTML page, single quotes and a date object.
Parsing JSON in Python from an API called through a mobile proxy: r.json() turns the response into a dict with keys such as city, region and org, nested quote data flattens into a table of 10 rows and 5 columns with pandas json_normalize, and the card lists the three errors met in the test, an HTML page, single quotes and a date object.

To parse JSON in Python, call r.json() on a requests response, json.loads(text) on a string, or json.load(f) on an open file. Each returns Python objects: objects become dict, arrays list, true and false become True and False, null becomes None. Then read values with keys and indexes.

Everything below ran on 20 November 2026 with Python 3.12.10 and requests 2.34.2, against two public JSON endpoints (ipinfo.io and the practice API of quotes.toscrape.com), through a QuanticData mobile proxy in the US.

Parse a JSON response from an API

import requests

PROXY = "http://USER-country-us-type-mobile-sid-j01-ttl-30m:PASS@mobile.quanticdata.io:8080"
px = {"http": PROXY, "https": PROXY}

r = requests.get("https://ipinfo.io/json", proxies=px, timeout=30)
print(r.headers["content-type"])     # application/json
data = r.json()
print(type(data).__name__, list(data))
# dict ['ip', 'city', 'region', 'country', 'loc', 'org', 'postal', 'timezone', 'readme']
print(data["city"], data["region"], data["org"])
# Detroit Michigan AS21928 T-Mobile USA, Inc.
print(data.get("postal_code", "n/a"))  # n/a
  • r.json() parses the body and returns a dict or list. No need to import json.
  • data["key"] raises KeyError when the key is missing; data.get("key", default) returns the default. Use get for optional fields.
  • Check the content-type before trusting the body: application/json here.

Nested JSON: dictionaries inside lists

Real APIs nest. The quotes API returns a page object whose quotes key is a list of dicts, each with an author dict and a tags list:

d = requests.get("https://quotes.toscrape.com/api/quotes", params={"page": 1},
                 proxies=px, timeout=30).json()
first = d["quotes"][0]
print(first["author"]["name"], first["tags"])
# Albert Einstein ['change', 'deep-thoughts', 'thinking', 'world']

for q in d["quotes"]:
    print(q["author"]["name"], len(q["tags"]))

Read the path from the outside in: key, index, key. When you do not know the structure, print json.dumps(d, indent=2)[:1000] and follow the brackets. d.keys() at each level shows what you can ask for next, and type() tells you whether the next step is a key or a list index.

Nested JSON to a table with pandas

import pandas as pd
df = pd.json_normalize(d["quotes"])
print(list(df.columns), df.shape)
# ['tags', 'text', 'author.goodreads_link', 'author.name', 'author.slug'] (10, 5)

json_normalize flattens nested dicts into dotted column names; lists such as tags stay lists in one cell. From there, df.to_csv() gives a spreadsheet. To collect every page of such an API, see paginating a JSON API.

Parse a JSON string and a JSON file

import json
obj = json.loads('{"name": "Café", "price": 51.77, "in_stock": true, "tags": null}')
print(obj["in_stock"], obj["tags"])     # True None

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(d["quotes"], f, ensure_ascii=False, indent=2)
with open("quotes.json", encoding="utf-8") as f:
    back = json.load(f)
print(len(back))                        # 10
FunctionFrom / to
json.loads(s)String to Python
json.load(f)File to Python
json.dumps(obj)Python to string
json.dump(obj, f)Python to file

The s stands for string. By default dumps escapes non-ASCII: json.dumps({"name": "Café"}) printed {"name": "Caf\u00e9"}; with ensure_ascii=False it kept Café. We opened the files with encoding="utf-8", which keeps the text the same on every system.

The errors, and what each one means

What we didError we gotFix
r.json() on an HTML pageJSONDecodeError: Expecting value: line 1 column 1 (char 0)Check status and content-type first; you got a page, not data
json.loads("{'name': 'x'}")Expecting property name enclosed in double quotes: line 1 column 2 (char 1)That is a Python repr, not JSON; fix the source or use ast.literal_eval
json.dumps({"t": date(2026, 11, 20)})TypeError: Object of type date is not JSON serializabledefault=str gave {"t": "2026-11-20"}

The first one deserves care when scraping. "Expecting value at char 0" means the body does not start with JSON at all, often because the server sent an error page, a login page or a block page instead of data. Guard the call:

r = requests.get(url, proxies=px, timeout=30)
if r.ok and "json" in r.headers.get("content-type", ""):
    data = r.json()
else:
    print(r.status_code, r.headers.get("content-type"), r.text[:200])

JSON hidden inside an HTML page

Pages built with JavaScript often carry their data as JSON inside a script tag. quotes.toscrape.com/js/ answered with 5,808 bytes of HTML and no visible quotes, but the script held a JavaScript array. A regular expression cut it out and json.loads read it:

import re
html = requests.get("https://quotes.toscrape.com/js/", proxies=px, timeout=30).text
m = re.search(r"var data = (\[.*?\]);", html, re.S)
quotes = json.loads(m.group(1))
print(len(quotes), sorted(quotes[0]))     # 10 ['author', 'tags', 'text']

Framework sites use the same idea with tags such as <script id="__NEXT_DATA__" type="application/json">: select the tag with Beautiful Soup, take its text and pass it to json.loads. The block has to be strict JSON; in our checks, unquoted keys ({a: 1}) and a trailing comma ({"a": 1,}) both raised the same "Expecting property name enclosed in double quotes" error as single quotes.

JSON Lines for scraped data

For results you append to over time, write one JSON object per line. Each line parses alone, so a crash loses at most one record:

with open("quotes.jsonl", "w", encoding="utf-8") as f:
    for q in d["quotes"]:
        f.write(json.dumps(q, ensure_ascii=False) + "\n")
print(pd.read_json("quotes.jsonl", lines=True).shape)   # (10, 3)

Where the JSON comes from

Many sites have JSON behind their pages: an API the page calls, or a JSON block inside the HTML. Finding it beats parsing HTML; see scraping dynamic pages. Our test calls went through a QuanticData mobile line in Detroit; mobile proxies and residential proxies take a country and a session in the username. For JSON from pages that have no API, our web scraping API returns structured data, and the reverse direction, posting JSON, is in sending JSON with requests POST. XML responses: parsing XML in Python.

Sources & further reading

FAQ

Quick answers on python parse json response.

Something else? Ask us

How do I parse a JSON response in Python?

With requests, call r.json(): it returns a dict or list. Check r.ok and the content-type first, because an HTML error page raises JSONDecodeError: Expecting value: line 1 column 1 (char 0).

What is the difference between json.load and json.loads?

json.loads parses a string; json.load reads an open file. Likewise json.dumps returns a string and json.dump writes to a file.

How do I get a value from nested JSON in Python?

Chain keys and indexes from the outside in: d["quotes"][0]["author"]["name"] returned Albert Einstein in our test. Use .get() at each level for keys that may be missing.

How do I convert nested JSON to a DataFrame?

pd.json_normalize(list_of_dicts) flattens nested dicts into columns such as author.name. Our 10 quotes became a table of 10 rows and 5 columns.

Why does json.loads fail with "Expecting property name enclosed in double quotes"?

The text uses single quotes, so it is not JSON; it is often a printed Python dict. Fix the source, or read it with ast.literal_eval if it really is a Python literal.

Call JSON APIs from a mobile or home IP

Mobile and residential proxies for requests and httpx, with the country and a session ID set in the username.

Collectors with Python examples

Related reading