To parse JSON in Python, call r.json() on a requests response, json.loads(text) on a string, or json.load(f) on an open file. Each returns Python objects: objects become dict, arrays list, true and false become True and False, null becomes None. Then read values with keys and indexes.
Everything below ran on 20 November 2026 with Python 3.12.10 and requests 2.34.2, against two public JSON endpoints (ipinfo.io and the practice API of quotes.toscrape.com), through a QuanticData mobile proxy in the US.
Parse a JSON response from an API
import requests
PROXY = "http://USER-country-us-type-mobile-sid-j01-ttl-30m:PASS@mobile.quanticdata.io:8080"
px = {"http": PROXY, "https": PROXY}
r = requests.get("https://ipinfo.io/json", proxies=px, timeout=30)
print(r.headers["content-type"]) # application/json
data = r.json()
print(type(data).__name__, list(data))
# dict ['ip', 'city', 'region', 'country', 'loc', 'org', 'postal', 'timezone', 'readme']
print(data["city"], data["region"], data["org"])
# Detroit Michigan AS21928 T-Mobile USA, Inc.
print(data.get("postal_code", "n/a")) # n/a
- r.json() parses the body and returns a dict or list. No need to import json.
- data["key"] raises KeyError when the key is missing; data.get("key", default) returns the default. Use get for optional fields.
- Check the content-type before trusting the body:
application/jsonhere.
Nested JSON: dictionaries inside lists
Real APIs nest. The quotes API returns a page object whose quotes key is a list of dicts, each with an author dict and a tags list:
d = requests.get("https://quotes.toscrape.com/api/quotes", params={"page": 1},
proxies=px, timeout=30).json()
first = d["quotes"][0]
print(first["author"]["name"], first["tags"])
# Albert Einstein ['change', 'deep-thoughts', 'thinking', 'world']
for q in d["quotes"]:
print(q["author"]["name"], len(q["tags"]))
Read the path from the outside in: key, index, key. When you do not know the structure, print json.dumps(d, indent=2)[:1000] and follow the brackets. d.keys() at each level shows what you can ask for next, and type() tells you whether the next step is a key or a list index.
Nested JSON to a table with pandas
import pandas as pd
df = pd.json_normalize(d["quotes"])
print(list(df.columns), df.shape)
# ['tags', 'text', 'author.goodreads_link', 'author.name', 'author.slug'] (10, 5)
json_normalize flattens nested dicts into dotted column names; lists such as tags stay lists in one cell. From there, df.to_csv() gives a spreadsheet. To collect every page of such an API, see paginating a JSON API.
Parse a JSON string and a JSON file
import json
obj = json.loads('{"name": "Café", "price": 51.77, "in_stock": true, "tags": null}')
print(obj["in_stock"], obj["tags"]) # True None
with open("quotes.json", "w", encoding="utf-8") as f:
json.dump(d["quotes"], f, ensure_ascii=False, indent=2)
with open("quotes.json", encoding="utf-8") as f:
back = json.load(f)
print(len(back)) # 10
| Function | From / to |
|---|---|
json.loads(s) | String to Python |
json.load(f) | File to Python |
json.dumps(obj) | Python to string |
json.dump(obj, f) | Python to file |
The s stands for string. By default dumps escapes non-ASCII: json.dumps({"name": "Café"}) printed {"name": "Caf\u00e9"}; with ensure_ascii=False it kept Café. We opened the files with encoding="utf-8", which keeps the text the same on every system.
The errors, and what each one means
| What we did | Error we got | Fix |
|---|---|---|
r.json() on an HTML page | JSONDecodeError: Expecting value: line 1 column 1 (char 0) | Check status and content-type first; you got a page, not data |
json.loads("{'name': 'x'}") | Expecting property name enclosed in double quotes: line 1 column 2 (char 1) | That is a Python repr, not JSON; fix the source or use ast.literal_eval |
json.dumps({"t": date(2026, 11, 20)}) | TypeError: Object of type date is not JSON serializable | default=str gave {"t": "2026-11-20"} |
The first one deserves care when scraping. "Expecting value at char 0" means the body does not start with JSON at all, often because the server sent an error page, a login page or a block page instead of data. Guard the call:
r = requests.get(url, proxies=px, timeout=30)
if r.ok and "json" in r.headers.get("content-type", ""):
data = r.json()
else:
print(r.status_code, r.headers.get("content-type"), r.text[:200])
JSON hidden inside an HTML page
Pages built with JavaScript often carry their data as JSON inside a script tag. quotes.toscrape.com/js/ answered with 5,808 bytes of HTML and no visible quotes, but the script held a JavaScript array. A regular expression cut it out and json.loads read it:
import re
html = requests.get("https://quotes.toscrape.com/js/", proxies=px, timeout=30).text
m = re.search(r"var data = (\[.*?\]);", html, re.S)
quotes = json.loads(m.group(1))
print(len(quotes), sorted(quotes[0])) # 10 ['author', 'tags', 'text']
Framework sites use the same idea with tags such as <script id="__NEXT_DATA__" type="application/json">: select the tag with Beautiful Soup, take its text and pass it to json.loads. The block has to be strict JSON; in our checks, unquoted keys ({a: 1}) and a trailing comma ({"a": 1,}) both raised the same "Expecting property name enclosed in double quotes" error as single quotes.
JSON Lines for scraped data
For results you append to over time, write one JSON object per line. Each line parses alone, so a crash loses at most one record:
with open("quotes.jsonl", "w", encoding="utf-8") as f:
for q in d["quotes"]:
f.write(json.dumps(q, ensure_ascii=False) + "\n")
print(pd.read_json("quotes.jsonl", lines=True).shape) # (10, 3)
Where the JSON comes from
Many sites have JSON behind their pages: an API the page calls, or a JSON block inside the HTML. Finding it beats parsing HTML; see scraping dynamic pages. Our test calls went through a QuanticData mobile line in Detroit; mobile proxies and residential proxies take a country and a session in the username. For JSON from pages that have no API, our web scraping API returns structured data, and the reverse direction, posting JSON, is in sending JSON with requests POST. XML responses: parsing XML in Python.


