# pandas read_html: Arguments, Errors, Cleaning

> pandas read_html explained on a real table: pick it with match and attrs, pass HTML in StringIO on pandas 3, keep links, fix 403 and "No tables found".

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / pandas read_html: Arguments, Errors, Cleaning

# pandas read_html: match, attrs, StringIO, links and cleaning, tested on a real table

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![pandas read_html reading the Wikipedia GDP table: of 8 tables on the page, match IMF and the wikitable class keep the one with 222 rows and 4 columns, which arrive as text and become float64 after the footnote marks are stripped and the numbers converted; the card lists the arguments used and the change of dtype.](https://quanticdata.io/blog/pandas-read-html/hero.webp?v=20261120h)

On this page

- [The basic call, and the 403](https://quanticdata.io/blog/pandas-read-html/#the-basic-call-and-the-403)
- [Pick the table: match and attrs](https://quanticdata.io/blog/pandas-read-html/#pick-the-table-match-and-attrs)
- [pandas 3: wrap HTML strings in StringIO](https://quanticdata.io/blog/pandas-read-html/#pandas-3-wrap-html-strings-in-stringio)
- [read_html through a proxy](https://quanticdata.io/blog/pandas-read-html/#read-html-through-a-proxy)
- [From text to numbers](https://quanticdata.io/blog/pandas-read-html/#from-text-to-numbers)
- [Keep the links](https://quanticdata.io/blog/pandas-read-html/#keep-the-links)
- [Fix "No tables found"](https://quanticdata.io/blog/pandas-read-html/#fix-no-tables-found)
- [Where the HTML comes from](https://quanticdata.io/blog/pandas-read-html/#where-the-html-comes-from)

`pandas.read_html()` reads every `<table>` in an HTML document and returns a list of DataFrames. Give it a URL, a file, or HTML wrapped in `io.StringIO`; keep only the table you want with `match="text"` and `attrs={"class": "..."}`; then clean the columns. It needs lxml, or Beautiful Soup with html5lib, installed.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=pandas-read-html) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

We ran every call below on 20 November 2026 with Python 3.12.10 and pandas 3.0.3, on the Wikipedia list of countries by GDP (nominal). For the whole job of finding a table on a page and saving it, start with our guide [How to scrape a table from a website with Python](https://quanticdata.io/blog/scrape-table-from-website-python/); this post is about read_html itself.

## The basic call, and the 403

```python
import pandas as pd
URL = "https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)"
pd.read_html(URL)
# urllib.error.HTTPError: HTTP Error 403: Forbidden
```

Given a URL, pandas downloads it with Python's urllib and its generic User-Agent. Wikimedia asks automated clients to identify themselves with a User-Agent that names the tool and a way to contact its owner, and refuses generic ones. Send one with `storage_options`:

```
UA = {"User-Agent": "table-reader-example/1.0 (+https://example.com/contact)"}
tables = pd.read_html(URL, storage_options=UA)
print(len(tables))   # 8
```

Put your own tool name and contact in it. That fixed the 403 and returned the 8 tables of the page.

## Pick the table: match and attrs

read_html always returns a list. On this page it held 8 DataFrames; two were article tables with the class `wikitable`, shaped (222, 4) and (14, 3). Two arguments narrow it down:

```python
import io
df = pd.read_html(io.StringIO(html), match="IMF", attrs={"class": "wikitable"})[0]
print(df.shape)              # (222, 4)
print(list(df.columns))
# ['Country/Territory', 'IMF (2026)[1]', 'World Bank (2025)[6]', 'United Nations (2024)[7]']
```

| Argument | What it does |
| --- | --- |
| `match="IMF"` | Keeps tables whose text contains the string or regex |
| `attrs={"class": "wikitable"}` | Keeps tables with these HTML attributes, such as an id |
| `header=0` | Uses that row as column names when the table has no `<th>` |
| `skiprows=1` | Drops rows at the top |
| `thousands=`, `decimal=` | Number format: on a test cell `1.234,5`, `thousands="."` and `decimal=","` gave 1234.5 |
| `extract_links="body"` | Returns cells as (text, link) |
| `flavor="bs4"` | Parses with Beautiful Soup and html5lib instead of lxml |

When nothing matches, read_html raises instead of returning an empty list: `match="zzz-no-such-text"` stopped with `ValueError: No tables found matching pattern 'zzz-no-such-text'`. Catch that error when the page you read changes often, and log the page you got, so a renamed column header does not silently stop a scheduled job.

## pandas 3: wrap HTML strings in StringIO

When you download the page yourself, pass the HTML in a StringIO. On pandas 3.0.3 a plain string failed:

```
pd.read_html(r.text)
# FileNotFoundError: [Errno 2] No such file or directory: <!DOCTYPE html> ...
pd.read_html(io.StringIO(r.text))   # works
```

pandas treats a string as a path or URL. `io.StringIO(r.text)` or `io.BytesIO(r.content)` tells it the content is the document.

## read_html through a proxy

read_html has no proxy argument. To fetch through one, download with requests, with the same identifying User-Agent, and hand the HTML over:

```python
import requests
PROXY = "http://USER-country-us-sid-t02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080"
r = requests.get(URL, headers=UA, proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content))   # 200 683373
html = r.text
tables = pd.read_html(io.StringIO(html))   # 8 tables
```

This also gives you timeouts, retries and cookies, which read_html's own download does not. The requests proxy setup is in [a proxy in Python requests](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/).

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=pandas-read-html) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## From text to numbers

All four columns of the GDP table came back as text (dtype `str` in pandas 3). The cells carry footnote marks, a few say `—N/a`, and some carry a year, such as `98,964 (2024)`. Clean the names, then the values:

```python
import re
df.columns = [re.sub(r"\[.*?\]", "", c).strip() for c in df.columns]
df["Country/Territory"] = df["Country/Territory"].str.replace(r"\[.*?\]", "", regex=True).str.strip()

col = "IMF (2026)"
df[col] = pd.to_numeric(df[col].str.extract(r"^([\d,]+)")[0].str.replace(",", ""),
                        errors="coerce")
print(df.dtypes[col])                       # float64
print(df[df["Country/Territory"] != "World"].nlargest(3, col)["Country/Territory"].tolist())
# ['United States', 'China', 'Germany']
```

Taking the leading number first matters. A plain `pd.to_numeric(..., errors="coerce")` turned 30 cells into NaN, because "98,964 (2024)" is not a number; extracting the digits before converting left 24, the cells that really have no value. Count the NaN after every conversion so you know what you lost.

## Keep the links

```
lk = pd.read_html(io.StringIO(html), match="IMF", attrs={"class": "wikitable"},
                  extract_links="body")[0]
print(lk.iloc[1, 0])
# ('United States', 'https://en.wikipedia.org/wiki/Economy_of_United_States')
```

Each body cell becomes a (text, href) pair, None where the cell has no link. Split them into two columns with `lk[c].str[0]` and `lk[c].str[1]`: for the first country that gave "United States" and the URL of its economy article, ready to follow for the next level of data.

## Fix "No tables found"

```
pd.read_html(io.StringIO("<html><body><div id='app'></div></body></html>"))
# ValueError: No tables found
```

read_html only sees `<table>` elements in the HTML it receives. The usual causes: the table is drawn by JavaScript (render it first, see [scraping dynamic web pages](https://quanticdata.io/blog/dynamic-web-scraping-python/)), the "table" is built from div elements (parse it with [Beautiful Soup web scraping](https://quanticdata.io/blog/beautifulsoup-web-scraping/)), or you downloaded an error or consent page instead of the content. Print the first 500 characters of what you passed in before changing the arguments.

## Where the HTML comes from

We fetched the GDP page through a QuanticData Residential Premium session in the US. [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/) take the country and a session ID in the username and work with requests exactly as above; the [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/) lets you run the code first.

### Sources & further reading

- [pandas.read_html (pandas documentation)](https://pandas.pydata.org/docs/reference/api/pandas.read_html.html)
- [Wikimedia Foundation: User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy)
- [Python docs: io.StringIO](https://docs.python.org/3/library/io.html#io.StringIO)
- [Wikipedia: List of countries by GDP (nominal)](https://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28nominal%29)

## FAQ

Quick answers on pandas read_html.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I read an HTML table into pandas?

Call `pd.read_html()` with a URL, a file, or HTML in `io.StringIO`. It returns a list of DataFrames, one per table; keep yours with `match=` or `attrs=` and index the list. On the GDP page it found 8 tables.

### Why does pandas read_html return 403 Forbidden?

pandas downloads with urllib and its generic User-Agent, which some sites refuse. Wikipedia asks for a User-Agent that names your tool and a contact; passing one with `storage_options={"User-Agent": "..."}` returned the 8 tables in our test.

### Why does read_html fail with FileNotFoundError on an HTML string?

pandas 3 reads a plain string as a path or URL. Wrap the HTML: `pd.read_html(io.StringIO(html))`.

### How do I fix "No tables found" in read_html?

The HTML you passed has no `<table>`: the table is built by JavaScript, made of div elements, or you received an error page. Render the page, parse divs with Beautiful Soup, or check what you downloaded.

### Why are read_html numbers text instead of numbers?

Footnote marks, notes such as a year in brackets, or placeholders like "N/a" make pandas keep the column as text. Strip them, extract the number, then use `pd.to_numeric(..., errors="coerce")` and count the NaN.

## Fetch the page, then let pandas read the table

Residential and mobile proxies for requests, with country and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=pandas-read-html) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Related reading

- [urllib3 vs requests: Speed and Bandwidth](https://quanticdata.io/blog/urllib3-vs-requests/): Guides · requests is built on urllib3 and adds sessions, JSON, exceptions on errors and compression by default. urllib3 alone was faster on a local server, the same through a proxy, and downloaded 9.5 times more bytes until we asked for compression. Measured, with the code.
- [Python requests POST: JSON, Form Data, Files](https://quanticdata.io/blog/python-requests-post/): Guides · requests.post(url, json=payload) sends a JSON body with the right Content-Type; data= sends form fields; files= uploads. We posted each one through a residential proxy to a test server and print what the server received, including the redirect that silently turns a POST into a GET.
- [Concurrency vs Parallelism, Measured in Python](https://quanticdata.io/blog/concurrency-vs-parallelism/): Guides · Concurrency is making progress on several tasks by switching between them; parallelism is running several at the same instant on different CPU cores. We timed both on a real scraping job in Python on 20 November 2026: threads cut 30 downloads from 10.3 s to 3.5 s, and only processes cut the parsing, from 3.5 s to 0.8 s.

---

Source: https://quanticdata.io/blog/pandas-read-html/ · Site index for AI: https://quanticdata.io/llms.txt
