Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

pandas read_html: match, attrs, StringIO, links and cleaning, tested on a real table

pandas read_html reading the Wikipedia GDP table: of 8 tables on the page, match IMF and the wikitable class keep the one with 222 rows and 4 columns, which arrive as text and become float64 after the footnote marks are stripped and the numbers converted; the card lists the arguments used and the change of dtype.
pandas read_html reading the Wikipedia GDP table: of 8 tables on the page, match IMF and the wikitable class keep the one with 222 rows and 4 columns, which arrive as text and become float64 after the footnote marks are stripped and the numbers converted; the card lists the arguments used and the change of dtype.

pandas.read_html() reads every <table> in an HTML document and returns a list of DataFrames. Give it a URL, a file, or HTML wrapped in io.StringIO; keep only the table you want with match="text" and attrs={"class": "..."}; then clean the columns. It needs lxml, or Beautiful Soup with html5lib, installed.

We ran every call below on 20 November 2026 with Python 3.12.10 and pandas 3.0.3, on the Wikipedia list of countries by GDP (nominal). For the whole job of finding a table on a page and saving it, start with our guide How to scrape a table from a website with Python; this post is about read_html itself.

The basic call, and the 403

import pandas as pd
URL = "https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)"
pd.read_html(URL)
# urllib.error.HTTPError: HTTP Error 403: Forbidden

Given a URL, pandas downloads it with Python's urllib and its generic User-Agent. Wikimedia asks automated clients to identify themselves with a User-Agent that names the tool and a way to contact its owner, and refuses generic ones. Send one with storage_options:

UA = {"User-Agent": "table-reader-example/1.0 (+https://example.com/contact)"}
tables = pd.read_html(URL, storage_options=UA)
print(len(tables))   # 8

Put your own tool name and contact in it. That fixed the 403 and returned the 8 tables of the page.

Pick the table: match and attrs

read_html always returns a list. On this page it held 8 DataFrames; two were article tables with the class wikitable, shaped (222, 4) and (14, 3). Two arguments narrow it down:

import io
df = pd.read_html(io.StringIO(html), match="IMF", attrs={"class": "wikitable"})[0]
print(df.shape)              # (222, 4)
print(list(df.columns))
# ['Country/Territory', 'IMF (2026)[1]', 'World Bank (2025)[6]', 'United Nations (2024)[7]']
ArgumentWhat it does
match="IMF"Keeps tables whose text contains the string or regex
attrs={"class": "wikitable"}Keeps tables with these HTML attributes, such as an id
header=0Uses that row as column names when the table has no <th>
skiprows=1Drops rows at the top
thousands=, decimal=Number format: on a test cell 1.234,5, thousands="." and decimal="," gave 1234.5
extract_links="body"Returns cells as (text, link)
flavor="bs4"Parses with Beautiful Soup and html5lib instead of lxml

When nothing matches, read_html raises instead of returning an empty list: match="zzz-no-such-text" stopped with ValueError: No tables found matching pattern 'zzz-no-such-text'. Catch that error when the page you read changes often, and log the page you got, so a renamed column header does not silently stop a scheduled job.

pandas 3: wrap HTML strings in StringIO

When you download the page yourself, pass the HTML in a StringIO. On pandas 3.0.3 a plain string failed:

pd.read_html(r.text)
# FileNotFoundError: [Errno 2] No such file or directory: <!DOCTYPE html> ...
pd.read_html(io.StringIO(r.text))   # works

pandas treats a string as a path or URL. io.StringIO(r.text) or io.BytesIO(r.content) tells it the content is the document.

read_html through a proxy

read_html has no proxy argument. To fetch through one, download with requests, with the same identifying User-Agent, and hand the HTML over:

import requests
PROXY = "http://USER-country-us-sid-t02-ttl-30m:PASS@residentialpremium.quanticdata.io:8080"
r = requests.get(URL, headers=UA, proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content))   # 200 683373
html = r.text
tables = pd.read_html(io.StringIO(html))   # 8 tables

This also gives you timeouts, retries and cookies, which read_html's own download does not. The requests proxy setup is in a proxy in Python requests.

From text to numbers

All four columns of the GDP table came back as text (dtype str in pandas 3). The cells carry footnote marks, a few say —N/a, and some carry a year, such as 98,964 (2024). Clean the names, then the values:

import re
df.columns = [re.sub(r"\[.*?\]", "", c).strip() for c in df.columns]
df["Country/Territory"] = df["Country/Territory"].str.replace(r"\[.*?\]", "", regex=True).str.strip()

col = "IMF (2026)"
df[col] = pd.to_numeric(df[col].str.extract(r"^([\d,]+)")[0].str.replace(",", ""),
                        errors="coerce")
print(df.dtypes[col])                       # float64
print(df[df["Country/Territory"] != "World"].nlargest(3, col)["Country/Territory"].tolist())
# ['United States', 'China', 'Germany']

Taking the leading number first matters. A plain pd.to_numeric(..., errors="coerce") turned 30 cells into NaN, because "98,964 (2024)" is not a number; extracting the digits before converting left 24, the cells that really have no value. Count the NaN after every conversion so you know what you lost.

lk = pd.read_html(io.StringIO(html), match="IMF", attrs={"class": "wikitable"},
                  extract_links="body")[0]
print(lk.iloc[1, 0])
# ('United States', 'https://en.wikipedia.org/wiki/Economy_of_United_States')

Each body cell becomes a (text, href) pair, None where the cell has no link. Split them into two columns with lk[c].str[0] and lk[c].str[1]: for the first country that gave "United States" and the URL of its economy article, ready to follow for the next level of data.

Fix "No tables found"

pd.read_html(io.StringIO("<html><body><div id='app'></div></body></html>"))
# ValueError: No tables found

read_html only sees <table> elements in the HTML it receives. The usual causes: the table is drawn by JavaScript (render it first, see scraping dynamic web pages), the "table" is built from div elements (parse it with Beautiful Soup web scraping), or you downloaded an error or consent page instead of the content. Print the first 500 characters of what you passed in before changing the arguments.

Where the HTML comes from

We fetched the GDP page through a QuanticData Residential Premium session in the US. residential proxies and mobile proxies take the country and a session ID in the username and work with requests exactly as above; the Residential Basic free trial lets you run the code first.

Sources & further reading

FAQ

Quick answers on pandas read_html.

Something else? Ask us

How do I read an HTML table into pandas?

Call pd.read_html() with a URL, a file, or HTML in io.StringIO. It returns a list of DataFrames, one per table; keep yours with match= or attrs= and index the list. On the GDP page it found 8 tables.

Why does pandas read_html return 403 Forbidden?

pandas downloads with urllib and its generic User-Agent, which some sites refuse. Wikipedia asks for a User-Agent that names your tool and a contact; passing one with storage_options={"User-Agent": "..."} returned the 8 tables in our test.

Why does read_html fail with FileNotFoundError on an HTML string?

pandas 3 reads a plain string as a path or URL. Wrap the HTML: pd.read_html(io.StringIO(html)).

How do I fix "No tables found" in read_html?

The HTML you passed has no <table>: the table is built by JavaScript, made of div elements, or you received an error page. Render the page, parse divs with Beautiful Soup, or check what you downloaded.

Why are read_html numbers text instead of numbers?

Footnote marks, notes such as a year in brackets, or placeholders like "N/a" make pandas keep the column as text. Strip them, extract the number, then use pd.to_numeric(..., errors="coerce") and count the NaN.

Fetch the page, then let pandas read the table

Residential and mobile proxies for requests, with country and session settings in the username.

Related reading