Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

How to scrape table data from a website using Python: BeautifulSoup, pandas and a proxy

A Python script downloads a Wikipedia page through a residential proxy and reads its population table two ways: BeautifulSoup row by row and pandas.read_html in one call; the card shows the result of the real run, 241 rows by 6 columns, 3 tables found on the page and a CSV file of 16,057 bytes.
A Python script downloads a Wikipedia page through a residential proxy and reads its population table two ways: BeautifulSoup row by row and pandas.read_html in one call; the card shows the result of the real run, 241 rows by 6 columns, 3 tables found on the page and a CSV file of 16,057 bytes.

To scrape a table from a website with Python, download the page with requests, then read the table: BeautifulSoup walks the rows of a <table> one by one, and pandas.read_html turns every table on the page into a DataFrame in one call. Save it with to_csv. Below is the code we ran and its real output.

We ran it on 20 November 2026 on a Windows 11 PC, Python 3.12, against the Wikipedia list of countries by population, once through a QuanticData residential proxy and once through a mobile proxy. Exit IPs are masked; USER and PASS stand for your credentials.

What you need

PackageVersion we ranWhat it does here
requests2.34.2Downloads the page, through the proxy
beautifulsoup44.14.3Finds the table and reads each cell
lxml6.1.1The HTML parser behind both
pandas3.0.3Reads whole tables into DataFrames and writes the CSV

Install them with pip install requests beautifulsoup4 lxml pandas. A proxy is optional for one page; it matters when you need the page as it is served in a given country.

Step 1: download the page

Send the request through the proxy, with a User-Agent that names your client and a contact URL, as Wikimedia's User-Agent policy asks. The first call checks where the request leaves from; the second downloads the page.

import os
from io import StringIO

import pandas as pd
import requests
from bs4 import BeautifulSoup

URL = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
proxy = os.environ["PROXY"]  # http://USER:PASS@HOST:PORT
proxies = {"http": proxy, "https": proxy}
headers = {"User-Agent": "ExampleResearchBot/1.0 (+https://example.com/bot)"}

ip = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=30).json()
print("exit:", "x.x.x.x", ip.get("city"), ip.get("region"), ip.get("country"), ip.get("org", "").split(" ", 1)[-1])

r = requests.get(URL, proxies=proxies, headers=headers, timeout=60)
print("status:", r.status_code, "bytes:", len(r.content))
exit: x.x.x.x Wilmington North Carolina US AT&T Enterprises, LLC
status: 200 bytes: 1390526

The proxy line is the one our generator gives, here a sticky US line of Residential Basic: http://USER-type-residential-country-us-session-t01-lifetime-30:PASS@residentialboson.quanticdata.io:10000. Keeping it in an environment variable keeps the password out of the script. For the full set of options, see a proxy in Python Requests.

Step 2: read the table with BeautifulSoup

Find the table with a CSS selector, then take the text of every header and data cell, row by row. On Wikipedia, data tables carry the class wikitable; on other sites, open the page's developer tools and look at the <table> element for an id or class to select.

soup = BeautifulSoup(r.text, "lxml")
table = soup.select_one("table.wikitable")
rows = []
for tr in table.select("tr"):
    cells = [c.get_text(" ", strip=True) for c in tr.find_all(["th", "td"])]
    if cells:
        rows.append(cells)
print("BeautifulSoup:", len(rows), "rows; header:", rows[0])
print("  first data row:", rows[1])
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
  first data row: ['World', '8,232,000,000', '100%', '13 Jun 2025', 'UN projection [ 1 ] [ 3 ]', '']

241 rows: the header and 240 data rows, 6 columns each. Two details worth knowing:

  • Read both th and td. Header cells are th; many tables also use th for the first cell of each row. Reading only td drops them.
  • Clean the footnotes. The cell text keeps reference marks such as [ 1 ]. Strip them with a regular expression, or drop the sup elements before reading the text.

Step 3: read every table with pandas.read_html

When the table is a plain HTML table, pandas does the walking for you. read_html returns a list with one DataFrame per <table> on the page:

tables = pd.read_html(StringIO(r.text))
print("pandas.read_html:", len(tables), "tables on the page")
df = tables[0]
print("  first table shape:", df.shape)
print(df.head(3).to_string())

df.to_csv("population.csv", index=False)
print("saved population.csv,", os.path.getsize("population.csv"), "bytes")
pandas.read_html: 3 tables on the page
  first table shape: (241, 6)
  Location    Population % of world         Date Source (official or from the United Nations) Notes
0    World  8.232000e+09       100%  13 Jun 2025                          UN projection[1][3]   NaN
1    India  1.429404e+09      17.3%   1 Jul 2026                       Official projection[4]   [b]
2    China  1.404890e+09      17.0%  31 Dec 2025                         Official estimate[5]   [c]
saved population.csv, 16057 bytes

pandas took the first row as column names, so its 241 rows are data rows, one more than BeautifulSoup found. The extra one is an empty <tr> in the page: the BeautifulSoup loop skips it (if cells), pandas keeps it as a row of NaN. df = df.dropna(how="all") removes it and leaves the same 240 rows. Note what else pandas did on its own: it read "8,232,000,000" as a number (shown as 8.232000e+09), and turned empty cells into NaN. Pass match="Population" to keep only tables that contain that text, which is safer than an index when a page has several tables, and wrap the HTML in StringIO as above: recent pandas versions expect a file-like object, not a raw string.

BeautifulSoup or pandas: which to use

BeautifulSouppandas.read_html
Code for one tableAbout 7 lines1 line
OutputLists of strings, exactly as shownA DataFrame with typed columns
Links and attributes in cellsYes, read a["href"] or any attributeText only by default
Irregular tables (merged cells, nested tables)You decide how to handle themBest effort
In our run241 rows: the header and 240 data rows, 6 columns3 tables; the first (241, 6), 240 rows after dropna

Start with read_html; switch to BeautifulSoup when you need the links in the cells, a specific attribute, or control over odd rows. Both read the HTML you downloaded, so both need the table to be in it.

When the table is not in the HTML

Two cases stop both methods:

  • The page builds the table with JavaScript. Look at the page source (Ctrl+U, not the developer tools): if the numbers are not in it, the table is filled in by a script. A headless browser renders it first; see Playwright for scraping.
  • The site answers with a block page. A 403 or a challenge page has no table in it, and read_html stops with "No tables found". Our guide to a 403 when scraping covers the causes.

The same script through a mobile proxy

We ran the script again with only the PROXY variable changed, to a sticky US line of our mobile proxies (mobile.quanticdata.io:8080). The request left from a carrier IP and the table came back the same:

exit: x.x.x.x Charlotte North Carolina US Verizon Business
status: 200 bytes: 1390526
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
pandas.read_html: 3 tables on the page
  first table shape: (241, 6)
saved population.csv, 16057 bytes

Our residential proxies give you a home connection in the country you pick; mobile lines give you a carrier IP, Verizon in this run. The parsing code does not change.

Getting the table from an API instead

If you would rather not run the download and the parser yourself, our web scraping API returns the page as Markdown, with HTML tables kept as tables. One call from Python:

r = requests.post(
    "https://api.quanticdata.io/v1/scraper/extract",
    headers={"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"},
    json={"url": "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population",
          "format": "markdown", "country": "us"},
    timeout=120,
)
data = r.json()
p = data.get("payload", data)
md = p.get("content") or p.get("markdown") or ""
print("status:", r.status_code, "| engine:", p.get("engine"), "| tables:", (p.get("quality") or {}).get("tables"))
lines = [l for l in md.splitlines() if l.startswith("|")]
print("markdown table lines:", len(lines))
print("\n".join(lines[:3]))

The output, with the link inside the header shortened to (...):

status: 200 | engine: tls | tables: 1
markdown table lines: 242
| Location | Population | % of  <br>world | Date | Source (official or from  <br>the [United Nations](...)) | Notes |
| --- | --- | --- | --- | --- | --- |
| **World** | 8,232,000,000 | 100% | 13 Jun 2025 | UN projection[\[1\]](#cite_note-UNFPA-1)[\[3\]](#cite_note-auto1-4) |     |

242 lines: the header, the separator and the 240 data rows of the same table, ready for a spreadsheet, a database or a language model. The call fetched the page over plain HTTP (engine tls). For more on the basics, see web scraping with Python.

Before you scrape a table

Check the site's terms and robots.txt, keep the request rate modest, and prefer an official download or API when the site offers one: many statistics tables are also published as CSV. Wikipedia's text is under a Creative Commons licence that asks for attribution when you reuse it.

Sources & further reading

FAQ

Quick answers on how to scrape table data from website using python.

Something else? Ask us

How do I scrape a table from a website using Python?

Download the page with requests.get, then either walk the <table> with BeautifulSoup, reading the th and td cells of each tr, or call pandas.read_html to get every table as a DataFrame. Save it with df.to_csv. In our run both read the same 240 data rows and 6 columns from a Wikipedia table.

How do I scrape a table with BeautifulSoup?

Select the table, for example soup.select_one("table.wikitable"), loop over table.select("tr"), and for each row take c.get_text(" ", strip=True) for every cell found with tr.find_all(["th", "td"]). The first row holds the headers.

Why does pandas.read_html say no tables found?

The HTML you passed has no <table> element. Either the page builds the table with JavaScript, so it is not in the downloaded source, or the site returned a block or consent page instead of the content. Check r.status_code and search r.text for a value you expect in the table.

How do I save a scraped table to CSV in Python?

With pandas: df.to_csv("population.csv", index=False). With BeautifulSoup rows, pass the list of lists to csv.writer(...).writerows(rows) or build a DataFrame with pd.DataFrame(rows[1:], columns=rows[0]) first. Our CSV of the 241-row table was 16,057 bytes.

Do I need a proxy to scrape a table?

Not for one page. A proxy helps when you need the page as served in a given country, or a home or carrier IP for your own script. In our test, the same script returned the same table through a residential and a mobile proxy with only the proxy line changed.

Tables from any site, as data

Residential and mobile proxies for your own Python scripts, or one API call that returns the page with its tables as Markdown.

Collectors with Python examples

Related reading