To scrape a table from a website with Python, download the page with requests, then read the table: BeautifulSoup walks the rows of a <table> one by one, and pandas.read_html turns every table on the page into a DataFrame in one call. Save it with to_csv. Below is the code we ran and its real output.
We ran it on 20 November 2026 on a Windows 11 PC, Python 3.12, against the Wikipedia list of countries by population, once through a QuanticData residential proxy and once through a mobile proxy. Exit IPs are masked; USER and PASS stand for your credentials.
What you need
| Package | Version we ran | What it does here |
|---|---|---|
| requests | 2.34.2 | Downloads the page, through the proxy |
| beautifulsoup4 | 4.14.3 | Finds the table and reads each cell |
| lxml | 6.1.1 | The HTML parser behind both |
| pandas | 3.0.3 | Reads whole tables into DataFrames and writes the CSV |
Install them with pip install requests beautifulsoup4 lxml pandas. A proxy is optional for one page; it matters when you need the page as it is served in a given country.
Step 1: download the page
Send the request through the proxy, with a User-Agent that names your client and a contact URL, as Wikimedia's User-Agent policy asks. The first call checks where the request leaves from; the second downloads the page.
import os
from io import StringIO
import pandas as pd
import requests
from bs4 import BeautifulSoup
URL = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
proxy = os.environ["PROXY"] # http://USER:PASS@HOST:PORT
proxies = {"http": proxy, "https": proxy}
headers = {"User-Agent": "ExampleResearchBot/1.0 (+https://example.com/bot)"}
ip = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=30).json()
print("exit:", "x.x.x.x", ip.get("city"), ip.get("region"), ip.get("country"), ip.get("org", "").split(" ", 1)[-1])
r = requests.get(URL, proxies=proxies, headers=headers, timeout=60)
print("status:", r.status_code, "bytes:", len(r.content))
exit: x.x.x.x Wilmington North Carolina US AT&T Enterprises, LLC
status: 200 bytes: 1390526
The proxy line is the one our generator gives, here a sticky US line of Residential Basic: http://USER-type-residential-country-us-session-t01-lifetime-30:PASS@residentialboson.quanticdata.io:10000. Keeping it in an environment variable keeps the password out of the script. For the full set of options, see a proxy in Python Requests.
Step 2: read the table with BeautifulSoup
Find the table with a CSS selector, then take the text of every header and data cell, row by row. On Wikipedia, data tables carry the class wikitable; on other sites, open the page's developer tools and look at the <table> element for an id or class to select.
soup = BeautifulSoup(r.text, "lxml")
table = soup.select_one("table.wikitable")
rows = []
for tr in table.select("tr"):
cells = [c.get_text(" ", strip=True) for c in tr.find_all(["th", "td"])]
if cells:
rows.append(cells)
print("BeautifulSoup:", len(rows), "rows; header:", rows[0])
print(" first data row:", rows[1])
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
first data row: ['World', '8,232,000,000', '100%', '13 Jun 2025', 'UN projection [ 1 ] [ 3 ]', '']
241 rows: the header and 240 data rows, 6 columns each. Two details worth knowing:
- Read both
thandtd. Header cells areth; many tables also usethfor the first cell of each row. Reading onlytddrops them. - Clean the footnotes. The cell text keeps reference marks such as
[ 1 ]. Strip them with a regular expression, or drop thesupelements before reading the text.
Step 3: read every table with pandas.read_html
When the table is a plain HTML table, pandas does the walking for you. read_html returns a list with one DataFrame per <table> on the page:
tables = pd.read_html(StringIO(r.text))
print("pandas.read_html:", len(tables), "tables on the page")
df = tables[0]
print(" first table shape:", df.shape)
print(df.head(3).to_string())
df.to_csv("population.csv", index=False)
print("saved population.csv,", os.path.getsize("population.csv"), "bytes")
pandas.read_html: 3 tables on the page
first table shape: (241, 6)
Location Population % of world Date Source (official or from the United Nations) Notes
0 World 8.232000e+09 100% 13 Jun 2025 UN projection[1][3] NaN
1 India 1.429404e+09 17.3% 1 Jul 2026 Official projection[4] [b]
2 China 1.404890e+09 17.0% 31 Dec 2025 Official estimate[5] [c]
saved population.csv, 16057 bytes
pandas took the first row as column names, so its 241 rows are data rows, one more than BeautifulSoup found. The extra one is an empty <tr> in the page: the BeautifulSoup loop skips it (if cells), pandas keeps it as a row of NaN. df = df.dropna(how="all") removes it and leaves the same 240 rows. Note what else pandas did on its own: it read "8,232,000,000" as a number (shown as 8.232000e+09), and turned empty cells into NaN. Pass match="Population" to keep only tables that contain that text, which is safer than an index when a page has several tables, and wrap the HTML in StringIO as above: recent pandas versions expect a file-like object, not a raw string.
BeautifulSoup or pandas: which to use
| BeautifulSoup | pandas.read_html | |
|---|---|---|
| Code for one table | About 7 lines | 1 line |
| Output | Lists of strings, exactly as shown | A DataFrame with typed columns |
| Links and attributes in cells | Yes, read a["href"] or any attribute | Text only by default |
| Irregular tables (merged cells, nested tables) | You decide how to handle them | Best effort |
| In our run | 241 rows: the header and 240 data rows, 6 columns | 3 tables; the first (241, 6), 240 rows after dropna |
Start with read_html; switch to BeautifulSoup when you need the links in the cells, a specific attribute, or control over odd rows. Both read the HTML you downloaded, so both need the table to be in it.
When the table is not in the HTML
Two cases stop both methods:
- The page builds the table with JavaScript. Look at the page source (Ctrl+U, not the developer tools): if the numbers are not in it, the table is filled in by a script. A headless browser renders it first; see Playwright for scraping.
- The site answers with a block page. A 403 or a challenge page has no table in it, and
read_htmlstops with "No tables found". Our guide to a 403 when scraping covers the causes.
The same script through a mobile proxy
We ran the script again with only the PROXY variable changed, to a sticky US line of our mobile proxies (mobile.quanticdata.io:8080). The request left from a carrier IP and the table came back the same:
exit: x.x.x.x Charlotte North Carolina US Verizon Business
status: 200 bytes: 1390526
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
pandas.read_html: 3 tables on the page
first table shape: (241, 6)
saved population.csv, 16057 bytes
Our residential proxies give you a home connection in the country you pick; mobile lines give you a carrier IP, Verizon in this run. The parsing code does not change.
Getting the table from an API instead
If you would rather not run the download and the parser yourself, our web scraping API returns the page as Markdown, with HTML tables kept as tables. One call from Python:
r = requests.post(
"https://api.quanticdata.io/v1/scraper/extract",
headers={"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"},
json={"url": "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population",
"format": "markdown", "country": "us"},
timeout=120,
)
data = r.json()
p = data.get("payload", data)
md = p.get("content") or p.get("markdown") or ""
print("status:", r.status_code, "| engine:", p.get("engine"), "| tables:", (p.get("quality") or {}).get("tables"))
lines = [l for l in md.splitlines() if l.startswith("|")]
print("markdown table lines:", len(lines))
print("\n".join(lines[:3]))
The output, with the link inside the header shortened to (...):
status: 200 | engine: tls | tables: 1
markdown table lines: 242
| Location | Population | % of <br>world | Date | Source (official or from <br>the [United Nations](...)) | Notes |
| --- | --- | --- | --- | --- | --- |
| **World** | 8,232,000,000 | 100% | 13 Jun 2025 | UN projection[\[1\]](#cite_note-UNFPA-1)[\[3\]](#cite_note-auto1-4) | |
242 lines: the header, the separator and the 240 data rows of the same table, ready for a spreadsheet, a database or a language model. The call fetched the page over plain HTTP (engine tls). For more on the basics, see web scraping with Python.
Before you scrape a table
Check the site's terms and robots.txt, keep the request rate modest, and prefer an official download or API when the site offers one: many statistics tables are also published as CSV. Wikipedia's text is under a Creative Commons licence that asks for attribution when you reuse it.


