# How to Scrape a Table From a Website in Python

> Scrape an HTML table with Python: BeautifulSoup row by row, pandas.read_html in one call, CSV out, through a proxy. Real output: 241 rows, 6 columns, 3 tables.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / How to Scrape a Table From a Website in Python

# How to scrape table data from a website using Python: BeautifulSoup, pandas and a proxy

Web scraping · Nov 20, 2026 · 7 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![A Python script downloads a Wikipedia page through a residential proxy and reads its population table two ways: BeautifulSoup row by row and pandas.read_html in one call; the card shows the result of the real run, 241 rows by 6 columns, 3 tables found on the page and a CSV file of 16,057 bytes.](https://quanticdata.io/blog/scrape-table-from-website-python/hero.webp?v=20261120h)

On this page

- [What you need](https://quanticdata.io/blog/scrape-table-from-website-python/#what-you-need)
- [Step 1: download the page](https://quanticdata.io/blog/scrape-table-from-website-python/#step-1-download-the-page)
- [Step 2: read the table with BeautifulSoup](https://quanticdata.io/blog/scrape-table-from-website-python/#step-2-read-the-table-with-beautifulsoup)
- [Step 3: read every table with pandas.read_html](https://quanticdata.io/blog/scrape-table-from-website-python/#step-3-read-every-table-with-pandas-read-html)
- [BeautifulSoup or pandas: which to use](https://quanticdata.io/blog/scrape-table-from-website-python/#beautifulsoup-or-pandas-which-to-use)
- [When the table is not in the HTML](https://quanticdata.io/blog/scrape-table-from-website-python/#when-the-table-is-not-in-the-html)
- [The same script through a mobile proxy](https://quanticdata.io/blog/scrape-table-from-website-python/#the-same-script-through-a-mobile-proxy)
- [Getting the table from an API instead](https://quanticdata.io/blog/scrape-table-from-website-python/#getting-the-table-from-an-api-instead)
- [Before you scrape a table](https://quanticdata.io/blog/scrape-table-from-website-python/#before-you-scrape-a-table)

To scrape a table from a website with Python, download the page with requests, then read the table: BeautifulSoup walks the rows of a `<table>` one by one, and `pandas.read_html` turns every table on the page into a DataFrame in one call. Save it with `to_csv`. Below is the code we ran and its real output.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=scrape-table-from-website-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

We ran it on 20 November 2026 on a Windows 11 PC, Python 3.12, against the Wikipedia list of countries by population, once through a QuanticData residential proxy and once through a mobile proxy. Exit IPs are masked; USER and PASS stand for your credentials.

## What you need

| Package | Version we ran | What it does here |
| --- | --- | --- |
| requests | 2.34.2 | Downloads the page, through the proxy |
| beautifulsoup4 | 4.14.3 | Finds the table and reads each cell |
| lxml | 6.1.1 | The HTML parser behind both |
| pandas | 3.0.3 | Reads whole tables into DataFrames and writes the CSV |

Install them with `pip install requests beautifulsoup4 lxml pandas`. A proxy is optional for one page; it matters when you need the page as it is served in a given country.

## Step 1: download the page

Send the request through the proxy, with a User-Agent that names your client and a contact URL, as Wikimedia's User-Agent policy asks. The first call checks where the request leaves from; the second downloads the page.

```python
import os
from io import StringIO

import pandas as pd
import requests
from bs4 import BeautifulSoup

URL = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
proxy = os.environ["PROXY"]  # http://USER:PASS@HOST:PORT
proxies = {"http": proxy, "https": proxy}
headers = {"User-Agent": "ExampleResearchBot/1.0 (+https://example.com/bot)"}

ip = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=30).json()
print("exit:", "x.x.x.x", ip.get("city"), ip.get("region"), ip.get("country"), ip.get("org", "").split(" ", 1)[-1])

r = requests.get(URL, proxies=proxies, headers=headers, timeout=60)
print("status:", r.status_code, "bytes:", len(r.content))
```

```
exit: x.x.x.x Wilmington North Carolina US AT&T Enterprises, LLC
status: 200 bytes: 1390526
```

The proxy line is the one our generator gives, here a sticky US line of Residential Basic: `http://USER-type-residential-country-us-session-t01-lifetime-30:PASS@residentialboson.quanticdata.io:10000`. Keeping it in an environment variable keeps the password out of the script. For the full set of options, see [a proxy in Python Requests](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/).

## Step 2: read the table with BeautifulSoup

Find the table with a CSS selector, then take the text of every header and data cell, row by row. On Wikipedia, data tables carry the class `wikitable`; on other sites, open the page's developer tools and look at the `<table>` element for an id or class to select.

```
soup = BeautifulSoup(r.text, "lxml")
table = soup.select_one("table.wikitable")
rows = []
for tr in table.select("tr"):
    cells = [c.get_text(" ", strip=True) for c in tr.find_all(["th", "td"])]
    if cells:
        rows.append(cells)
print("BeautifulSoup:", len(rows), "rows; header:", rows[0])
print("  first data row:", rows[1])
```

```
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
  first data row: ['World', '8,232,000,000', '100%', '13 Jun 2025', 'UN projection [ 1 ] [ 3 ]', '']
```

241 rows: the header and 240 data rows, 6 columns each. Two details worth knowing:

- **Read both `th` and `td`.** Header cells are `th`; many tables also use `th` for the first cell of each row. Reading only `td` drops them.
- **Clean the footnotes.** The cell text keeps reference marks such as `[ 1 ]`. Strip them with a regular expression, or drop the `sup` elements before reading the text.

## Step 3: read every table with pandas.read_html

When the table is a plain HTML table, pandas does the walking for you. `read_html` returns a list with one DataFrame per `<table>` on the page:

```
tables = pd.read_html(StringIO(r.text))
print("pandas.read_html:", len(tables), "tables on the page")
df = tables[0]
print("  first table shape:", df.shape)
print(df.head(3).to_string())

df.to_csv("population.csv", index=False)
print("saved population.csv,", os.path.getsize("population.csv"), "bytes")
```

```
pandas.read_html: 3 tables on the page
  first table shape: (241, 6)
  Location    Population % of world         Date Source (official or from the United Nations) Notes
0    World  8.232000e+09       100%  13 Jun 2025                          UN projection[1][3]   NaN
1    India  1.429404e+09      17.3%   1 Jul 2026                       Official projection[4]   [b]
2    China  1.404890e+09      17.0%  31 Dec 2025                         Official estimate[5]   [c]
saved population.csv, 16057 bytes
```

pandas took the first row as column names, so its 241 rows are data rows, one more than BeautifulSoup found. The extra one is an empty `<tr>` in the page: the BeautifulSoup loop skips it (`if cells`), pandas keeps it as a row of NaN. `df = df.dropna(how="all")` removes it and leaves the same 240 rows. Note what else pandas did on its own: it read "8,232,000,000" as a number (shown as 8.232000e+09), and turned empty cells into NaN. Pass `match="Population"` to keep only tables that contain that text, which is safer than an index when a page has several tables, and wrap the HTML in `StringIO` as above: recent pandas versions expect a file-like object, not a raw string.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=scrape-table-from-website-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## BeautifulSoup or pandas: which to use

|  | BeautifulSoup | pandas.read_html |
| --- | --- | --- |
| Code for one table | About 7 lines | 1 line |
| Output | Lists of strings, exactly as shown | A DataFrame with typed columns |
| Links and attributes in cells | Yes, read `a["href"]` or any attribute | Text only by default |
| Irregular tables (merged cells, nested tables) | You decide how to handle them | Best effort |
| In our run | 241 rows: the header and 240 data rows, 6 columns | 3 tables; the first (241, 6), 240 rows after dropna |

Start with `read_html`; switch to BeautifulSoup when you need the links in the cells, a specific attribute, or control over odd rows. Both read the HTML you downloaded, so both need the table to be in it.

## When the table is not in the HTML

Two cases stop both methods:

- **The page builds the table with JavaScript.** Look at the page source (Ctrl+U, not the developer tools): if the numbers are not in it, the table is filled in by a script. A headless browser renders it first; see [Playwright for scraping](https://quanticdata.io/blog/how-to-use-playwright-for-scraping/).
- **The site answers with a block page.** A 403 or a challenge page has no table in it, and `read_html` stops with "No tables found". Our guide to [a 403 when scraping](https://quanticdata.io/blog/web-scraping-403-forbidden/) covers the causes.

## The same script through a mobile proxy

We ran the script again with only the `PROXY` variable changed, to a sticky US line of our [mobile proxies](https://quanticdata.io/mobile-proxies/) (`mobile.quanticdata.io:8080`). The request left from a carrier IP and the table came back the same:

```
exit: x.x.x.x Charlotte North Carolina US Verizon Business
status: 200 bytes: 1390526
BeautifulSoup: 241 rows; header: ['Location', 'Population', '% of world', 'Date', 'Source (official or from the United Nations )', 'Notes']
pandas.read_html: 3 tables on the page
  first table shape: (241, 6)
saved population.csv, 16057 bytes
```

Our [residential proxies](https://quanticdata.io/residential-proxies/) give you a home connection in the country you pick; mobile lines give you a carrier IP, Verizon in this run. The parsing code does not change.

## Getting the table from an API instead

If you would rather not run the download and the parser yourself, our [web scraping API](https://quanticdata.io/web-scraping-api/) returns the page as Markdown, with HTML tables kept as tables. One call from Python:

```
r = requests.post(
    "https://api.quanticdata.io/v1/scraper/extract",
    headers={"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"},
    json={"url": "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population",
          "format": "markdown", "country": "us"},
    timeout=120,
)
data = r.json()
p = data.get("payload", data)
md = p.get("content") or p.get("markdown") or ""
print("status:", r.status_code, "| engine:", p.get("engine"), "| tables:", (p.get("quality") or {}).get("tables"))
lines = [l for l in md.splitlines() if l.startswith("|")]
print("markdown table lines:", len(lines))
print("\n".join(lines[:3]))
```

The output, with the link inside the header shortened to `(...)`:

```
status: 200 | engine: tls | tables: 1
markdown table lines: 242
| Location | Population | % of  <br>world | Date | Source (official or from  <br>the [United Nations](...)) | Notes |
| --- | --- | --- | --- | --- | --- |
| **World** | 8,232,000,000 | 100% | 13 Jun 2025 | UN projection[\[1\]](#cite_note-UNFPA-1)[\[3\]](#cite_note-auto1-4) |     |
```

242 lines: the header, the separator and the 240 data rows of the same table, ready for a spreadsheet, a database or a language model. The call fetched the page over plain HTTP (engine tls). For more on the basics, see [web scraping with Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/).

## Before you scrape a table

Check the site's terms and robots.txt, keep the request rate modest, and prefer an official download or API when the site offers one: many statistics tables are also published as CSV. Wikipedia's text is under a Creative Commons licence that asks for attribution when you reuse it.

### Sources & further reading

- [pandas documentation: pandas.read_html](https://pandas.pydata.org/docs/reference/api/pandas.read_html.html)
- [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
- [Requests documentation: Proxies](https://requests.readthedocs.io/en/latest/user/advanced/#proxies)
- [Wikimedia Foundation: User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy)
- [Wikipedia: List of countries and dependencies by population](https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population)

## FAQ

Quick answers on how to scrape table data from website using python.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I scrape a table from a website using Python?

Download the page with `requests.get`, then either walk the `<table>` with BeautifulSoup, reading the `th` and `td` cells of each `tr`, or call `pandas.read_html` to get every table as a DataFrame. Save it with `df.to_csv`. In our run both read the same 240 data rows and 6 columns from a Wikipedia table.

### How do I scrape a table with BeautifulSoup?

Select the table, for example `soup.select_one("table.wikitable")`, loop over `table.select("tr")`, and for each row take `c.get_text(" ", strip=True)` for every cell found with `tr.find_all(["th", "td"])`. The first row holds the headers.

### Why does pandas.read_html say no tables found?

The HTML you passed has no `<table>` element. Either the page builds the table with JavaScript, so it is not in the downloaded source, or the site returned a block or consent page instead of the content. Check `r.status_code` and search `r.text` for a value you expect in the table.

### How do I save a scraped table to CSV in Python?

With pandas: `df.to_csv("population.csv", index=False)`. With BeautifulSoup rows, pass the list of lists to `csv.writer(...).writerows(rows)` or build a DataFrame with `pd.DataFrame(rows[1:], columns=rows[0])` first. Our CSV of the 241-row table was 16,057 bytes.

### Do I need a proxy to scrape a table?

Not for one page. A proxy helps when you need the page as served in a given country, or a home or carrier IP for your own script. In our test, the same script returned the same table through a residential and a mobile proxy with only the proxy line changed.

## Tables from any site, as data

Residential and mobile proxies for your own Python scripts, or one API call that returns the page with its tables as Markdown.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=scrape-table-from-website-python) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Collectors with Python examples

- [AliExpress scraper API in Python](https://quanticdata.io/collectors/aliexpress-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the AliExpress scraper collector, with Python examples.
- [eBay scraper API in Python](https://quanticdata.io/collectors/ebay-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the eBay scraper collector, with Python examples.
- [Google Images scraper API in Python](https://quanticdata.io/collectors/image-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Google Images scraper collector, with Python examples.
- [Video search API in Python](https://quanticdata.io/collectors/search-videos-api/python/): Collector · Install, first call, async runs, output fields and errors of the Video search collector, with Python examples.

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/scrape-table-from-website-python/ · Site index for AI: https://quanticdata.io/llms.txt
