# Python Web Scraping Libraries, Tested

> Python web scraping libraries compared on the same pages: requests, httpx, aiohttp, Beautiful Soup, lxml, parsel, Scrapy, Playwright, Selenium, with timings.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Python Web Scraping Libraries, Tested

# Python web scraping libraries: ten tools run on the same pages

Web scraping · Nov 20, 2026 · 6 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Python web scraping libraries sorted into four jobs: fetch the page with requests, httpx or aiohttp, parse it with Beautiful Soup, lxml or parsel, crawl a whole site with Scrapy, render JavaScript with Playwright or Selenium; the card shows our parse timings on the same page, from 15.28 ms with Beautiful Soup to 1.58 ms with parsel.](https://quanticdata.io/blog/python-web-scraping-libraries/hero.webp?v=20261120h)

On this page

- [The ten libraries at a glance](https://quanticdata.io/blog/python-web-scraping-libraries/#the-ten-libraries-at-a-glance)
- [Fetching: requests, httpx, aiohttp, urllib3](https://quanticdata.io/blog/python-web-scraping-libraries/#fetching-requests-httpx-aiohttp-urllib3)
- [Parsing: Beautiful Soup, lxml, parsel](https://quanticdata.io/blog/python-web-scraping-libraries/#parsing-beautiful-soup-lxml-parsel)
- [Crawling: Scrapy](https://quanticdata.io/blog/python-web-scraping-libraries/#crawling-scrapy)
- [JavaScript pages: Playwright and Selenium](https://quanticdata.io/blog/python-web-scraping-libraries/#javascript-pages-playwright-and-selenium)
- [Which Python web scraping library to choose](https://quanticdata.io/blog/python-web-scraping-libraries/#which-python-web-scraping-library-to-choose)
- [Which proxy for which library](https://quanticdata.io/blog/python-web-scraping-libraries/#which-proxy-for-which-library)

The best Python web scraping libraries split into four jobs: **requests** or **httpx** to download a page, **Beautiful Soup** or **lxml** to read it, **Scrapy** to crawl a whole site, and **Playwright** or **Selenium** when the page builds itself with JavaScript. Pick one per job; most scrapers use two.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=python-web-scraping-libraries) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

Lists of "top libraries" rarely run them. We did: on 20 November 2026 we pointed ten libraries at the same public practice pages (books.toscrape.com and quotes.toscrape.com, built for scraping practice) from Python 3.12.10 on a Windows 11 PC, through QuanticData residential and mobile proxies. Every timing below comes from those runs.

## The ten libraries at a glance

| Library | Job | Version we ran | What it did in our test |
| --- | --- | --- | --- |
| requests | Fetch (blocking) | 2.34.2 | 200, 51,294 bytes through Residential Basic |
| httpx | Fetch (sync and async) | 0.28.1 | Same page, same bytes; 50 pages in 6.0 s async |
| aiohttp | Fetch (async) | 3.14.3 | 50 pages in 6.4 s, 10 at a time |
| urllib3 | Fetch (low level) | 2.8.0 | Same page, same bytes |
| Beautiful Soup | Parse | 4.14.3 | Parse and extract 20 books: 12.91 ms with the lxml parser |
| lxml | Parse (XPath) | 6.1.1 | 20 books in 1.81 ms |
| parsel | Parse (CSS and XPath) | 1.12.1 | 20 books in 1.58 ms |
| Scrapy | Crawl | 2.20.0 | 50 pages, 1,000 items, following the Next link |
| Playwright | Render JavaScript | 1.64.1 | 10 quotes from a JavaScript page in 3.34 s |
| Selenium | Render JavaScript | 4.53.0 | Same elements, Microsoft Edge 157 headless |

## Fetching: requests, httpx, aiohttp, urllib3

All four downloaded books.toscrape.com through the same Residential Basic gateway and got the same 51,294 bytes with status 200. On one page they are interchangeable; the difference is how they behave at volume.

```python
import requests
PROXY = "http://USER:PASS@residentialboson.quanticdata.io:9000"
r = requests.get("https://books.toscrape.com/",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content))   # 200 51294
```

- **requests** is the default for scripts: the simplest API, sessions, cookies, and the proxy dictionary everyone knows. Setup details: [a proxy in Python requests](https://quanticdata.io/blog/how-to-use-a-proxy-with-python-requests/).
- **httpx** has nearly the same API as requests and adds async and HTTP/2. It takes one `proxy=` argument: see [the httpx proxy setting](https://quanticdata.io/blog/httpx-proxy/).
- **aiohttp** is async only and built for many requests at once; see [aiohttp with a proxy](https://quanticdata.io/blog/aiohttp-proxy-setup/).
- **urllib3** sits under requests. Use it directly only when you want the connection pool without the extras.

Where async pays: we fetched the 50 catalogue pages of the practice shop through one of our proxies. requests, one page after another, took 15.6 s. aiohttp with 10 requests in flight took 6.4 s, httpx's AsyncClient 6.0 s. Same pages, same 1,000 books.

## Parsing: Beautiful Soup, lxml, parsel

We parsed the same downloaded page 200 times with each parser and extracted the 20 book titles and prices, so these times are parse and extract together. All four returned identical lists. Median time per page:

| Parser | Median per page | Selectors |
| --- | --- | --- |
| Beautiful Soup + html.parser | 15.28 ms | find, find_all, CSS via select |
| Beautiful Soup + lxml | 12.91 ms | Same API, faster backend |
| lxml (XPath) | 1.81 ms | XPath, CSS with cssselect |
| parsel (CSS) | 1.58 ms | CSS and XPath, the selector engine of Scrapy |

Beautiful Soup is the easiest to learn and forgives broken HTML; it is fast enough for thousands of pages. When you parse millions, lxml or parsel are roughly eight times faster on this page. The same extraction in each:

```python
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")
books = [(a.h3.a["title"], a.select_one(".price_color").text)
         for a in soup.select("article.product_pod")]

import lxml.html
doc = lxml.html.fromstring(html)
titles = doc.xpath('//article[@class="product_pod"]/h3/a/@title')

import parsel
sel = parsel.Selector(text=html)
titles = sel.css("article.product_pod h3 a::attr(title)").getall()
```

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=python-web-scraping-libraries) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Crawling: Scrapy

Scrapy is a framework, not a library you call: you write a spider, and it schedules requests, retries, follows links and exports items. Our spider started on the first catalogue page and followed the Next link: 50 pages and 1,000 books, with no loop or retry code of ours. The proxy goes in `meta={"proxy": PROXY}` on each request; more in [proxies in Scrapy](https://quanticdata.io/blog/scrapy-proxy-middleware/).

```
class Books(scrapy.Spider):
    name = "books"
    def parse(self, r):
        for b in r.css("article.product_pod"):
            yield {"title": b.css("h3 a::attr(title)").get(),
                   "price": b.css("p.price_color::text").get()}
        nxt = r.css("li.next a::attr(href)").get()
        if nxt:
            yield r.follow(nxt, self.parse, meta={"proxy": PROXY})
```

## JavaScript pages: Playwright and Selenium

quotes.toscrape.com/js/ draws its quotes with JavaScript. requests downloaded the page (5,808 bytes) and Beautiful Soup found zero quotes: the data sits in a script tag. Playwright, driving Chromium 156 through a mobile line in the US (Verizon Business), found all 10 quotes in 3.34 s.

```python
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER", "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")
    print(page.locator("div.quote").count())   # 10
```

Selenium does the same job with a different API (`find_element(By.CSS_SELECTOR, ...)`) and drives the browsers you have installed; we ran it with Microsoft Edge 157. Playwright takes the proxy username and password in `launch()`, which saves a step; details in [Playwright with a proxy](https://quanticdata.io/blog/playwright-proxy/).

## Which Python web scraping library to choose

| Your job | Library |
| --- | --- |
| A few static pages in a script | requests + Beautiful Soup |
| Thousands of pages, fast | httpx or aiohttp + lxml or parsel |
| A whole site, with link following and export | Scrapy |
| Pages that need JavaScript, clicks or scrolling | Playwright (or Selenium) |
| Data without running any of it | Our [web scraping API](https://quanticdata.io/web-scraping-api/), which returns the page or structured data |

A longer walk-through of a first scraper is in [how to web scrape using Python](https://quanticdata.io/blog/how-to-web-scraping-using-python/).

## Which proxy for which library

Every library above takes a proxy URL. On practice pages any exit works; on real sites the exit decides what you get back.

- **Residential Basic** has a rotating gateway on port 9000 and a session gateway on port 10000, with the country in the username: it served our four HTTP clients.
- **Residential Premium** served our Scrapy crawl: home IPs by country, with a session ID in the username.
- **Mobile** exits on carrier networks (our Playwright test left from Verizon Business in Arkansas); use it for sites that serve mobile visitors differently.

Plans are on the pages for [residential proxies](https://quanticdata.io/residential-proxies/) and [mobile proxies](https://quanticdata.io/mobile-proxies/), and the [Residential Basic free trial](https://quanticdata.io/residential-proxy-free-trial/) lets you run the code above before you buy.

### Sources & further reading

- [Requests documentation](https://requests.readthedocs.io/en/latest/)
- [Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
- [lxml: XPath and XSLT with lxml](https://lxml.de/xpathxslt.html)
- [Scrapy documentation](https://docs.scrapy.org/en/latest/)
- [Playwright for Python documentation](https://playwright.dev/python/docs/intro)

## FAQ

Quick answers on python web scraping libraries.

[Something else? Ask us](mailto:hello@quanticdata.io)

### What is the best Python library for web scraping?

For most jobs, requests to download and Beautiful Soup to parse. For speed at volume, httpx or aiohttp with lxml or parsel. For whole sites, Scrapy. For JavaScript pages, Playwright. In our test on the same page, parsel parsed in 1.58 ms and Beautiful Soup with html.parser in 15.28 ms.

### Is Beautiful Soup or lxml better for scraping?

Beautiful Soup is easier and more forgiving; lxml is faster and speaks XPath. To parse and extract 20 books on our test page, lxml took 1.81 ms against 12.91 ms for Beautiful Soup with the lxml backend. You can also combine them: `BeautifulSoup(html, "lxml")`.

### Is Scrapy better than requests and Beautiful Soup?

For a whole site, yes: Scrapy follows links, retries and exports without code of yours. In our test it crawled 50 pages and 1,000 items by following the Next link. For one page in a script, requests and Beautiful Soup are less setup.

### Which Python library scrapes JavaScript websites?

Playwright or Selenium, which drive a real browser. On a page that draws its content with JavaScript, requests and Beautiful Soup found 0 quotes; Playwright found all 10 in 3.34 s. Sometimes the data is also in a script tag or an API call you can fetch directly.

### Can these Python libraries use a proxy?

All of them. requests takes a `proxies` dictionary, httpx a `proxy=` argument, aiohttp `proxy=` per request, Scrapy `meta={"proxy": ...}`, Playwright a `proxy` dictionary in `launch()`. We ran every test in this post through QuanticData residential or mobile proxies.

## Run any Python scraping library through a home or mobile IP

Residential and mobile proxies that plug into requests, httpx, aiohttp, Scrapy, Playwright and Selenium with one proxy URL.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=python-web-scraping-libraries) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Collectors with Python examples

- [AliExpress scraper API in Python](https://quanticdata.io/collectors/aliexpress-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the AliExpress scraper collector, with Python examples.
- [eBay scraper API in Python](https://quanticdata.io/collectors/ebay-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the eBay scraper collector, with Python examples.
- [Google Images scraper API in Python](https://quanticdata.io/collectors/image-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Google Images scraper collector, with Python examples.
- [Video search API in Python](https://quanticdata.io/collectors/search-videos-api/python/): Collector · Install, first call, async runs, output fields and errors of the Video search collector, with Python examples.

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/python-web-scraping-libraries/ · Site index for AI: https://quanticdata.io/llms.txt
