The best Python web scraping libraries split into four jobs: requests or httpx to download a page, Beautiful Soup or lxml to read it, Scrapy to crawl a whole site, and Playwright or Selenium when the page builds itself with JavaScript. Pick one per job; most scrapers use two.
Lists of "top libraries" rarely run them. We did: on 20 November 2026 we pointed ten libraries at the same public practice pages (books.toscrape.com and quotes.toscrape.com, built for scraping practice) from Python 3.12.10 on a Windows 11 PC, through QuanticData residential and mobile proxies. Every timing below comes from those runs.
The ten libraries at a glance
| Library | Job | Version we ran | What it did in our test |
|---|---|---|---|
| requests | Fetch (blocking) | 2.34.2 | 200, 51,294 bytes through Residential Basic |
| httpx | Fetch (sync and async) | 0.28.1 | Same page, same bytes; 50 pages in 6.0 s async |
| aiohttp | Fetch (async) | 3.14.3 | 50 pages in 6.4 s, 10 at a time |
| urllib3 | Fetch (low level) | 2.8.0 | Same page, same bytes |
| Beautiful Soup | Parse | 4.14.3 | Parse and extract 20 books: 12.91 ms with the lxml parser |
| lxml | Parse (XPath) | 6.1.1 | 20 books in 1.81 ms |
| parsel | Parse (CSS and XPath) | 1.12.1 | 20 books in 1.58 ms |
| Scrapy | Crawl | 2.20.0 | 50 pages, 1,000 items, following the Next link |
| Playwright | Render JavaScript | 1.64.1 | 10 quotes from a JavaScript page in 3.34 s |
| Selenium | Render JavaScript | 4.53.0 | Same elements, Microsoft Edge 157 headless |
Fetching: requests, httpx, aiohttp, urllib3
All four downloaded books.toscrape.com through the same Residential Basic gateway and got the same 51,294 bytes with status 200. On one page they are interchangeable; the difference is how they behave at volume.
import requests
PROXY = "http://USER:PASS@residentialboson.quanticdata.io:9000"
r = requests.get("https://books.toscrape.com/",
proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content)) # 200 51294
- requests is the default for scripts: the simplest API, sessions, cookies, and the proxy dictionary everyone knows. Setup details: a proxy in Python requests.
- httpx has nearly the same API as requests and adds async and HTTP/2. It takes one
proxy=argument: see the httpx proxy setting. - aiohttp is async only and built for many requests at once; see aiohttp with a proxy.
- urllib3 sits under requests. Use it directly only when you want the connection pool without the extras.
Where async pays: we fetched the 50 catalogue pages of the practice shop through one of our proxies. requests, one page after another, took 15.6 s. aiohttp with 10 requests in flight took 6.4 s, httpx's AsyncClient 6.0 s. Same pages, same 1,000 books.
Parsing: Beautiful Soup, lxml, parsel
We parsed the same downloaded page 200 times with each parser and extracted the 20 book titles and prices, so these times are parse and extract together. All four returned identical lists. Median time per page:
| Parser | Median per page | Selectors |
|---|---|---|
| Beautiful Soup + html.parser | 15.28 ms | find, find_all, CSS via select |
| Beautiful Soup + lxml | 12.91 ms | Same API, faster backend |
| lxml (XPath) | 1.81 ms | XPath, CSS with cssselect |
| parsel (CSS) | 1.58 ms | CSS and XPath, the selector engine of Scrapy |
Beautiful Soup is the easiest to learn and forgives broken HTML; it is fast enough for thousands of pages. When you parse millions, lxml or parsel are roughly eight times faster on this page. The same extraction in each:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")
books = [(a.h3.a["title"], a.select_one(".price_color").text)
for a in soup.select("article.product_pod")]
import lxml.html
doc = lxml.html.fromstring(html)
titles = doc.xpath('//article[@class="product_pod"]/h3/a/@title')
import parsel
sel = parsel.Selector(text=html)
titles = sel.css("article.product_pod h3 a::attr(title)").getall()
Crawling: Scrapy
Scrapy is a framework, not a library you call: you write a spider, and it schedules requests, retries, follows links and exports items. Our spider started on the first catalogue page and followed the Next link: 50 pages and 1,000 books, with no loop or retry code of ours. The proxy goes in meta={"proxy": PROXY} on each request; more in proxies in Scrapy.
class Books(scrapy.Spider):
name = "books"
def parse(self, r):
for b in r.css("article.product_pod"):
yield {"title": b.css("h3 a::attr(title)").get(),
"price": b.css("p.price_color::text").get()}
nxt = r.css("li.next a::attr(href)").get()
if nxt:
yield r.follow(nxt, self.parse, meta={"proxy": PROXY})
JavaScript pages: Playwright and Selenium
quotes.toscrape.com/js/ draws its quotes with JavaScript. requests downloaded the page (5,808 bytes) and Beautiful Soup found zero quotes: the data sits in a script tag. Playwright, driving Chromium 156 through a mobile line in the US (Verizon Business), found all 10 quotes in 3.34 s.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
"username": "USER", "password": "PASS"})
page = b.new_page()
page.goto("https://quotes.toscrape.com/js/")
page.wait_for_selector("div.quote")
print(page.locator("div.quote").count()) # 10
Selenium does the same job with a different API (find_element(By.CSS_SELECTOR, ...)) and drives the browsers you have installed; we ran it with Microsoft Edge 157. Playwright takes the proxy username and password in launch(), which saves a step; details in Playwright with a proxy.
Which Python web scraping library to choose
| Your job | Library |
|---|---|
| A few static pages in a script | requests + Beautiful Soup |
| Thousands of pages, fast | httpx or aiohttp + lxml or parsel |
| A whole site, with link following and export | Scrapy |
| Pages that need JavaScript, clicks or scrolling | Playwright (or Selenium) |
| Data without running any of it | Our web scraping API, which returns the page or structured data |
A longer walk-through of a first scraper is in how to web scrape using Python.
Which proxy for which library
Every library above takes a proxy URL. On practice pages any exit works; on real sites the exit decides what you get back.
- Residential Basic has a rotating gateway on port 9000 and a session gateway on port 10000, with the country in the username: it served our four HTTP clients.
- Residential Premium served our Scrapy crawl: home IPs by country, with a session ID in the username.
- Mobile exits on carrier networks (our Playwright test left from Verizon Business in Arkansas); use it for sites that serve mobile visitors differently.
Plans are on the pages for residential proxies and mobile proxies, and the Residential Basic free trial lets you run the code above before you buy.


