Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Python web scraping libraries: ten tools run on the same pages

Python web scraping libraries sorted into four jobs: fetch the page with requests, httpx or aiohttp, parse it with Beautiful Soup, lxml or parsel, crawl a whole site with Scrapy, render JavaScript with Playwright or Selenium; the card shows our parse timings on the same page, from 15.28 ms with Beautiful Soup to 1.58 ms with parsel.
Python web scraping libraries sorted into four jobs: fetch the page with requests, httpx or aiohttp, parse it with Beautiful Soup, lxml or parsel, crawl a whole site with Scrapy, render JavaScript with Playwright or Selenium; the card shows our parse timings on the same page, from 15.28 ms with Beautiful Soup to 1.58 ms with parsel.

The best Python web scraping libraries split into four jobs: requests or httpx to download a page, Beautiful Soup or lxml to read it, Scrapy to crawl a whole site, and Playwright or Selenium when the page builds itself with JavaScript. Pick one per job; most scrapers use two.

Lists of "top libraries" rarely run them. We did: on 20 November 2026 we pointed ten libraries at the same public practice pages (books.toscrape.com and quotes.toscrape.com, built for scraping practice) from Python 3.12.10 on a Windows 11 PC, through QuanticData residential and mobile proxies. Every timing below comes from those runs.

The ten libraries at a glance

LibraryJobVersion we ranWhat it did in our test
requestsFetch (blocking)2.34.2200, 51,294 bytes through Residential Basic
httpxFetch (sync and async)0.28.1Same page, same bytes; 50 pages in 6.0 s async
aiohttpFetch (async)3.14.350 pages in 6.4 s, 10 at a time
urllib3Fetch (low level)2.8.0Same page, same bytes
Beautiful SoupParse4.14.3Parse and extract 20 books: 12.91 ms with the lxml parser
lxmlParse (XPath)6.1.120 books in 1.81 ms
parselParse (CSS and XPath)1.12.120 books in 1.58 ms
ScrapyCrawl2.20.050 pages, 1,000 items, following the Next link
PlaywrightRender JavaScript1.64.110 quotes from a JavaScript page in 3.34 s
SeleniumRender JavaScript4.53.0Same elements, Microsoft Edge 157 headless

Fetching: requests, httpx, aiohttp, urllib3

All four downloaded books.toscrape.com through the same Residential Basic gateway and got the same 51,294 bytes with status 200. On one page they are interchangeable; the difference is how they behave at volume.

import requests
PROXY = "http://USER:PASS@residentialboson.quanticdata.io:9000"
r = requests.get("https://books.toscrape.com/",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)
print(r.status_code, len(r.content))   # 200 51294
  • requests is the default for scripts: the simplest API, sessions, cookies, and the proxy dictionary everyone knows. Setup details: a proxy in Python requests.
  • httpx has nearly the same API as requests and adds async and HTTP/2. It takes one proxy= argument: see the httpx proxy setting.
  • aiohttp is async only and built for many requests at once; see aiohttp with a proxy.
  • urllib3 sits under requests. Use it directly only when you want the connection pool without the extras.

Where async pays: we fetched the 50 catalogue pages of the practice shop through one of our proxies. requests, one page after another, took 15.6 s. aiohttp with 10 requests in flight took 6.4 s, httpx's AsyncClient 6.0 s. Same pages, same 1,000 books.

Parsing: Beautiful Soup, lxml, parsel

We parsed the same downloaded page 200 times with each parser and extracted the 20 book titles and prices, so these times are parse and extract together. All four returned identical lists. Median time per page:

ParserMedian per pageSelectors
Beautiful Soup + html.parser15.28 msfind, find_all, CSS via select
Beautiful Soup + lxml12.91 msSame API, faster backend
lxml (XPath)1.81 msXPath, CSS with cssselect
parsel (CSS)1.58 msCSS and XPath, the selector engine of Scrapy

Beautiful Soup is the easiest to learn and forgives broken HTML; it is fast enough for thousands of pages. When you parse millions, lxml or parsel are roughly eight times faster on this page. The same extraction in each:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")
books = [(a.h3.a["title"], a.select_one(".price_color").text)
         for a in soup.select("article.product_pod")]

import lxml.html
doc = lxml.html.fromstring(html)
titles = doc.xpath('//article[@class="product_pod"]/h3/a/@title')

import parsel
sel = parsel.Selector(text=html)
titles = sel.css("article.product_pod h3 a::attr(title)").getall()

Crawling: Scrapy

Scrapy is a framework, not a library you call: you write a spider, and it schedules requests, retries, follows links and exports items. Our spider started on the first catalogue page and followed the Next link: 50 pages and 1,000 books, with no loop or retry code of ours. The proxy goes in meta={"proxy": PROXY} on each request; more in proxies in Scrapy.

class Books(scrapy.Spider):
    name = "books"
    def parse(self, r):
        for b in r.css("article.product_pod"):
            yield {"title": b.css("h3 a::attr(title)").get(),
                   "price": b.css("p.price_color::text").get()}
        nxt = r.css("li.next a::attr(href)").get()
        if nxt:
            yield r.follow(nxt, self.parse, meta={"proxy": PROXY})

JavaScript pages: Playwright and Selenium

quotes.toscrape.com/js/ draws its quotes with JavaScript. requests downloaded the page (5,808 bytes) and Beautiful Soup found zero quotes: the data sits in a script tag. Playwright, driving Chromium 156 through a mobile line in the US (Verizon Business), found all 10 quotes in 3.34 s.

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    b = p.chromium.launch(proxy={"server": "http://mobile.quanticdata.io:8080",
                                 "username": "USER", "password": "PASS"})
    page = b.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")
    print(page.locator("div.quote").count())   # 10

Selenium does the same job with a different API (find_element(By.CSS_SELECTOR, ...)) and drives the browsers you have installed; we ran it with Microsoft Edge 157. Playwright takes the proxy username and password in launch(), which saves a step; details in Playwright with a proxy.

Which Python web scraping library to choose

Your jobLibrary
A few static pages in a scriptrequests + Beautiful Soup
Thousands of pages, fasthttpx or aiohttp + lxml or parsel
A whole site, with link following and exportScrapy
Pages that need JavaScript, clicks or scrollingPlaywright (or Selenium)
Data without running any of itOur web scraping API, which returns the page or structured data

A longer walk-through of a first scraper is in how to web scrape using Python.

Which proxy for which library

Every library above takes a proxy URL. On practice pages any exit works; on real sites the exit decides what you get back.

  • Residential Basic has a rotating gateway on port 9000 and a session gateway on port 10000, with the country in the username: it served our four HTTP clients.
  • Residential Premium served our Scrapy crawl: home IPs by country, with a session ID in the username.
  • Mobile exits on carrier networks (our Playwright test left from Verizon Business in Arkansas); use it for sites that serve mobile visitors differently.

Plans are on the pages for residential proxies and mobile proxies, and the Residential Basic free trial lets you run the code above before you buy.

Sources & further reading

FAQ

Quick answers on python web scraping libraries.

Something else? Ask us

What is the best Python library for web scraping?

For most jobs, requests to download and Beautiful Soup to parse. For speed at volume, httpx or aiohttp with lxml or parsel. For whole sites, Scrapy. For JavaScript pages, Playwright. In our test on the same page, parsel parsed in 1.58 ms and Beautiful Soup with html.parser in 15.28 ms.

Is Beautiful Soup or lxml better for scraping?

Beautiful Soup is easier and more forgiving; lxml is faster and speaks XPath. To parse and extract 20 books on our test page, lxml took 1.81 ms against 12.91 ms for Beautiful Soup with the lxml backend. You can also combine them: BeautifulSoup(html, "lxml").

Is Scrapy better than requests and Beautiful Soup?

For a whole site, yes: Scrapy follows links, retries and exports without code of yours. In our test it crawled 50 pages and 1,000 items by following the Next link. For one page in a script, requests and Beautiful Soup are less setup.

Which Python library scrapes JavaScript websites?

Playwright or Selenium, which drive a real browser. On a page that draws its content with JavaScript, requests and Beautiful Soup found 0 quotes; Playwright found all 10 in 3.34 s. Sometimes the data is also in a script tag or an API call you can fetch directly.

Can these Python libraries use a proxy?

All of them. requests takes a proxies dictionary, httpx a proxy= argument, aiohttp proxy= per request, Scrapy meta={"proxy": ...}, Playwright a proxy dictionary in launch(). We ran every test in this post through QuanticData residential or mobile proxies.

Run any Python scraping library through a home or mobile IP

Residential and mobile proxies that plug into requests, httpx, aiohttp, Scrapy, Playwright and Selenium with one proxy URL.

Collectors with Python examples

Related reading