Documentation Python quickstart Blog Free tools Enterprise solutions hello@quanticdata.ioLog in

Async web scraping in Python: 50 pages with aiohttp and httpx, timed against requests

Async web scraping in Python compared with plain requests on the same 50 catalogue pages through a QuanticData proxy: httpx with asyncio and a semaphore of 10 finished in 6.0 seconds, aiohttp in 6.4 seconds, requests one page after another in 15.6 seconds; the card lists the pieces of an async scraper.
Async web scraping in Python compared with plain requests on the same 50 catalogue pages through a QuanticData proxy: httpx with asyncio and a semaphore of 10 finished in 6.0 seconds, aiohttp in 6.4 seconds, requests one page after another in 15.6 seconds; the card lists the pieces of an async scraper.

Async web scraping in Python sends many requests and waits for them together instead of one after another. Use asyncio with an async HTTP client such as aiohttp or httpx.AsyncClient, share one client, cap the requests in flight with asyncio.Semaphore, and collect results with asyncio.gather.

To show what that buys, on 20 November 2026 we fetched the same 50 catalogue pages of books.toscrape.com (a public shop made for scraping practice) three ways, with Python 3.12.10, through the same QuanticData proxy, and parsed 1,000 books each time.

The timings

ClientRequests in flight50 pages
httpx 0.28.1, AsyncClient106.0 s
aiohttp 3.14.3106.4 s
requests 2.34.2, one page after another115.6 s

Every run returned all 50 pages and 1,000 books with 0 errors. The async clients kept ten downloads waiting at once; the sequential run waited for every page in turn, which is all the difference. (Our Scrapy vs Beautiful Soup comparison was a separate run, with its own numbers.)

Async scraping with aiohttp, step by step

  1. Open one ClientSession for the whole run, with a timeout. A session per request throws away connection reuse.
  2. Create a Semaphore that allows N requests at a time. Without it, gather fires every request at once.
  3. Write one coroutine that fetches and parses one page inside async with sem.
  4. gather them all, with return_exceptions=True so one failure does not cancel the rest.
import asyncio, aiohttp
from bs4 import BeautifulSoup

PROXY = "http://USER:PASS@HOST:PORT"   # your QuanticData proxy line
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]

def titles(html):
    return [a["title"] for a in BeautifulSoup(html, "lxml").select("article.product_pod h3 a")]

async def main(limit=10):
    sem = asyncio.Semaphore(limit)
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as s:
        async def one(url):
            async with sem:
                async with s.get(url, proxy=PROXY) as r:
                    r.raise_for_status()
                    return titles(await r.text())
        return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)

results = asyncio.run(main())
ok = [x for x in results if isinstance(x, list)]
errors = [x for x in results if not isinstance(x, list)]
print(len(ok), sum(map(len, ok)), len(errors))   # 50 1000 0

aiohttp takes the proxy per request with proxy=; username and password can sit in the URL, as here. All its proxy options, SOCKS5 included, are in aiohttp proxy setup.

The last line is the check worth keeping in any async scraper: 50 pages came back as lists, 1,000 titles in total, and 0 results were exceptions. When that last number is not zero, the exceptions are in the same list, in the same order as the URLs, so you can see which pages to fetch again.

The same scraper with httpx

httpx has the requests-style API and an async client; the proxy is set once on the client:

import httpx

async def main(limit=10):
    sem = asyncio.Semaphore(limit)
    async with httpx.AsyncClient(proxy=PROXY, timeout=30) as c:
        async def one(url):
            async with sem:
                r = await c.get(url)
                r.raise_for_status()
                return titles(r.text)
        return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)

Note proxy=, singular: httpx 0.28 removed the old proxies= argument. Version details are in the httpx proxy guide.

aiohttp or httpx?

aiohttphttpx
APIIts own, context managers per responseLike requests
Sync mode tooNoYes, same API
Proxyproxy= per request (or on the session)proxy= on the client
Our 50 pages6.4 s6.0 s

Both did the job; pick httpx if you also want a sync API or already know requests, aiohttp if your code base already uses it. The gap between them is smaller than the gap between either one and sequential requests.

How many requests at once?

The semaphore value is a choice about the site, not about Python. Ten was plenty for 50 pages. More in flight finishes sooner until the site starts answering 429 Too Many Requests or slowing down, and then everything gets slower. Start low, watch status codes and response times, and raise it gradually. For different sites, keep a semaphore per domain.

Errors, timeouts and retries

  • Always set a timeout. One hung request otherwise holds a semaphore slot forever. We used 30 seconds total per request.
  • Use return_exceptions=True and sort results afterwards, as in the code above: the run finishes, and you retry only the URLs that failed.
  • Retry with a pause, for example with tenacity around one(). The same rules as for sync code apply: retries and timeouts in requests.
  • Parse outside the network wait. Beautiful Soup is synchronous; on 50 pages it is fine inside the coroutine, on very large pages move parsing to a thread with asyncio.to_thread.

Async scraping and proxies

The proxy goes on the client or on each request, and the rotating gateway or a session ID is chosen in the username. In our run, ten requests sent at once to ipinfo.io/ip through httpx came back from 10 different IPs, on Mobile and on Residential Premium alike.

  • Mobile: carrier IPs, with the country in the username.
  • Residential Premium: home IPs by country, with the country in the username.
  • For a task that belongs together (a login, a basket), put a session ID in the proxy username instead of using the rotating gateway.

Plans: mobile proxies, residential proxies and rotating proxies. For whole-site crawls with concurrency built in, see Scrapy vs Beautiful Soup.

Sources & further reading

FAQ

Quick answers on async web scraping python.

Something else? Ask us

How do I scrape websites asynchronously in Python?

Use asyncio with aiohttp or httpx.AsyncClient: one shared session, a coroutine per URL wrapped in async with semaphore, and asyncio.gather(*tasks, return_exceptions=True). Our version fetched 50 pages in 6.0 to 6.4 s.

Is async scraping faster than requests?

For many pages, yes, because the waits overlap. Through the same proxy, httpx async fetched 50 pages in 6.0 s and requests, one page at a time, in 15.6 s. For a single page there is no gain.

Should I use aiohttp or httpx for async scraping?

Both work. httpx has a requests-like API and a sync mode; aiohttp is async only. In our run httpx took 6.0 s and aiohttp 6.4 s for the same 50 pages.

How many concurrent requests should an async scraper send?

Start low, around 5 to 10 per site, and raise it while status codes stay 200 and response times stay flat. Cap it with asyncio.Semaphore; 10 was enough for our 50 pages.

How do I use proxies with async scraping?

Pass proxy= per request in aiohttp or on the AsyncClient in httpx. In our run, ten requests sent at once came back from 10 different IPs.

Rotating proxies for aiohttp and httpx

Mobile and residential proxies for aiohttp and httpx, with rotation and session settings in the username.

Collectors with Python examples

Related reading