Async web scraping in Python sends many requests and waits for them together instead of one after another. Use asyncio with an async HTTP client such as aiohttp or httpx.AsyncClient, share one client, cap the requests in flight with asyncio.Semaphore, and collect results with asyncio.gather.
To show what that buys, on 20 November 2026 we fetched the same 50 catalogue pages of books.toscrape.com (a public shop made for scraping practice) three ways, with Python 3.12.10, through the same QuanticData proxy, and parsed 1,000 books each time.
The timings
| Client | Requests in flight | 50 pages |
|---|---|---|
| httpx 0.28.1, AsyncClient | 10 | 6.0 s |
| aiohttp 3.14.3 | 10 | 6.4 s |
| requests 2.34.2, one page after another | 1 | 15.6 s |
Every run returned all 50 pages and 1,000 books with 0 errors. The async clients kept ten downloads waiting at once; the sequential run waited for every page in turn, which is all the difference. (Our Scrapy vs Beautiful Soup comparison was a separate run, with its own numbers.)
Async scraping with aiohttp, step by step
- Open one ClientSession for the whole run, with a timeout. A session per request throws away connection reuse.
- Create a Semaphore that allows N requests at a time. Without it, gather fires every request at once.
- Write one coroutine that fetches and parses one page inside
async with sem. - gather them all, with
return_exceptions=Trueso one failure does not cancel the rest.
import asyncio, aiohttp
from bs4 import BeautifulSoup
PROXY = "http://USER:PASS@HOST:PORT" # your QuanticData proxy line
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]
def titles(html):
return [a["title"] for a in BeautifulSoup(html, "lxml").select("article.product_pod h3 a")]
async def main(limit=10):
sem = asyncio.Semaphore(limit)
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as s:
async def one(url):
async with sem:
async with s.get(url, proxy=PROXY) as r:
r.raise_for_status()
return titles(await r.text())
return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)
results = asyncio.run(main())
ok = [x for x in results if isinstance(x, list)]
errors = [x for x in results if not isinstance(x, list)]
print(len(ok), sum(map(len, ok)), len(errors)) # 50 1000 0
aiohttp takes the proxy per request with proxy=; username and password can sit in the URL, as here. All its proxy options, SOCKS5 included, are in aiohttp proxy setup.
The last line is the check worth keeping in any async scraper: 50 pages came back as lists, 1,000 titles in total, and 0 results were exceptions. When that last number is not zero, the exceptions are in the same list, in the same order as the URLs, so you can see which pages to fetch again.
The same scraper with httpx
httpx has the requests-style API and an async client; the proxy is set once on the client:
import httpx
async def main(limit=10):
sem = asyncio.Semaphore(limit)
async with httpx.AsyncClient(proxy=PROXY, timeout=30) as c:
async def one(url):
async with sem:
r = await c.get(url)
r.raise_for_status()
return titles(r.text)
return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)
Note proxy=, singular: httpx 0.28 removed the old proxies= argument. Version details are in the httpx proxy guide.
aiohttp or httpx?
| aiohttp | httpx | |
|---|---|---|
| API | Its own, context managers per response | Like requests |
| Sync mode too | No | Yes, same API |
| Proxy | proxy= per request (or on the session) | proxy= on the client |
| Our 50 pages | 6.4 s | 6.0 s |
Both did the job; pick httpx if you also want a sync API or already know requests, aiohttp if your code base already uses it. The gap between them is smaller than the gap between either one and sequential requests.
How many requests at once?
The semaphore value is a choice about the site, not about Python. Ten was plenty for 50 pages. More in flight finishes sooner until the site starts answering 429 Too Many Requests or slowing down, and then everything gets slower. Start low, watch status codes and response times, and raise it gradually. For different sites, keep a semaphore per domain.
Errors, timeouts and retries
- Always set a timeout. One hung request otherwise holds a semaphore slot forever. We used 30 seconds total per request.
- Use return_exceptions=True and sort results afterwards, as in the code above: the run finishes, and you retry only the URLs that failed.
- Retry with a pause, for example with tenacity around
one(). The same rules as for sync code apply: retries and timeouts in requests. - Parse outside the network wait. Beautiful Soup is synchronous; on 50 pages it is fine inside the coroutine, on very large pages move parsing to a thread with
asyncio.to_thread.
Async scraping and proxies
The proxy goes on the client or on each request, and the rotating gateway or a session ID is chosen in the username. In our run, ten requests sent at once to ipinfo.io/ip through httpx came back from 10 different IPs, on Mobile and on Residential Premium alike.
- Mobile: carrier IPs, with the country in the username.
- Residential Premium: home IPs by country, with the country in the username.
- For a task that belongs together (a login, a basket), put a session ID in the proxy username instead of using the rotating gateway.
Plans: mobile proxies, residential proxies and rotating proxies. For whole-site crawls with concurrency built in, see Scrapy vs Beautiful Soup.


