# Async Web Scraping in Python: aiohttp, httpx

> Async web scraping in Python with asyncio, aiohttp and httpx: a semaphore, timeouts, errors, proxies, and 50 pages in 6.0 s against 15.6 s with requests.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / Async Web Scraping in Python: aiohttp, httpx

# Async web scraping in Python: 50 pages with aiohttp and httpx, timed against requests

Guides · Nov 20, 2026 · 5 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![Async web scraping in Python compared with plain requests on the same 50 catalogue pages through a QuanticData proxy: httpx with asyncio and a semaphore of 10 finished in 6.0 seconds, aiohttp in 6.4 seconds, requests one page after another in 15.6 seconds; the card lists the pieces of an async scraper.](https://quanticdata.io/blog/async-web-scraping-python/hero.webp?v=20261120h)

On this page

- [The timings](https://quanticdata.io/blog/async-web-scraping-python/#the-timings)
- [Async scraping with aiohttp, step by step](https://quanticdata.io/blog/async-web-scraping-python/#async-scraping-with-aiohttp-step-by-step)
- [The same scraper with httpx](https://quanticdata.io/blog/async-web-scraping-python/#the-same-scraper-with-httpx)
- [aiohttp or httpx?](https://quanticdata.io/blog/async-web-scraping-python/#aiohttp-or-httpx)
- [How many requests at once?](https://quanticdata.io/blog/async-web-scraping-python/#how-many-requests-at-once)
- [Errors, timeouts and retries](https://quanticdata.io/blog/async-web-scraping-python/#errors-timeouts-and-retries)
- [Async scraping and proxies](https://quanticdata.io/blog/async-web-scraping-python/#async-scraping-and-proxies)

Async web scraping in Python sends many requests and waits for them together instead of one after another. Use `asyncio` with an async HTTP client such as `aiohttp` or `httpx.AsyncClient`, share one client, cap the requests in flight with `asyncio.Semaphore`, and collect results with `asyncio.gather`.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=async-web-scraping-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

To show what that buys, on 20 November 2026 we fetched the same 50 catalogue pages of books.toscrape.com (a public shop made for scraping practice) three ways, with Python 3.12.10, through the same QuanticData proxy, and parsed 1,000 books each time.

## The timings

| Client | Requests in flight | 50 pages |
| --- | --- | --- |
| httpx 0.28.1, AsyncClient | 10 | 6.0 s |
| aiohttp 3.14.3 | 10 | 6.4 s |
| requests 2.34.2, one page after another | 1 | 15.6 s |

Every run returned all 50 pages and 1,000 books with 0 errors. The async clients kept ten downloads waiting at once; the sequential run waited for every page in turn, which is all the difference. (Our Scrapy vs Beautiful Soup comparison was a separate run, with its own numbers.)

## Async scraping with aiohttp, step by step

1. **Open one ClientSession** for the whole run, with a timeout. A session per request throws away connection reuse.
2. **Create a Semaphore** that allows N requests at a time. Without it, gather fires every request at once.
3. **Write one coroutine** that fetches and parses one page inside `async with sem`.
4. **gather** them all, with `return_exceptions=True` so one failure does not cancel the rest.

```python
import asyncio, aiohttp
from bs4 import BeautifulSoup

PROXY = "http://USER:PASS@HOST:PORT"   # your QuanticData proxy line
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]

def titles(html):
    return [a["title"] for a in BeautifulSoup(html, "lxml").select("article.product_pod h3 a")]

async def main(limit=10):
    sem = asyncio.Semaphore(limit)
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as s:
        async def one(url):
            async with sem:
                async with s.get(url, proxy=PROXY) as r:
                    r.raise_for_status()
                    return titles(await r.text())
        return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)

results = asyncio.run(main())
ok = [x for x in results if isinstance(x, list)]
errors = [x for x in results if not isinstance(x, list)]
print(len(ok), sum(map(len, ok)), len(errors))   # 50 1000 0
```

aiohttp takes the proxy per request with `proxy=`; username and password can sit in the URL, as here. All its proxy options, SOCKS5 included, are in [aiohttp proxy setup](https://quanticdata.io/blog/aiohttp-proxy-setup/).

The last line is the check worth keeping in any async scraper: 50 pages came back as lists, 1,000 titles in total, and 0 results were exceptions. When that last number is not zero, the exceptions are in the same list, in the same order as the URLs, so you can see which pages to fetch again.

## The same scraper with httpx

httpx has the requests-style API and an async client; the proxy is set once on the client:

```python
import httpx

async def main(limit=10):
    sem = asyncio.Semaphore(limit)
    async with httpx.AsyncClient(proxy=PROXY, timeout=30) as c:
        async def one(url):
            async with sem:
                r = await c.get(url)
                r.raise_for_status()
                return titles(r.text)
        return await asyncio.gather(*(one(u) for u in URLS), return_exceptions=True)
```

Note `proxy=`, singular: httpx 0.28 removed the old `proxies=` argument. Version details are in [the httpx proxy guide](https://quanticdata.io/blog/httpx-proxy/).

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=async-web-scraping-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## aiohttp or httpx?

|  | aiohttp | httpx |
| --- | --- | --- |
| API | Its own, context managers per response | Like requests |
| Sync mode too | No | Yes, same API |
| Proxy | `proxy=` per request (or on the session) | `proxy=` on the client |
| Our 50 pages | 6.4 s | 6.0 s |

Both did the job; pick httpx if you also want a sync API or already know requests, aiohttp if your code base already uses it. The gap between them is smaller than the gap between either one and sequential requests.

## How many requests at once?

The semaphore value is a choice about the site, not about Python. Ten was plenty for 50 pages. More in flight finishes sooner until the site starts answering [429 Too Many Requests](https://quanticdata.io/blog/429-too-many-requests-web-scraping/) or slowing down, and then everything gets slower. Start low, watch status codes and response times, and raise it gradually. For different sites, keep a semaphore per domain.

## Errors, timeouts and retries

- **Always set a timeout.** One hung request otherwise holds a semaphore slot forever. We used 30 seconds total per request.
- **Use return_exceptions=True** and sort results afterwards, as in the code above: the run finishes, and you retry only the URLs that failed.
- **Retry with a pause**, for example with tenacity around `one()`. The same rules as for sync code apply: [retries and timeouts in requests](https://quanticdata.io/blog/python-requests-retry/).
- **Parse outside the network wait.** Beautiful Soup is synchronous; on 50 pages it is fine inside the coroutine, on very large pages move parsing to a thread with `asyncio.to_thread`.

## Async scraping and proxies

The proxy goes on the client or on each request, and the rotating gateway or a session ID is chosen in the username. In our run, ten requests sent at once to ipinfo.io/ip through httpx came back from 10 different IPs, on Mobile and on Residential Premium alike.

- **Mobile**: carrier IPs, with the country in the username.
- **Residential Premium**: home IPs by country, with the country in the username.
- For a task that belongs together (a login, a basket), put a session ID in the proxy username instead of using the rotating gateway.

Plans: [mobile proxies](https://quanticdata.io/mobile-proxies/), [residential proxies](https://quanticdata.io/residential-proxies/) and [rotating proxies](https://quanticdata.io/rotating-proxies/). For whole-site crawls with concurrency built in, see [Scrapy vs Beautiful Soup](https://quanticdata.io/blog/scrapy-vs-beautifulsoup/).

### Sources & further reading

- [Python docs: asyncio](https://docs.python.org/3/library/asyncio.html)
- [Python docs: asyncio.Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore)
- [aiohttp client quickstart](https://docs.aiohttp.org/en/stable/client_quickstart.html)
- [HTTPX: async support](https://www.python-httpx.org/async/)

## FAQ

Quick answers on async web scraping python.

[Something else? Ask us](mailto:hello@quanticdata.io)

### How do I scrape websites asynchronously in Python?

Use asyncio with aiohttp or httpx.AsyncClient: one shared session, a coroutine per URL wrapped in `async with semaphore`, and `asyncio.gather(*tasks, return_exceptions=True)`. Our version fetched 50 pages in 6.0 to 6.4 s.

### Is async scraping faster than requests?

For many pages, yes, because the waits overlap. Through the same proxy, httpx async fetched 50 pages in 6.0 s and requests, one page at a time, in 15.6 s. For a single page there is no gain.

### Should I use aiohttp or httpx for async scraping?

Both work. httpx has a requests-like API and a sync mode; aiohttp is async only. In our run httpx took 6.0 s and aiohttp 6.4 s for the same 50 pages.

### How many concurrent requests should an async scraper send?

Start low, around 5 to 10 per site, and raise it while status codes stay 200 and response times stay flat. Cap it with `asyncio.Semaphore`; 10 was enough for our 50 pages.

### How do I use proxies with async scraping?

Pass `proxy=` per request in aiohttp or on the AsyncClient in httpx. In our run, ten requests sent at once came back from 10 different IPs.

## Rotating proxies for aiohttp and httpx

Mobile and residential proxies for aiohttp and httpx, with rotation and session settings in the username.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=async-web-scraping-python) · [Explore Residential Proxies from $0.80/GB](https://quanticdata.io/residential-proxies/)

## Collectors with Python examples

- [eBay scraper API in Python](https://quanticdata.io/collectors/ebay-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the eBay scraper collector, with Python examples.
- [Google Images scraper API in Python](https://quanticdata.io/collectors/image-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Google Images scraper collector, with Python examples.
- [Video search API in Python](https://quanticdata.io/collectors/search-videos-api/python/): Collector · Install, first call, async runs, output fields and errors of the Video search collector, with Python examples.
- [Scrape YouTube videos in Python](https://quanticdata.io/collectors/youtube-scraper-api/python/): Collector · Install, first call, async runs, output fields and errors of the Scrape YouTube videos collector, with Python examples.

## Related reading

- [urllib3 vs requests: Speed and Bandwidth](https://quanticdata.io/blog/urllib3-vs-requests/): Guides · requests is built on urllib3 and adds sessions, JSON, exceptions on errors and compression by default. urllib3 alone was faster on a local server, the same through a proxy, and downloaded 9.5 times more bytes until we asked for compression. Measured, with the code.
- [Python requests POST: JSON, Form Data, Files](https://quanticdata.io/blog/python-requests-post/): Guides · requests.post(url, json=payload) sends a JSON body with the right Content-Type; data= sends form fields; files= uploads. We posted each one through a residential proxy to a test server and print what the server received, including the redirect that silently turns a POST into a GET.
- [Concurrency vs Parallelism, Measured in Python](https://quanticdata.io/blog/concurrency-vs-parallelism/): Guides · Concurrency is making progress on several tasks by switching between them; parallelism is running several at the same instant on different CPU cores. We timed both on a real scraping job in Python on 20 November 2026: threads cut 30 downloads from 10.3 s to 3.5 s, and only processes cut the parsing, from 3.5 s to 0.8 s.

---

Source: https://quanticdata.io/blog/async-web-scraping-python/ · Site index for AI: https://quanticdata.io/llms.txt
