# How to Fix 429 Too Many Requests When Scraping

> A 429 is a rate limit with a key: IP, session or account. Find the key before adding proxies, honour Retry-After, and size concurrency with a formula.

[Home](https://quanticdata.io/)/[Blog](https://quanticdata.io/blog/)/How to Fix 429 Too Many Requests When Scraping

# How to Fix 429 Too Many Requests When Scraping

TroubleshootingSep 3, 2026·7 min read·By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

On this page [What a 429 tells you that a 403 does not](/blog/429-too-many-requests-web-scraping/#what-a-429-tells-you-that-a-403-does-not) [Find the key before you add proxies](/blog/429-too-many-requests-web-scraping/#find-the-key-before-you-add-proxies) [Honour Retry-After and back off properly](/blog/429-too-many-requests-web-scraping/#honour-retry-after-and-back-off-properly) [The concurrency math](/blog/429-too-many-requests-web-scraping/#the-concurrency-math) [Make it structural: a token bucket per host](/blog/429-too-many-requests-web-scraping/#make-it-structural-a-token-bucket-per-host) [When the 429 is not about pace](/blog/429-too-many-requests-web-scraping/#when-the-429-is-not-about-pace)

A 429 Too Many Requests is a rate limit: the server counted your requests against a key and you crossed the threshold. It is the most honest block there is, because it usually tells you how long to wait and rarely blacklists you. The fix is not simply “add proxies”. It is to find what the limit is keyed on, honour `Retry-After`, and size your concurrency so each key stays under its threshold.

## What a 429 tells you that a 403 does not

RFC 6585 introduced 429 for exactly one purpose: “the user has sent too many requests in a given amount of time”. Unlike a [403](https://quanticdata.io/blog/web-scraping-403-forbidden/), it is not a judgement on what you are; it is a counter that overflowed. Three headers can come with it and each is worth parsing:

- **`Retry-After`:** seconds to wait, or an HTTP date. Honouring it is the single highest-return change you can make, and ignoring it is how a temporary limit becomes a permanent ban.

- **`RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`:** the IETF draft that many APIs and some sites already send. Reddit's own pages return `x-ratelimit-remaining` and `x-ratelimit-reset` on every response, so you can throttle before the 429 ever appears.

- **A vendor body:** Cloudflare's rate limiting arrives as a 429 with error 1015 in the HTML. That one is configured per site by its owner, typically per IP over a short window.

What the response does not tell you is the *key*: whether the counter is per IP, per session cookie, per API token, per account, or per fingerprint. Everything else depends on that.

## Find the key before you add proxies

Two requests answer it. Get yourself rate-limited, then:

1. **Same IP, fresh session.** Drop cookies and local storage, keep the IP. If the 429 disappears, the limit is keyed on the session or a token. Proxies are irrelevant; you need more sessions, or fewer requests per session.

2. **New IP, same session.** Keep the cookies, change the exit. If the 429 disappears, the limit is keyed on the IP. This is the case where rotation works, and the math below applies.

If neither changes anything, the key is your account or your fingerprint, and the only honest fix is to slow down to the documented rate. If both change it, the site keys on the pair, and you need to rotate sessions and IPs together, keeping each session on one sticky IP so the pair looks like a person. [Rotating proxies](https://quanticdata.io/rotating-proxies/) with sticky sessions of up to 120 minutes exist for that shape.

## Honour Retry-After and back off properly

Most 429 loops are self-inflicted: a retry decorator that fires immediately, in parallel, from every worker, which is precisely the pattern that triggered the limit. The rules are simple. Read `Retry-After` if present and sleep exactly that long. Otherwise back off exponentially with jitter, so thirty workers do not all retry in the same second. Cap the retries; a fifth consecutive 429 from one key is a signal to stop, not to try harder.

```
import random, time, requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(resp, attempt):
    ra = resp.headers.get("Retry-After")
    if ra:
        if ra.isdigit():
            return int(ra)
        return max(0, (parsedate_to_datetime(ra) - datetime.now(timezone.utc)).total_seconds())
    return min(60, (2 ** attempt) + random.uniform(0, 1))   # exponential + jitter, capped

def get(session, url, max_attempts=5):
    for attempt in range(max_attempts):
        r = session.get(url, timeout=30)
        if r.status_code != 429:
            return r
        time.sleep(retry_after_seconds(r, attempt))
    raise RuntimeError(f"still 429 after {max_attempts} attempts: {url}")
```

If you use `urllib3.Retry` through a requests adapter, set `status_forcelist=[429]` and leave `respect_retry_after_header=True`, its default; the adapter then does the arithmetic above for you. In Scrapy, add 429 to `RETRY_HTTP_CODES` and turn on `AUTOTHROTTLE_ENABLED`, which adjusts the delay from observed latency, with `AUTOTHROTTLE_TARGET_CONCURRENCY` as the knob.

## The concurrency math

When the key is the IP, the number of exits you need is arithmetic, not intuition:

```
IPs in flight  =  pages per minute you need  ÷  requests per minute one IP tolerates
```

Say you need 60,000 pages an hour, which is 1,000 a minute, and the target tolerates about 10 requests a minute per IP before answering 429. You need 100 IPs in flight at any moment. Notice what is not in the formula: the total number of pages, the size of the proxy pool, or the number of distinct IPs you will see over the month. Those matter for cost and for reputation, not for the 429. A rotating gateway gives you a fresh exit per request from one endpoint, so “100 IPs in flight” is simply 100 concurrent connections, and the pool underneath takes care of variety.

The tolerance figure is the part you have to measure. Ramp one IP up until the first 429, note the rate, then run at 60 to 70 percent of it. Sites often tolerate short bursts and punish sustained rates, so measure over ten minutes rather than ten seconds. And remember that the limit is per key, so if the site keys on session plus IP, a single sticky session gets its own budget and rotating the IP under it resets nothing.

## Make it structural: a token bucket per host

Retries handle the failure; a per-host limiter prevents it. The bucket refills at the tolerated rate, every request takes a token, and workers wait when it is empty. This is what keeps thirty async workers from behaving like one very fast client:

```
import asyncio, time

class HostBucket:
    def __init__(self, rate_per_min, burst=None):
        self.rate = rate_per_min / 60.0
        self.capacity = burst or max(1, rate_per_min // 6)
        self.tokens = self.capacity
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def take(self):
        async with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

buckets = {}   # host -> HostBucket, e.g. buckets["example.com"] = HostBucket(rate_per_min=8)
```

Combine it with the retry logic and a 429 becomes rare and cheap: the bucket keeps you under the line, and when a site changes its line the retry sleeps for exactly the time it asks. If you also rotate exits, give each exit its own bucket, because that is how the server counts.

## When the 429 is not about pace

Cloudflare's error 1015 is a real rate limit, but it is set by the site owner and it is often absurdly low for a datacenter IP and generous for a residential one, because the rule may be scoped by IP reputation. If a residential exit gets ten times the budget of a cloud IP on the same site, that is not a bug in your throttle, it is the policy. A 429 that arrives on the very first request from a fresh IP is either a shared IP that another tenant already exhausted, or a fingerprint rule wearing a 429 costume; treat it like a 403 and run that guide's decision tree.

And there is a point where the whole apparatus, buckets, retries, sticky sessions, exit accounting, is not the thing you wanted to build. The [web scraping API](https://quanticdata.io/web-scraping-api/) and the [SERP API](https://quanticdata.io/serp-api/) run it internally and bill per successful page, so a request that ends in a 429 at every attempt costs nothing. Whether that trade is worth it depends on how many hosts you scrape and how often they change their line; for one or two stable targets, the code above is enough.

### Sources & further reading

- [RFC 6585 — Additional HTTP Status Codes, §4 429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585.html#section-4)

- [MDN — 429 Too Many Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429)

- [IETF draft — RateLimit header fields for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/)

- [Cloudflare — Error 1015: You are being rate limited](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1015/)

- [urllib3 — Retry, respect_retry_after_header](https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry)

- [Scrapy documentation — AutoThrottle extension](https://docs.scrapy.org/en/latest/topics/autothrottle.html)

## FAQ

Quick answers on 429 too many requests web scraping.

[Something else? Ask us →](mailto:hello@quanticdata.io)

### How do I fix a 429 Too Many Requests error when web scraping?

Honour the Retry-After header if present, back off exponentially with jitter if not, and cap the retries. Then find what the limit is keyed on: if a fresh session on the same IP clears it, the key is the session; if a new IP with the same cookies clears it, the key is the IP and rotating exits helps. Finally add a per-host token bucket so your workers stay under the measured rate instead of triggering the limit again.

### Do proxies fix 429 errors?

Only when the limit is keyed on the IP address. If the site counts requests per session cookie, per token or per account, changing the IP resets nothing. Test it with two requests: same IP with fresh cookies, and new IP with the same cookies. Rotate exits only if the second test clears the 429.

### How many IPs do I need to avoid 429?

Divide the pages per minute you need by the requests per minute one IP tolerates before the site answers 429. For 1,000 pages a minute against a site that tolerates 10 per IP, you need 100 IPs in flight. Measure the tolerance over ten minutes and run at 60 to 70 percent of it.

### What does Retry-After mean and should I respect it?

It is the server telling you how many seconds to wait, or the date after which to retry. Respect it exactly. It is the cheapest fix available, and ignoring it is how a temporary rate limit turns into a permanent block. Python requests exposes it as resp.headers["Retry-After"]; urllib3.Retry honours it by default.

### What is Cloudflare error 1015?

Cloudflare's rate-limiting response, delivered as a 429 with error 1015 in the body. The site owner configures the threshold and window, and rules can be scoped by IP reputation, so a datacenter IP may hit it at a rate a residential IP never sees. Back off, then reduce the per-IP rate or spread the load across residential exits.

### What is the difference between 429 and 403?

A 429 is a counter that overflowed: you sent too many requests against some key in some window, and it usually clears by itself. A 403 is a judgement that your request looks automated or is not permitted, and it does not clear with time. A 429 on the very first request from a fresh IP is usually a 403 in disguise and should be treated as one.

## IPs in flight without a pool to manage

Rotating proxies from $0.50/GB give a fresh exit on every request from one endpoint, with sticky sessions up to 120 minutes for limits keyed on session plus IP. Or let the web scraping API handle pacing and retries and pay $0.0002 per successful page, with $2 of free usage every month.

[Start free — $2/month included](https://quanticdata.io/signup/)[Explore Rotating Proxies from $0.50/GB](https://quanticdata.io/rotating-proxies/)

## Related reading

[Troubleshooting How to Fix 407 Proxy Authentication Required A 407 is the proxy refusing to forward your request until it sees credentials it accepts, so rotating headers or slowing down cannot fix it. On HTTPS it does not even arrive as a status code. How to read the challenge, tell the five causes apart, and fix each one in curl, Python requests, httpx, Node and Scrapy. Read →](https://quanticdata.io/blog/how-to-fix-407-proxy-authentication-required/) [Troubleshooting Web Scraping 403 Forbidden: Causes and Fixes The browser loads the page and your script gets 403. The response headers usually name the system that refused you, and that decides the fix: a user agent, the full header set, the IP type, or the TLS fingerprint. A decision tree you can run in five minutes, Python fixes for each branch, and what we saw fetching 16 major sites with a pure HTTP client behind US residential IPs. Read →](https://quanticdata.io/blog/web-scraping-403-forbidden/) [Troubleshooting Proxy Not Working? A 10-Step Checklist “Proxy not working” is four different problems wearing one name: you cannot reach the proxy, the proxy refuses you, the proxy reaches the site but the site refuses it, or it works and is slow. Each layer has its own error strings and its own fix. A table that maps the message to the layer, curl timings that separate slow from broken, and a 10-step checklist in the order that saves the most time. Read →](https://quanticdata.io/blog/proxy-not-working-checklist/)

## Also on this site

Quantic**Data**

Residential proxies & web data APIs for AI.

#### Proxies

- [Residential Basic](https://quanticdata.io/residential-proxies/#basic)

- [Residential Premium](https://quanticdata.io/residential-proxies/#plans)

- [Cheap Residential](https://quanticdata.io/cheap-residential-proxies/)

- [Mobile Proxies](https://quanticdata.io/mobile-proxies/)

- [Datacenter Proxies](https://quanticdata.io/datacenter-proxies/)

- [ISP Proxies](https://quanticdata.io/isp-proxies/)

- [Rotating Proxies](https://quanticdata.io/rotating-proxies/)

- [Sneaker Proxies](https://quanticdata.io/sneaker-proxies/)

- [SOCKS5 Proxies](https://quanticdata.io/socks5-proxies/)

- [IPv6 Proxies](https://quanticdata.io/ipv6-proxies/)

- [Proxy locations](https://quanticdata.io/proxies/)

#### Data APIs

- [MCP Server](https://quanticdata.io/mcp-server/)

- [Web Scraper API](https://quanticdata.io/web-scraping-api/)

- [SERP API](https://quanticdata.io/serp-api/)

- [Collectors](https://quanticdata.io/collectors/)

- [Web Data for AI](https://quanticdata.io/web-data-api-for-ai/)

- [Quantic AI](https://quanticdata.io/ai-web-scraping-service/)

- [Crawl & Map](https://quanticdata.io/crawl-map/)

- [SEO Audit](https://quanticdata.io/seo-audit/)

#### Use cases

- [Company data](https://quanticdata.io/scrape-company-data/)

- [Price monitoring](https://quanticdata.io/competitor-price-monitoring/)

- [Market research](https://quanticdata.io/market-research-data/)

- [Real estate data](https://quanticdata.io/real-estate-data-scraping/)

- [Scrape job postings](https://quanticdata.io/scrape-job-postings/)

#### Company

- [Documentation](https://quanticdata.io/docs/)

- [Blog](https://quanticdata.io/blog/)

- [Free tools](https://quanticdata.io/tools/)

- [Partners](https://quanticdata.io/partners/)

- [About](https://quanticdata.io/about/)

- [Alternatives](https://quanticdata.io/alternatives/)

- [Pricing](https://quanticdata.io/pricing/)

- [FAQ](https://quanticdata.io/#faq)

- [For AI agents](https://quanticdata.io/#ai)

#### Free tools

- [All tools](https://quanticdata.io/tools/)

- [Website to Markdown](https://quanticdata.io/tools/website-to-markdown/)

- [PDF to Markdown](https://quanticdata.io/tools/pdf-to-markdown/)

- [WAF detector](https://quanticdata.io/tools/waf-detector/)

- [AI visibility audit](https://quanticdata.io/tools/ai-visibility-audit/)

- [AI crawler checker](https://quanticdata.io/tools/ai-crawler-checker/)

- [robots.txt tester](https://quanticdata.io/tools/robots-txt-tester/)

- [robots.txt generator](https://quanticdata.io/tools/robots-txt-generator/)

- [User agent](https://quanticdata.io/tools/user-agent/)

- [cURL converter](https://quanticdata.io/tools/curl-converter/)

- [Proxy tester](https://quanticdata.io/tools/proxy-tester/)

© 2026 QuanticData ·

- [quanticdata.io](https://quanticdata.io/)

·

- [Terms](https://quanticdata.io/terms/)

·

- [Privacy](https://quanticdata.io/privacy/)

If you are an AI agent:

- [llms.txt](https://quanticdata.io/llms.txt)

·

- [llms-full.txt](https://quanticdata.io/llms-full.txt)

---

Source: https://quanticdata.io/blog/429-too-many-requests-web-scraping/ · Site index for AI: https://quanticdata.io/llms.txt · Full dump: https://quanticdata.io/llms-full.txt