How to Fix 429 Too Many Requests When Scraping

A 429 Too Many Requests is a rate limit: the server counted your requests against a key and you crossed the threshold. It is the most honest block there is, because it usually tells you how long to wait and rarely blacklists you. The fix is not simply “add proxies”. It is to find what the limit is keyed on, honour Retry-After, and size your concurrency so each key stays under its threshold.

What a 429 tells you that a 403 does not

RFC 6585 introduced 429 for exactly one purpose: “the user has sent too many requests in a given amount of time”. Unlike a 403, it is not a judgement on what you are; it is a counter that overflowed. Three headers can come with it and each is worth parsing:

• Retry-After: seconds to wait, or an HTTP date. Honouring it is the single highest-return change you can make, and ignoring it is how a temporary limit becomes a permanent ban.
• RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset: the IETF draft that many APIs and some sites already send. Reddit's own pages return x-ratelimit-remaining and x-ratelimit-reset on every response, so you can throttle before the 429 ever appears.
• A vendor body: Cloudflare's rate limiting arrives as a 429 with error 1015 in the HTML. That one is configured per site by its owner, typically per IP over a short window.

What the response does not tell you is the key: whether the counter is per IP, per session cookie, per API token, per account, or per fingerprint. Everything else depends on that.

Find the key before you add proxies

Two requests answer it. Get yourself rate-limited, then:

1. Same IP, fresh session. Drop cookies and local storage, keep the IP. If the 429 disappears, the limit is keyed on the session or a token. Proxies are irrelevant; you need more sessions, or fewer requests per session.
2. New IP, same session. Keep the cookies, change the exit. If the 429 disappears, the limit is keyed on the IP. This is the case where rotation works, and the math below applies.

If neither changes anything, the key is your account or your fingerprint, and the only honest fix is to slow down to the documented rate. If both change it, the site keys on the pair, and you need to rotate sessions and IPs together, keeping each session on one sticky IP so the pair looks like a person. Rotating proxies with sticky sessions of up to 120 minutes exist for that shape.

Honour Retry-After and back off properly

Most 429 loops are self-inflicted: a retry decorator that fires immediately, in parallel, from every worker, which is precisely the pattern that triggered the limit. The rules are simple. Read Retry-After if present and sleep exactly that long. Otherwise back off exponentially with jitter, so thirty workers do not all retry in the same second. Cap the retries; a fifth consecutive 429 from one key is a signal to stop, not to try harder.

import random, time, requests
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after_seconds(resp, attempt):
    ra = resp.headers.get("Retry-After")
    if ra:
        if ra.isdigit():
            return int(ra)
        return max(0, (parsedate_to_datetime(ra) - datetime.now(timezone.utc)).total_seconds())
    return min(60, (2 ** attempt) + random.uniform(0, 1))   # exponential + jitter, capped

def get(session, url, max_attempts=5):
    for attempt in range(max_attempts):
        r = session.get(url, timeout=30)
        if r.status_code != 429:
            return r
        time.sleep(retry_after_seconds(r, attempt))
    raise RuntimeError(f"still 429 after {max_attempts} attempts: {url}")

If you use urllib3.Retry through a requests adapter, set status_forcelist=[429] and leave respect_retry_after_header=True, its default; the adapter then does the arithmetic above for you. In Scrapy, add 429 to RETRY_HTTP_CODES and turn on AUTOTHROTTLE_ENABLED, which adjusts the delay from observed latency, with AUTOTHROTTLE_TARGET_CONCURRENCY as the knob.

The concurrency math

When the key is the IP, the number of exits you need is arithmetic, not intuition:

IPs in flight  =  pages per minute you need  ÷  requests per minute one IP tolerates

Say you need 60,000 pages an hour, which is 1,000 a minute, and the target tolerates about 10 requests a minute per IP before answering 429. You need 100 IPs in flight at any moment. Notice what is not in the formula: the total number of pages, the size of the proxy pool, or the number of distinct IPs you will see over the month. Those matter for cost and for reputation, not for the 429. A rotating gateway gives you a fresh exit per request from one endpoint, so “100 IPs in flight” is simply 100 concurrent connections, and the pool underneath takes care of variety.

The tolerance figure is the part you have to measure. Ramp one IP up until the first 429, note the rate, then run at 60 to 70 percent of it. Sites often tolerate short bursts and punish sustained rates, so measure over ten minutes rather than ten seconds. And remember that the limit is per key, so if the site keys on session plus IP, a single sticky session gets its own budget and rotating the IP under it resets nothing.

Make it structural: a token bucket per host

Retries handle the failure; a per-host limiter prevents it. The bucket refills at the tolerated rate, every request takes a token, and workers wait when it is empty. This is what keeps thirty async workers from behaving like one very fast client:

import asyncio, time

class HostBucket:
    def __init__(self, rate_per_min, burst=None):
        self.rate = rate_per_min / 60.0
        self.capacity = burst or max(1, rate_per_min // 6)
        self.tokens = self.capacity
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def take(self):
        async with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                await asyncio.sleep((1 - self.tokens) / self.rate)

buckets = {}   # host -> HostBucket, e.g. buckets["example.com"] = HostBucket(rate_per_min=8)

Combine it with the retry logic and a 429 becomes rare and cheap: the bucket keeps you under the line, and when a site changes its line the retry sleeps for exactly the time it asks. If you also rotate exits, give each exit its own bucket, because that is how the server counts.

When the 429 is not about pace

Cloudflare's error 1015 is a real rate limit, but it is set by the site owner and it is often absurdly low for a datacenter IP and generous for a residential one, because the rule may be scoped by IP reputation. If a residential exit gets ten times the budget of a cloud IP on the same site, that is not a bug in your throttle, it is the policy. A 429 that arrives on the very first request from a fresh IP is either a shared IP that another tenant already exhausted, or a fingerprint rule wearing a 429 costume; treat it like a 403 and run that guide's decision tree.

And there is a point where the whole apparatus, buckets, retries, sticky sessions, exit accounting, is not the thing you wanted to build. The web scraping API and the SERP API run it internally and bill per successful page, so a request that ends in a 429 at every attempt costs nothing. Whether that trade is worth it depends on how many hosts you scrape and how often they change their line; for one or two stable targets, the code above is enough.