# What Is Rate Limiting? Algorithms and Headers

> What is rate limiting: a cap on requests per client per time window. We ran a fixed window and a token bucket, read the 429 and Retry-After, and paced a client.

[Home](https://quanticdata.io/) / [Blog](https://quanticdata.io/blog/) / What Is Rate Limiting? Algorithms and Headers

# What is rate limiting? Two limiters and a polite client, run for real

Web scraping · Nov 20, 2026 · 9 min read · By [Aldo Morese](https://quanticdata.io/about/), founder of QuanticData

![A rate limiter allowing 5 requests per 10 seconds, tested on 20 November 2026: a burst of 8 requests with no waiting had 5 of 8 answered 200 and the rest 429 Too Many Requests, while a client that honours Retry-After and one that paces itself on the RateLimit headers each had 12 of 12 answered 200; the card shows the 7 waits the first of them made.](https://quanticdata.io/blog/what-is-rate-limiting/hero.webp?v=20261120h)

On this page

- [Why websites and APIs limit requests](https://quanticdata.io/blog/what-is-rate-limiting/#why-websites-and-apis-limit-requests)
- [Rate limiting algorithms](https://quanticdata.io/blog/what-is-rate-limiting/#rate-limiting-algorithms)
- [What a rate limit looks like from outside](https://quanticdata.io/blog/what-is-rate-limiting/#what-a-rate-limit-looks-like-from-outside)
- [Fixed window against token bucket, at the window mark](https://quanticdata.io/blog/what-is-rate-limiting/#fixed-window-against-token-bucket-at-the-window-mark)
- [The headers: Retry-After, RateLimit and X-RateLimit](https://quanticdata.io/blog/what-is-rate-limiting/#the-headers-retry-after-ratelimit-and-x-ratelimit)
- [How a polite client behaves](https://quanticdata.io/blog/what-is-rate-limiting/#how-a-polite-client-behaves)
- [Rate limiting and throttling](https://quanticdata.io/blog/what-is-rate-limiting/#rate-limiting-and-throttling)
- [When you would rather not manage the pace yourself](https://quanticdata.io/blog/what-is-rate-limiting/#when-you-would-rather-not-manage-the-pace-yourself)

Rate limiting is a rule a server applies to each client: no more than a set number of requests in a set period, for example 5 per 10 seconds or 60 per hour. Requests inside the limit are served normally. Requests over it are refused, usually with the HTTP status 429 Too Many Requests and a header saying when to try again.

**Run it on the free credit.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-lead&utm_content=what-is-rate-limiting) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

Sites and APIs use it to keep one client from crowding out the others. To show how it behaves we wrote two small limiters and a client on 20 November 2026 and ran them on one PC; we also read the rate limit headers of a large public API. The code and the logs are below.

## Why websites and APIs limit requests

- **Fair shares.** A server has a finite capacity. A limit per client keeps one busy script from slowing the site for everyone else.
- **Cost.** Every request costs the operator compute and bandwidth, and some endpoints (search, exports, AI models) cost far more than others.
- **Protection.** A cap on login attempts or password resets blunts guessing attacks; a cap on everything blunts floods.
- **Plans.** APIs sell tiers: so many requests per minute on one plan, more on the next.

The limit is counted per "client", and the server decides what that means: an API key, a logged-in account, an IP address, or a combination. It is often enforced in a gateway or [reverse proxy](https://quanticdata.io/blog/forward-proxy-vs-reverse-proxy/) in front of the application.

## Rate limiting algorithms

| Algorithm | How it counts | Behaviour |
| --- | --- | --- |
| Fixed window | A counter per clock window (each minute, each 10 seconds), reset at the mark | Simplest. Allows a double burst around the mark: 10 of 10 accepted in our test |
| Sliding window | Requests in the last N seconds, measured from now | No double burst; needs a timestamp log or a weighted pair of counters |
| Token bucket | A bucket of tokens refilled at a steady rate; a request spends one | Allows a burst up to the bucket size, then the refill rate: 5 of 10 accepted in the same test |
| Leaky bucket | A queue drained at a steady rate | Smooths output to a constant pace; excess waits or is dropped |

We implemented the first and the third. Both allow 5 requests per 10 seconds per client IP:

```python
import math, time
LIMIT, WINDOW = 5, 10            # 5 requests per 10 seconds
RATE = LIMIT / WINDOW            # token bucket refill: 0.5 tokens per second
windows, buckets = {}, {}

def fixed_window(key, now):
    start = now - now % WINDOW                      # windows start at 0, 10, 20 ... seconds
    count = windows.get((key, start), 0)
    reset = math.ceil(start + WINDOW - now)
    if count >= LIMIT:
        return False, 0, reset
    windows[(key, start)] = count + 1
    return True, LIMIT - count - 1, reset

def token_bucket(key, now):
    tokens, last = buckets.get(key, (LIMIT, now))
    tokens = min(LIMIT, tokens + (now - last) * RATE)   # refill for the time that passed
    if tokens < 1:
        buckets[key] = (tokens, now)
        return False, 0, math.ceil((1 - tokens) / RATE)
    buckets[key] = (tokens - 1, now)
    return True, int(tokens - 1), math.ceil((1 - (tokens - 1) % 1) / RATE)
```

Each function returns three things: allowed or not, how many requests remain, and in how many seconds the client may try again. A small Python `http.server` handler calls one of them per request (path `/fixed` or `/bucket`) and turns the answer into a status and headers.

## What a rate limit looks like from outside

Eight requests sent with no pause, to each limiter:

```
/fixed   8 requests at once -> 200 200 200 200 200 429 429 429
/bucket  8 requests at once -> 200 200 200 200 200 429 429 429
```

Five were served and three refused, on both. This is one of the refusals in full:

```bash
curl -i http://127.0.0.1:8429/bucket
HTTP/1.0 429 Too Many Requests
Server: BaseHTTP/0.6 Python/3.12.10
Date: Fri, 20 Nov 2026 18:33:20 GMT
RateLimit-Limit: 5
RateLimit-Remaining: 0
RateLimit-Reset: 2
Retry-After: 2
Content-Length: 18

Too Many Requests
```

The status says what happened, and the headers say what to do. `Retry-After: 2` is the instruction: wait two seconds. The three `RateLimit` lines give the context: the limit, what is left of it, and when it refills.

## Fixed window against token bucket, at the window mark

The two limiters gave the same answer to a single burst. They differ when a burst straddles the moment the fixed window resets. We sent 5 requests just before a 10-second mark and 5 just after it:

```
/fixed   before the mark: 200 200 200 200 200 | after: 200 200 200 200 200 | accepted 10 of 10 in about a second
/bucket  before the mark: 200 200 200 200 200 | after: 429 429 429 429 429 | accepted 5 of 10 in about a second
```

The fixed window served all ten within about a second, twice the rate its owner intended, because each half fell in a different window. The token bucket served five and made the rest wait for the refill. This is why operators who care about bursts choose a token bucket or a sliding window, and it is also why the same nominal limit can feel different from one service to another.

**Try it on your own targets.** Every QuanticData account gets $2 of free API usage per month.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-inline&utm_content=what-is-rate-limiting) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## The headers: Retry-After, RateLimit and X-RateLimit

| Header | Meaning | Seen in our tests |
| --- | --- | --- |
| `Retry-After` | Seconds to wait (or a date) before the next request. Standard HTTP, defined in RFC 9110 | `2` on our 429 |
| `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset` | The quota, what is left, and seconds until it resets. From an IETF draft, in wide use | 5, 0 and 2 on our 429 |
| `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` | The older, unofficial spelling of the same three; the reset is often a Unix time | GitHub's API, below |

A real example. We made two unauthenticated requests to GitHub's public REST API from this PC and kept the rate limit headers:

```bash
curl -s -i https://api.github.com/rate_limit
HTTP/1.1 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 47
X-RateLimit-Used: 13
X-RateLimit-Resource: core
X-RateLimit-Reset: 1795202051

curl -s -i https://api.github.com/zen
HTTP/1.1 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 46
X-RateLimit-Used: 14
X-RateLimit-Resource: core
X-RateLimit-Reset: 1795202051
```

The limit is 60 requests per hour for a client without a token, and the count went from 47 to 46 remaining between the two calls. The reset is a Unix timestamp, the moment the hour's window ends. The headers arrive on successful answers too, so a client can see the limit coming long before it reaches it. Header names in general are covered in [HTTP headers explained](https://quanticdata.io/blog/http-headers-explained/).

## How a polite client behaves

A rate limit is the operator telling you the pace the service accepts. The correct response is to keep to it. There are two ways, and we ran both against the token bucket.

### React: honour Retry-After

```python
import random, time, requests

def polite_get(url, max_attempts=5):
    """Honour Retry-After; without it, exponential backoff with jitter; give up after max_attempts."""
    for attempt in range(max_attempts):
        r = requests.get(url, timeout=10)
        if r.status_code != 429:
            return r, attempt
        wait = r.headers.get("Retry-After")
        wait = int(wait) if wait and wait.isdigit() else min(60, 2 ** attempt + random.uniform(0, 1))
        time.sleep(wait)
    raise RuntimeError(f"still 429 after {max_attempts} attempts, stopping")
```

```
  0.0s request  1: 200  remaining 4
  0.0s request  2: 200  remaining 3
  0.0s request  3: 200  remaining 2
  0.1s request  4: 200  remaining 1
  0.1s request  5: 200  remaining 0
  0.1s   429, Retry-After: 2 -> sleeping 2 s
  2.1s request  6: 200  remaining 0
  2.1s   429, Retry-After: 2 -> sleeping 2 s
  4.1s request  7: 200  remaining 0
  ...
 14.3s request 12: 200  remaining 0
 14.3s done: 12 of 12 answered 200, 7 waits
```

All 12 requests were answered 200. The first five used the bucket; after that the client met a 429, slept the two seconds it was told, and got its next request through, seven times. When a server sends no `Retry-After`, the function falls back to exponential backoff with jitter: wait about 1, 2, 4, 8 seconds, each with a random fraction added so that many clients do not all return at the same instant. And it stops after five attempts instead of hammering.

### Better: pace yourself and never see a 429

```
def paced(path, n=12):
    """Never hit the limit: when RateLimit-Remaining reaches 0, wait RateLimit-Reset seconds first."""
    codes = []
    for i in range(n):
        r = requests.get(BASE + path, timeout=10)
        codes.append(r.status_code)
        if r.headers["RateLimit-Remaining"] == "0":
            time.sleep(int(r.headers["RateLimit-Reset"]))
    print(f"paced: {codes.count(200)} of {n} answered 200, {codes.count(429)} answered 429")
```

```
 16.1s paced: 12 of 12 answered 200, 0 answered 429
```

Same 12 requests, no refusal at all: the client read `RateLimit-Remaining` on every answer and waited before the server had to say no.

| Client | Answered 200 | Answered 429 | Finished in |
| --- | --- | --- | --- |
| Burst of 8, no waiting | 5 of 8 | 3 | Under a second, with 3 requests lost |
| Honours `Retry-After` | 12 of 12 | 7, each followed by the wait it asked for | 14.3 s |
| Paces on `RateLimit-Remaining` | 12 of 12 | 0 | 16.1 s |

The rest of good behaviour follows from the same idea:

- **Cap your concurrency.** Ten workers multiply your request rate by ten; see [concurrency vs parallelism](https://quanticdata.io/blog/concurrency-vs-parallelism/).
- **Cache.** A page you already have costs the site nothing the second time.
- **Retry only what is temporary.** Timeouts and 502, 503, 504 deserve a bounded retry ([retries and timeouts in Python Requests](https://quanticdata.io/blog/python-requests-retry/)). A 429 deserves a wait. A 403 is a refusal, not a queue: stop and check the site's terms.
- **Use the official route for volume.** If you need more than the public limit, the operator usually offers a key, a paid tier or a bulk export.

The same list from the crawler's side is in [crawling a website politely](https://quanticdata.io/blog/crawl-a-website-without-getting-blocked/); what the status looks like in a scraper's logs is in [429 Too Many Requests when scraping](https://quanticdata.io/blog/429-too-many-requests-web-scraping/).

## Rate limiting and throttling

The two words are often used as one. Where they are told apart, rate limiting refuses the excess (the 429 above) and throttling slows it down: the server queues or delays requests so that they complete later instead of failing. A leaky bucket that makes requests wait is throttling; our token bucket, which refused them, is rate limiting. From the client's side the advice is the same: read the headers and slow down.

## When you would rather not manage the pace yourself

Pacing, waiting and caching are work, and they are the same work for every site you read. Our [Web Scraping API](https://quanticdata.io/web-scraping-api/) takes a URL and returns the page, and our [collectors](https://quanticdata.io/collectors/) return structured rows for common public sources; both are called with one API key, and a client written like the two above is all the calling code they need. The status codes you can meet when a request travels through a proxy are listed in [proxy error codes](https://quanticdata.io/blog/proxy-error-codes/).

### Sources & further reading

- [RFC 6585: Additional HTTP Status Codes, 429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585#section-4)
- [RFC 9110: HTTP Semantics, Retry-After](https://www.rfc-editor.org/rfc/rfc9110#name-retry-after)
- [IETF draft: RateLimit header fields for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/)
- [GitHub Docs: Rate limits for the REST API](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api)
- [MDN: 429 Too Many Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429)

## FAQ

Quick answers on what is rate limiting.

[Something else? Ask us](mailto:hello@quanticdata.io)

### What is rate limiting?

A rule that caps how many requests one client may send to a server in a period of time. Requests over the cap are refused, normally with HTTP 429 Too Many Requests. In our test a limit of 5 per 10 seconds served 5 of a burst of 8 and refused 3.

### What is rate limiting in an API?

The same rule counted per API key or per client: for example GitHub's public REST API answered our unauthenticated requests with `X-RateLimit-Limit: 60`, a quota of 60 requests per hour, and counted down the remaining ones on every response.

### What does 429 Too Many Requests mean?

That the client sent more requests than the server allows in the current period. It is temporary. The response often carries `Retry-After` with the number of seconds to wait; ours said 2.

### What is the Retry-After header?

A standard HTTP response header that tells the client how long to wait before the next request, in seconds or as a date. A client that slept for the value it was given had 12 of 12 requests answered 200 in our run.

### What is the difference between a fixed window and a token bucket?

A fixed window counts requests per clock interval and resets at the mark; a token bucket refills steadily. Around a window mark our fixed window accepted 10 of 10 requests in about a second, while the token bucket accepted 5 of 10.

### What is the difference between rate limiting and throttling?

Rate limiting refuses requests over the limit; throttling delays them so they finish later. Many products use the words interchangeably. In both cases the client should slow down.

### What should a client do when it is rate limited?

Wait for the time in `Retry-After`, or back off exponentially with jitter if the header is absent, lower its concurrency, and give up after a few attempts. Better still, read the remaining quota on each response and pause before reaching zero.

## Web data without running the crawl yourself

The Web Scraping API and collectors return public pages and structured rows through one API key.

[Start free — $2/month included](https://quanticdata.io/signup/?utm_source=blog&utm_medium=website&utm_campaign=blog-end&utm_content=what-is-rate-limiting) · [Explore Web Scraping API](https://quanticdata.io/web-scraping-api/)

## Related reading

- [Web Scraping Project Ideas, Started for Real](https://quanticdata.io/blog/web-scraping-project-ideas/): Web scraping · Five web scraping project ideas, each one started for real on 20 November 2026: a price tracker, a library release dashboard, a question analysis, an open-source landscape and a topic timeline, with the code, the first run and what to build next.
- [Retry-After Header: Read It, Wait, Retry](https://quanticdata.io/blog/retry-after-header/): Web scraping · Retry-After tells a client how long to wait before the next request, as a number of seconds or as an HTTP date. Below: both formats, a Python and a Node helper we ran against a real 429 on 20 November 2026 (each waited 10 seconds and then got 200), the rate-limit headers of the QuanticData API, and how one batch job replaced 20 calls.
- [How to Use an API Key: curl, Python and JS](https://quanticdata.io/blog/how-to-use-an-api-key/): Web scraping · An API key is a secret string you send with every request so the service knows which account is calling. Nothing is installed: you add one header. Below: the same call in curl, Python, JavaScript and PowerShell, run on 20 November 2026 against the QuanticData API, the exact 401 messages for the usual mistakes, and how to keep the key out of your code.

---

Source: https://quanticdata.io/blog/what-is-rate-limiting/ · Site index for AI: https://quanticdata.io/llms.txt
