Rate Limiting
Token buckets and 429s — protecting a service from its own users and from you.
30 minDifficulty 3/5reliability · apiAI-writtenWritten by a model on 16 August 2026 and not yet read by a person. Checked automatically: schema, the pedagogical rules the hand-written material is held to, and every diagram parsed for real.
Before this
Why this exists
A single misbehaving script, or a genuinely malicious one, hammers a login endpoint with a thousand requests per second, trying every password in a list against every username. Without a limit, the server tries to handle all of them, and legitimate users trying to log in get crowded out by traffic that was never a real user in the first place.
The mental model
A rate limiter is a gatekeeper that counts requests from a given identity (an IP address, an API key, a user account) over a time window, and rejects requests once the count exceeds an allowed threshold — the specific algorithm decides HOW that counting and rejecting behaves at the edges, which matters more than it first appears.
How it works
Fixed windows have a boundary problem
A limit of '100 requests per minute', counted in fixed clock-aligned windows, allows a client to send 100 requests at 11:00:59 and another 100 at 11:01:00 — 200 requests in two seconds, both technically within their respective windows' limits. The fixed window's edge is where the abuse hides.
The token bucket algorithm allows controlled bursts while enforcing a true average rate
A bucket holds tokens, refilled at a steady rate (say, one per second up to a max of 20); each request consumes one token, and a request with no tokens available is rejected. This naturally allows a short burst (spending up to 20 saved-up tokens quickly) while still enforcing the long-run average rate, matching how real usage actually behaves — bursty, not perfectly smooth.
429 with Retry-After tells the client exactly what to do
Returning HTTP 429 ('Too Many Requests') with a `Retry-After: 30` header tells a well-behaved client precisely how long to wait before trying again — this is far more useful than a generic error, and lets a client implement automatic backoff correctly rather than guessing or retrying immediately, which would just make the situation worse.
Rate limiting protects the service from itself, not just from attackers
A legitimate client with a bug — a retry loop with no backoff, a misconfigured polling interval — can accidentally generate the same traffic pattern as an attack. Rate limiting protects the server's capacity regardless of intent, which is why it's a standard practice even for APIs with no adversarial users, purely as a safety net against accidental overload.
The mechanism
Each incoming request is attributed to an identity (typically by IP or API key) and checked against that identity's current usage state, stored centrally (often in a fast in-memory store like Redis so multiple server instances share the same count). If the request would exceed the allowed rate — computed according to the chosen algorithm's rules — it's rejected with a 429 and, ideally, guidance on when to retry; otherwise it proceeds and the usage state is updated.
What people get wrong
- Rate limiting is only necessary for APIs that expect malicious traffic.
- It also protects against accidental overload from legitimate clients — a bug causing an aggressive retry loop, a misconfigured cron job polling too frequently — none of which requires any malicious intent to genuinely overwhelm a service. APIs with no adversarial users at all still benefit from rate limiting as a safety net against their own legitimate clients' bugs, not just as a security control.
- A fixed window counter (reset every minute, on the minute) is an adequate rate limiting strategy.
- A fixed window has a boundary problem: a client can send the full limit right at the end of one window and again right at the start of the next, achieving nearly double the intended rate in a short burst spanning the boundary. This is a subtle enough issue that a fixed-window implementation LOOKS correct in normal testing but fails to actually cap the worst-case burst rate the way it appears to.
- When rate limited, a client should retry immediately, or as fast as possible, to get through.
- Retrying immediately after a 429 typically makes the underlying overload worse, not better — the correct behaviour is respecting the `Retry-After` header (or an exponential backoff if none is given) before attempting again. A client library that retries aggressively on a 429 can turn a brief, manageable overload into a sustained one, as every rejected client immediately re-attempts and compounds the load.
When not to use it
- You need to allow occasional bursts of legitimate activity (a user rapidly clicking through several actions) without a hard, punishing limit.
- A token bucket algorithm, which naturally accommodates bursts up to the bucket's capacity while still enforcing a true average rate over time.
- The limit needs to be extremely simple to reason about and implement, and burst tolerance isn't a concern.
- A sliding window log or sliding window counter, which is more accurate than a fixed window (no boundary problem) at a modest implementation complexity cost, without needing a full token bucket.
Terms
- Token bucket
- — A rate limiting algorithm where tokens refill at a steady rate up to a cap, and each request consumes one, naturally permitting controlled bursts up to the bucket's capacity.
- 429 Too Many Requests
- — The HTTP status code indicating a client has exceeded the allowed rate, conventionally paired with a Retry-After header.
- Fixed window
- — A rate limiting strategy that resets a request counter at fixed clock-aligned intervals, which has a known boundary problem allowing near-double bursts across a window edge.
- Retry-After
- — An HTTP header telling the client how long to wait before making another request, commonly sent alongside a 429 or 503 response.
In an interview
Why can a fixed-window rate limiter of '100 requests per minute' still allow 200 requests in a two-second span?
- the window resets at a fixed clock boundary, e.g. on the minute
- a client can send 100 requests in the last second of one window and another 100 in the first second of the next window
- both bursts are individually within their own window's limit, but together they exceed the intended rate over that short span
Can you recall it?
How does a token bucket algorithm allow short bursts while still enforcing a genuine average rate limit over time?