Rate Limiting

The 30-second version

That agreed number is a Rate limitRate limitA cap on how many requests are allowed per unit of time — e.g. 100 per second. It says nothing about how many may be running at once.: "at most 100 requests per minute per user", say. The limiter answers one question for every incoming request, and it has to answer fast: "Has this client used up its allowance?" If not, the request goes through. If so, the server sends back a quick "too many requests, try later" instead of doing the work.

Here is one client's real-looking traffic over ten seconds. The server is happy to handle 5 requests per second from any one client:

When a request is refused, the server answers with the HTTP status 429 Too Many Requests429 Too Many RequestsThe HTTP status code a server returns when you've exceeded its rate limit, often with a Retry-After header saying how long to wait., usually with a Retry-AfterRetry-AfterAn HTTP response header telling the client how long to wait before trying again, e.g. Retry-After: 1 means “in 1 second”. header saying how many seconds to wait. That answer costs the server almost nothing: no database query, no real work.

Why limit at all?

What this topic covers

Every limiter has to decide how to count "recently". That choice, the algorithm, decides how bursty traffic is treated, and it's the heart of this topic:

• §02–§05: five algorithms (fixed window, sliding log, sliding counter, token bucket, leaky bucket), all run on the same traffic so you can compare them.
• §06: real limiters in Go and JavaScript.
• §07: limiting across many servers with Redis, and the race that breaks naive versions.
• §08–§11: what to count by (user? IP?), the HTTP side (429, headers, polite clients), where the limit should live, and running it in production.

Fixed window: count per minute

Think of a gym that allows 10 visits per calendar month. On the 1st, everyone's card resets to 0. Simple to run and simple to explain: that's why it's the first limiter most people write.

How it works

With a limit of 10 requests per minute:

The only state is "which window am I counting, and how many so far" per client. To find the window, divide the current time by the window length and round down: every timestamp in the same minute gives the same number.

type counter struct {
    window int64 // which window this count belongs to
    count  int
}

var (
    mu    sync.Mutex // many requests run at once: guard the map
    state = map[string]*counter{}
)

func allow(user string, limit int, window time.Duration) bool {
    w := time.Now().UnixNano() / int64(window) // which window are we in?
    mu.Lock()
    defer mu.Unlock()
    c, ok := state[user]
    if !ok || c.window != w { // first request, or a new window: start from zero
        c = &counter{window: w}
        state[user] = c
    }
    c.count++
    return c.count <= limit
}

Both versions print true true true false for four quick requests with a limit of 3. (They keep one small entry per client forever; real code deletes entries for clients that have gone quiet. §06 shows how.)

The catch: the window edge

The counter resets at 12:01:00 sharp, whether or not the client just used its whole allowance at 12:00:59. Try it: two bursts of 10, one just before the minute and one just after.

With the defaults, all 20 requests are allowed, inside about 10 seconds, though the limit says 10 per minute. This is the boundary burst: around a window edge, a fixed window can let through up to 2× the limit in a short time. Switching to the sliding log (next section) closes the gap.

When fixed windows are fine

Don't throw it away. A fixed window is one integer per client, it's trivial to store in RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages). (§07), and it's easy to explain to customers ("1,000 calls per hour, resets on the hour"). For long periods, like a daily quotaQuotaA limit over a long period — 10,000 calls per day or per month — usually tied to a pricing plan. A rate limit stops bursts; a quota caps total usage., the edge matters less: a quota is about total usage per customer, not about protecting the server from a spike, so a customer squeezing two days' allowance around midnight is usually acceptable. The edge burst hurts most with short windows whose job is to protect a server.

Sliding windows: log and counter

The fixed window's problem is that "this minute" jumps: at 12:01:00 it suddenly forgets everything. A sliding window is always the 60 seconds ending now. It moves forward smoothly with the clock, like a ruler you keep sliding along a timeline.

Exact: the sliding window log

The Sliding window logSliding window logA rate-limit algorithm that stores the timestamp of every request and counts how many fall in the last N seconds. Exact, but it costs memory per request. keeps the time of every allowed request. For each new request:

1. Throw away timestamps older than now − 60 s.
2. Count what's left.
3. If the count is under the limit, allow the request and add now to the log.

Worked example, limit 3 per 60 s:

At 1:05, the 60 seconds since 0:05 still contain three requests, so it's rejected, even though a new calendar minute began at 1:00. At 1:12 the 0:10 entry has slid out (it's more than 60 s old), so there's room again. Step through the same five requests:

const logs = new Map(); // user → array of timestamps (ms), oldest first

function allow(user, limit = 3, windowMs = 60_000) {
  const now = Date.now();
  const log = logs.get(user) ?? [];
  while (log.length && log[0] <= now - windowMs) log.shift(); // 1. drop what slid out
  if (log.length >= limit) return false;                       // 2–3. full → reject
  log.push(now);
  logs.set(user, log);
  return true;
}

It's exact: never more than the limit in any 60 seconds. The cost is memory. It stores one timestamp per allowed request, so a limit of 1,000 per hour for a million users can mean up to a billion timestamps. At 8 bytes each that's 8 GB before any overhead. Fine for small limits (like 5 login attempts per minute), expensive for big ones.

Cheap: the sliding window counter

The Sliding window counterSliding window counterA rate-limit algorithm that keeps only two numbers — the count for the previous window and for the current one — and estimates the last N seconds as a weighted mix of the two. Cheap and nearly exact. keeps just two numbers per client: how many requests in the previous fixed window, and how many so far in the current one. It then estimates the sliding window by assuming last minute's requests were spread evenly:

estimate = previous_count × (1 − elapsed) + current_count

elapsed is how far we are into the current window, from 0 to 1. Worked example, limit 10 per minute: last minute had 8 requests, this minute has 3 so far, and we're 15 seconds in.

1. 15 s of 60 s → elapsed = 0.25.
2. The last 60 seconds cover the final 45 s of last minute, which is 75% of it. So count 75% of its requests: 8 × 0.75 = 6.
3. Add this minute's: 6 + 3 = 9.
4. Allowing one more makes 10, which is not over 10 → allowed.

Drag the sliders to see the estimate change:

How wrong is the estimate?

It's an approximation, and it can be wrong both ways. If last minute's requests all came early, it over-counts and rejects a request it could have allowed. If they all came at the very end, it under-counts and lets a few too many through. The edge burst from §02 is now mostly absorbed, though: right after the edge, last minute still weighs close to 100%.

In practice, traffic from many clients is spread out enough that the error is tiny. Cloudflare, which uses this method for its rate limiting, reported that only about 0.003% of requests were wrongly allowed or limited in their measurements. Two integers per client instead of a whole log is usually a great trade.

Token bucket and leaky bucket

Worker pools §08 introduced the Token bucketToken bucketA rate-limit algorithm: a bucket refills with tokens at a steady rate (say 10 per second) up to a maximum. Each request takes one token; no token, no request. The maximum allows short bursts.: a bucket holds up to b tokens and refills at r tokens per second; each request takes a token, and an empty bucket means "no". Here we look at how it behaves on real traffic, how it's implemented, and how its sibling, the leaky bucket, differs.

Two knobs, two different jobs

• Rate r is the long-run limit. Over an hour, a client can't average more than r per second.
• Size b is the burstBurstA lot of requests arriving close together — e.g. 20 in half a second — even if the average over a minute is low. A limiter's burst size is how many it lets through at once after a quiet period. allowance. After a quiet spell the bucket is full, so the client can send b requests back to back.

Together they give the one formula worth memorizing: in any stretch of t seconds, a token bucket allows at most b + r × t requests. The b is what was saved up; r × t is what refilled meanwhile. With r = 5/s and b = 5, a client can get about 5 + 5 = 10 requests in one second after a quiet spell, and then 5 per second after that. It isn't a strict "5 per second" limiter, and that's on purpose: real users are bursty (a page load fires 8 API calls at once, then nothing for 30 seconds).

Try the steady 8/s trace with the defaults: the first second gets the saved-up burst, then exactly 5 per second get through. Then switch back to the bursty mix and slide the bucket size to 1: no saving up is possible, so requests must be spaced at least 200 ms apart and the burst at 2 s is almost entirely rejected.

No timer needed: lazy refill

It's tempting to picture a timer adding a token every 200 ms. With a million clients that's a million timers. Real implementations store just two numbers per client, the token count and the time it was last updated, and do the refill arithmetic when a request arrives:

tokens = min(b, tokens + (now − last) × r)    ← catch up on refills since last time
last   = now
if tokens ≥ 1: tokens −= 1, allow             ← spend one
else: reject; next token in (1 − tokens) / r seconds

Worked example, r = 2/s, b = 3. Three requests at t = 0 empty the bucket. A fourth request at t = 0 is rejected, and the limiter can say exactly when to retry: (1 − 0) / 2 = 0.5 s. A request at t = 0.6 s finds 0 + 0.6 × 2 = 1.2 tokens, takes one, and leaves 0.2. The pseudocode's "next token in (1 − tokens) / r" line comes for free: it's exactly the value for the Retry-AfterRetry-AfterAn HTTP response header telling the client how long to wait before trying again, e.g. Retry-After: 1 means “in 1 second”. header (§09).

Two more details that matter in practice:

• Start full or empty? Most libraries (including Go's, §06) start full, so a new client can burst immediately. Starting empty is friendlier to the server after a restart, when every client is "new" at once.
• Costs other than 1. Nothing says a request must take exactly one token. An expensive export can take 10. That's the basis of cost-based limits (§08).

The leaky bucket

Picture a bucket with a small hole in the bottom. Water (requests) pours in at any pace; it leaks out at a fixed rate. If water comes in faster than it leaks, the bucket fills; when it's full, extra water spills over (is rejected). The Leaky bucketLeaky bucketA rate-limit algorithm pictured as a bucket with a hole: requests pour in, and leak out at a fixed rate. As a queue it smooths traffic (requests wait their turn); as a meter it just accepts or rejects, and then behaves like a token bucket. is used in two different ways, and people mix them up:

1. As a queue. Requests really wait in the bucket and are passed on to your server at a steady rate. Output is perfectly smooth; the price is waiting time. nginx's limit_req with a burst setting works like this (§10).
2. As a meter. Nobody waits. The bucket is just a number used to decide yes or no. Used this way it makes exactly the same decisions as a token bucket: a bucket that's 3/5 full of water is the same as a token bucket with 2 of 5 tokens left.

Flip between the two on the bursty trace:

With the defaults (5 per second, room for 4 to wait), the queue accepts 44 of 63 requests, and the bottom row is evenly spaced: never more than 5 in any second. But during the bursts, requests wait up to 800 ms. Switch to "meter" and the accepted set doesn't change at all; only the waiting disappears.

GCRA: a leaky-bucket meter in one number

The meter version has an elegant implementation called GCRAGCRAGeneric Cell Rate Algorithm: a leaky-bucket meter that stores just one timestamp per key — the earliest time the next request would be “on schedule”. It makes the same decisions as a token bucket with less state.. Instead of counting tokens, it remembers a single timestamp per client: TAT, the "theoretical arrival time" when the client would next be on schedule if it sent exactly at rate r.

T   = 1 / r               ← spacing between requests at the allowed rate
tau = (b − 1) × T         ← how far ahead of schedule a client may get (the burst)

on request at time now:
  tat = max(tat, now)     ← a client that was idle doesn't bank more than a full burst
  if tat − now > tau: reject          ← too far ahead of schedule
  else: tat = tat + T, allow

With r = 5/s (T = 200 ms) and b = 5 (tau = 800 ms), five requests at t = 0 push TAT to 200, 400, 600, 800 and 1000 ms. A sixth at t = 0 sees 1000 − 0 = 1000 > 800 and is rejected. At t = 200 ms, 1000 − 200 = 800, which is not over 800, so it's allowed. Step through it:

The algorithm race

Below, the five limiters from §02–§04 watch exactly the same requests. Each is configured for "5 per second": windows of 1 second with a limit of 5, and buckets with rate 5/s and room for a burst of 5. Switch traces and compare the rows.

What to notice

On the bursty mix (63 requests):

• Fixed window lets 37 through, but its "busiest 1 s" is 9: the burst straddling the 5 s edge got through on both sides, exactly the §02 problem.
• Sliding log is the strictest: 32 allowed, and never more than 5 in any second. It's the only one that literally enforces "5 in any 1 second".
• Sliding counter lands close to the log (33), but it let 7 through in one second around the burst at 2 s, which happens to start just before the 2 s window edge. The previous window's 3 requests all came in its last 0.3 s, but the counter assumes they were spread evenly over the whole second, so it "forgets" them too quickly. That's the approximation error from §03, in action.
• Token bucket is the most generous, 44, because quiet periods refill the bucket and the bursts spend it. Its busiest second is 9: close to burst + rate = 10.
• Leaky bucket (queue) accepts the same 44 as the token bucket (the "same decisions" from §04), but delays the extras so your server never sees more than 5 in a second.

Now try edge bursts: the fixed window lets 10 through in 0.4 seconds, twice. And steady 8/s: every algorithm ends up near 5 per second; they mostly differ in how they handle the start. The sliding counter is noticeably stingier here (41 vs the log's 50): its even-spread assumption is wrong for a client that's over the limit all the time.

"Allows 5 per second" means different things

The race shows that the limit's name doesn't tell you the worst case. In any one second:

In code: x/time/rate and a hand-rolled limiter

Go: three ways to ask for a token

rate.NewLimiter(r, b) creates a token bucket with rate r per second and size b, starting full. It's safe to use from many goroutinesGoroutineA very lightweight task in Go, started with the go keyword. The Go runtime runs thousands of them on just a few OS threads. at once. It gives you three ways to ask for a token, and picking the right one is most of the skill:

package main

import (
    "context"
    "fmt"
    "time"

    "golang.org/x/time/rate"
)

func main() {
    lim := rate.NewLimiter(2, 3) // 2 tokens per second, bucket holds 3

    // 1. Allow: take a token now, or say no. Never waits.
    for i := 1; i <= 5; i++ {
        fmt.Printf("Allow #%d: %v\n", i, lim.Allow())
    }

    // 2. Wait: block until a token is ready (or ctx ends).
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    start := time.Now()
    for i := 1; i <= 3; i++ {
        if err := lim.Wait(ctx); err != nil {
            fmt.Println("Wait error:", err)
            break
        }
        fmt.Printf("Wait #%d returned after %v\n", i, time.Since(start).Round(50*time.Millisecond))
    }

    // 3. Reserve: book a future token and decide yourself.
    r := lim.Reserve()
    d := r.Delay()
    fmt.Printf("Reserve: ok=%v, must wait %v\n", r.OK(), d.Round(50*time.Millisecond))
    if d > 300*time.Millisecond {
        r.Cancel() // too long: give the token back
        fmt.Println("too slow, cancelled")
    }
}

Allow #1: true
Allow #2: true
Allow #3: true
Allow #4: false
Allow #5: false
Wait #1 returned after 500ms
Wait #2 returned after 1s
Wait #3 returned after 1.5s
Reserve: ok=true, must wait 500ms
too slow, cancelled

The full bucket gives three trues, then it's empty. Each Wait then sleeps until the next token, one every 500 ms at 2 per second. The reservation books the token due in another 500 ms; we decide that's too long and cancel it.

A per-client limiter as HTTP middleware

One limiter protects one thing. To limit each client separately, keep a map from client key to limiter, and wrap your handler in MiddlewareMiddlewareA function that wraps an HTTP handler to run code before and/or after it — logging, auth, limits — without changing the handler itself. that checks it (the wrapping pattern is explained in Structural patterns §05). The part people forget is eviction: without it, the map grows by one limiter per client ever seen.

type client struct {
    lim      *rate.Limiter
    lastSeen time.Time
}

type PerKey struct {
    mu      sync.Mutex
    clients map[string]*client
    r       rate.Limit
    b       int
    valid   func(key string) bool // is this a real API key? (a fast, cached lookup)
}

func NewPerKey(r rate.Limit, b int, valid func(string) bool) *PerKey {
    p := &PerKey{clients: map[string]*client{}, r: r, b: b, valid: valid}
    go p.evictLoop()
    return p
}

func (p *PerKey) get(key string) *rate.Limiter {
    p.mu.Lock()
    defer p.mu.Unlock()
    c, ok := p.clients[key]
    if !ok {
        c = &client{lim: rate.NewLimiter(p.r, p.b)} // first time we see this key
        p.clients[key] = c
    }
    c.lastSeen = time.Now()
    return c.lim
}

// evictLoop forgets clients that have been quiet for 3 minutes.
// Safe as long as an empty bucket refills completely in under 3 minutes
// (b / r < 180 s): a forgotten client comes back to a full bucket either way.
func (p *PerKey) evictLoop() {
    for range time.Tick(time.Minute) {
        p.mu.Lock()
        for k, c := range p.clients {
            if time.Since(c.lastSeen) > 3*time.Minute {
                delete(p.clients, k)
            }
        }
        p.mu.Unlock()
    }
}

func (p *PerKey) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        key := r.Header.Get("X-API-Key")
        if !p.valid(key) { // unknown key: refuse BEFORE creating a limiter for it
            http.Error(w, "invalid API key", http.StatusUnauthorized)
            return
        }
        if !p.get(key).Allow() {
            w.Header().Set("Retry-After", "1")
            http.Error(w, "rate limit exceeded", http.StatusTooManyRequests) // 429
            return
        }
        next.ServeHTTP(w, r)
    })
}

Wired up with NewPerKey(5, 3, valid) (5 per second, bursts of 3, and valid accepting only alice and bob) and hit with four quick requests from alice, one from bob, and one with a made-up key:

alice 200
alice 200
alice 200
alice 429 1
bob 200
x7f3-made-up 401

Alice's fourth request finds her bucket empty and gets 429 with Retry-After: 1. Bob has his own bucket, so Alice's burst doesn't affect him at all. That's the fairness from §01. The made-up key is refused with 401 before any limiter is created for it. Without that check, an attacker could send a fresh random key with every request, get a full bucket each time, and grow the map without limit (§08, §12).

JavaScript: token bucket and sliding window counter

The token bucket from Worker pools §08, with two upgrades: it can cost more than 1, and on rejection it says when to retry (the arithmetic from §04). The clock is passed in, which makes it easy to test. Next to it, the sliding window counter from §03.

// Token bucket: `rate` tokens per second, holds at most `burst`.
export class TokenBucket {
  constructor(rate, burst, now = () => performance.now()) {
    this.rate = rate;
    this.burst = burst;
    this.now = now;          // injectable clock → easy to test
    this.tokens = burst;     // start full
    this.last = now();
  }
  tryTake(cost = 1) {
    const t = this.now();
    this.tokens = Math.min(this.burst, this.tokens + ((t - this.last) / 1000) * this.rate);
    this.last = t;
    if (this.tokens < cost) {
      const waitMs = ((cost - this.tokens) / this.rate) * 1000; // when will we have enough?
      return { ok: false, retryAfterMs: Math.ceil(waitMs) };
    }
    this.tokens -= cost;
    return { ok: true, remaining: Math.floor(this.tokens) };
  }
}

// Sliding window counter: `limit` requests per `windowMs`.
export class SlidingWindowCounter {
  constructor(limit, windowMs, now = () => Date.now()) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.now = now;
    this.windowStart = Math.floor(now() / windowMs) * windowMs;
    this.prev = 0; // count in the previous window
    this.curr = 0; // count in the current window
  }
  tryTake() {
    const t = this.now();
    const start = Math.floor(t / this.windowMs) * this.windowMs;
    if (start !== this.windowStart) {
      // Moved on: the current window becomes "previous" — unless we skipped a whole window.
      this.prev = start - this.windowStart === this.windowMs ? this.curr : 0;
      this.curr = 0;
      this.windowStart = start;
    }
    const elapsed = (t - start) / this.windowMs;         // 0.0 … 1.0 into this window
    const estimate = this.prev * (1 - elapsed) + this.curr;
    if (estimate + 1 > this.limit) return false;         // one more would go over
    this.curr += 1;
    return true;
  }
}

A demo with a fake clock: a plain variable we set by hand, so the output is identical on every run.

import { TokenBucket, SlidingWindowCounter } from "./limiters.mjs";

let fake = 0;                 // a fake clock in ms
const clock = () => fake;

const tb = new TokenBucket(2, 3, clock); // 2/s, burst 3
for (let i = 1; i <= 4; i++) console.log(`t=${fake}ms take #${i}`, tb.tryTake());
fake = 600;
console.log(`t=${fake}ms take #5`, tb.tryTake());

const sw = new SlidingWindowCounter(10, 60_000, clock); // 10 per minute
let ok = 0;
for (let i = 0; i < 12; i++) if (sw.tryTake()) ok++;
console.log(`minute 1: ${ok} of 12 allowed`);
fake = 75_000; // 15 s into minute 2: previous window still weighs 75%
console.log("estimate =", 10 * (1 - 15 / 60), "→ allowed?", sw.tryTake(), sw.tryTake(), sw.tryTake());
fake = 105_000; // 45 s into minute 2: previous weighs 25%
let later = 0;
for (let i = 0; i < 10; i++) if (sw.tryTake()) later++;
console.log(`at 1:45, ${later} more allowed`);

t=0ms take #1 { ok: true, remaining: 2 }
t=0ms take #2 { ok: true, remaining: 1 }
t=0ms take #3 { ok: true, remaining: 0 }
t=0ms take #4 { ok: false, retryAfterMs: 500 }
t=600ms take #5 { ok: true, remaining: 0 }
minute 1: 10 of 12 allowed
estimate = 7.5 → allowed? true true false
at 1:45, 5 more allowed

Read the sliding counter lines with the formula: at 1:15 the estimate starts at 7.5, so two more fit (8.5, 9.5) and the third would make 10.5. At 1:45 the previous minute only weighs 25%: 10 × 0.25 + 2 = 4.5, so five more fit before reaching 10.

Many servers, one limit: Redis

Your API runs on several servers behind a Load balancerLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped.. If each server keeps its own limiter, a client whose requests are spread across 3 servers gets 3 times its limit. Nobody did anything wrong; each server just counted what it saw.

There are three ways out:

1. Divide the limit by the number of servers (10/s each for 30/s overall). Free, but it breaks when servers are added or removed, and when the load balancer doesn't spread one client evenly.
2. Send each client to the same server, for example by hashing the API key (Load balancing §05). Then a local limiter is enough, until that server dies or a single huge client overloads it.
3. Share the count in a fast central store. Almost always RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages)., because it keeps everything in memory and answers in well under a millisecond inside a data center.

This section is about option 3.

Why Redis fits: one command at a time

Redis runs commands one at a time, in the order they arrive. Two clients can't be "in the middle of" a command together. So each single command is an atomic operationAtomic operationAn operation that happens as one indivisible step: nobody can ever see it half-done or squeeze in between its parts. Go's sync/atomic package offers atomic counters.. That's exactly the guarantee a shared counter needs, as long as each decision fits in one command.

Fixed window: INCR + EXPIRE

INCR key adds 1 and returns the new value (creating the key at 1 if it's missing). Put the window number in the key, give the key an expiry so old windows clean themselves up, and you have the §02 limiter for a whole fleet:

func allowFixed(ctx context.Context, rdb *redis.Client, user string, limit int64, window time.Duration) (bool, error) {
    slot := time.Now().Unix() / int64(window.Seconds())
    key := fmt.Sprintf("rl:fixed:%s:%d", user, slot) // one key per user per window

    pipe := rdb.TxPipeline()            // MULTI … EXEC: both commands run together
    incr := pipe.Incr(ctx, key)         // +1, creates the key at 1 if missing
    pipe.ExpireNX(ctx, key, window)     // set a TTL only if none is set yet (Redis 7+)
    if _, err := pipe.Exec(ctx); err != nil {
        return false, err
    }
    return incr.Val() <= limit, nil
}

fixed #1 allowed=true err=<nil>
fixed #2 allowed=true err=<nil>
fixed #3 allowed=true err=<nil>
fixed #4 allowed=false err=<nil>

The race that breaks naive limiters

A very common first attempt reads the counter, checks it in the app, and writes it back. Each step is atomic on its own, but the three steps together aren't. Step through what happens when two servers handle the same client at the same moment:

1// runs on every server, for every request
2func allow(ctx context.Context, key string) bool {
3    n, _ := rdb.Get(ctx, key).Int() // 1. read
4    if n >= 5 {                      // 2. decide
5        return false
6    }
7    rdb.Set(ctx, key, n+1, time.Minute) // 3. write
8    return true
9}

This is a Read-modify-writeRead-modify-writeRead a value, compute a new one from it, write it back — like count++. Unsafe if someone else can write in between your read and your write. race, and the result is a Lost updateLost updateTwo writers each read the same old value and write back their change; the second write silently erases the first., exactly like the database version in Race conditions §05. It isn't rare. Here is a real run: two "servers" (two separate Redis connections), each handling 50 simultaneous requests, against a limit of 50 in total:

GET+SET limit 50 → admitted 100, counter says 4
INCR    limit 50 → admitted 50, counter says 100

With GET-then-SET, all 100 requests got in, and the counter ended at 4: almost every request read a small number before anyone wrote. (The exact counter varied between runs: 4, 6, 21. The 100 admitted didn't.) With INCR, exactly 50 were admitted. The counter says 100 because rejected requests were counted too; that's harmless, since the key expires with its window.

Sliding log: a sorted set

The §03 sliding log maps neatly onto a Redis sorted setSorted set (Redis)A Redis data type holding unique members, each with a number called a score, kept ordered by score. With timestamps as scores you can count or delete “everything older than X” in one command.: each request is a member, scored by its timestamp. "Forget everything older than the window" is a single command.

async function allowSlidingLog(user, limit, windowMs) {
  const key = `rl:log:${user}`;
  const now = Date.now();
  const [, , [, count]] = await redis
    .multi()                                        // MULTI … EXEC: runs as one unit
    .zremrangebyscore(key, 0, now - windowMs)       // 1. forget requests older than the window
    .zadd(key, now, `${now}-${randomUUID()}`)       // 2. log this request (unique member)
    .zcard(key)                                     // 3. how many are in the window now?
    .pexpire(key, windowMs)                         // 4. delete the key if the user goes quiet
    .exec();                                        // → [[err, result], [err, result], …]
  return count <= limit;
}

log #1 true
log #2 true
log #3 true
log #4 false
entries stored: 4

Notice entries stored: 4: the rejected request was logged too, because MULTI can't say "add it only if the count is low". The commands in a MULTI can't look at each other's results. Counting rejected attempts means a client that keeps hammering stays blocked until it actually slows down. That might be what you want (it punishes abuse) or not. If not, you need logic between the steps, which brings us to Lua.

Token bucket: one Lua script

A token bucket must read the state, decide, then write, with the write depending on the read. MULTI can't do that. A Lua scriptLua script (Redis)A small program in the Lua language that you send to Redis with EVAL. Redis runs the whole script without running any other command in between, so a multi-step read–decide–write becomes one atomic step. can: Redis runs the whole script as one step, with no other command in between.

var tokenBucket = redis.NewScript(`
local key   = KEYS[1]
local rate  = tonumber(ARGV[1])   -- tokens per second
local burst = tonumber(ARGV[2])   -- bucket size
local cost  = tonumber(ARGV[3])

local t   = redis.call('TIME')                    -- Redis's own clock: {seconds, microseconds}
local now = tonumber(t[1]) + tonumber(t[2]) / 1e6

local state  = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or burst        -- new key → full bucket
local ts     = tonumber(state[2]) or now

tokens = math.min(burst, tokens + (now - ts) * rate)   -- lazy refill (§04)
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', key, math.ceil(burst / rate) * 2)  -- idle buckets disappear
return { allowed, string.format('%.2f', tokens) }
`)

// rate 1/s, burst 3, cost 1
res, err := tokenBucket.Run(ctx, rdb, []string{"rl:tb:alice"}, 1, 3, 1).Slice()

The same script, loaded from a file, works from JavaScript:

redis.defineCommand("tokenBucket", {
  numberOfKeys: 1,
  lua: readFileSync("token_bucket.lua", "utf8"),   // the same script as the Go version
});

const [allowed, left] = await redis.tokenBucket("rl:tb:carol", 1, 3, 1); // 1/s, burst 3, cost 1

bucket #1 allowed=1 tokens left=2.00
bucket #2 allowed=1 tokens left=1.00
bucket #3 allowed=1 tokens left=0.00
bucket #4 allowed=0 tokens left=0.00
bucket #5 allowed=0 tokens left=0.00

Who are we counting? Keys, tiers and costs

Every limiter so far had a user or key argument. That choice matters as much as the algorithm: count by the wrong thing and you'll block innocent people while the real culprit walks around the limit.

The usual keys

Real systems layer them. A login endpoint might allow 10 attempts per minute per IP and 5 per hour per account, so a bot can't try one password on a million accounts from one IP, or a million passwords on one account from many IPs. That's the defence against credential stuffingCredential stuffingAn attack where bots try huge lists of leaked username/password pairs against a login form, hoping people reused passwords. Per-account and per-IP rate limits on login slow it down.. A busy API might check per key, then per organisation, then a global cap on top.

Why IP addresses are a trap

The IP address is tempting because every request has one. But it identifies a network connection, not a person:

• Many people, one address. Offices, universities and mobile carriers put thousands of people behind a few public addresses using NATNATNetwork Address Translation: a router rewrites private addresses (like 192.168.1.20) to its one public address on the way out, and back on the way in. Why your laptop isn't directly reachable from the internet. (and at internet-provider scale, CGNATCGNAT (carrier-grade NAT)NAT run by your internet provider: many customers share one public IPv4 address. It uses the range 100.64.0.0/10, so your home router's “public” address may not be public at all.). A per-IP limit that's fine for one person blocks a whole campus at 9 a.m.
• One person, many addresses. With IPv4 / IPv6IPv4 / IPv6Two versions of IP. IPv4 addresses are 32 bits, written as four numbers 0–255 (192.168.1.37) — only about 4.3 billion exist. IPv6 addresses are 128 bits, written in hex groups (2001:db8::1), so there are practically unlimited., one home typically gets a block of billions of addresses. Limiting each address separately is useless. Limit by the network prefix instead: commonly the first 64 bits (the /64), or a /56 or /48 for stricter limits, since homes are often given that many /64s.
• Whose address is it anyway? Behind a Reverse proxyReverse proxyA proxy that sits in front of your servers (nginx, Envoy, a cloud load balancer): clients connect to it, and it forwards each request to one of your backends. or load balancer, the connection comes from the proxy, so every request seems to share one IP. The original client is in the X-Forwarded-ForX-Forwarded-ForAn HTTP header a proxy adds to say who the original client was: X-Forwarded-For: 203.0.113.7. Each proxy appends the address it received the request from. Only trust the entries your own proxies added. header. But anyone can send that header, so only trust the entries your own proxies added. Trusting the leftmost entry lets an attacker pick a fresh "IP" for every request.

Use per-IP limits as a generous outer fence against floods, and do the real, fair limiting on an identity the client can't change at will: the account or a validated API key. An API key you haven't checked is just a string the client chose: an attacker can send a fresh random one with every request and get a full bucket each time. Reject unknown keys (a cheap, cached lookup) before you limit by them, and keep the per-IP fence in front.

Tiers and quotas

Once the key is a customer, the limit can depend on who they are. A typical plan table:

The rate limit stops bursts that hurt your servers; the quotaQuotaA limit over a long period — 10,000 calls per day or per month — usually tied to a pricing plan. A rate limit stops bursts; a quota caps total usage. caps total usage (and usually maps to billing). They're usually separate limiters: a token bucket for the rate, and a monthly counter (a fixed window is perfect here) for the quota.

Not all requests cost the same

GET /items/42 is a cache lookup; POST /reports/export might scan a million rows. Counting both as "1 request" lets a client stay under the limit while hammering your most expensive endpoint. Cost-based limits charge each request a weight, so it takes several tokens from the bucket:

var cost = map[string]int{
    "GET /items":    1,
    "GET /search":   5,
    "POST /exports": 50,
}

// lim was created with rate.NewLimiter(100, 200): 100 points per second, bursts of 200
if !lim.AllowN(time.Now(), cost[route]) {
    w.Header().Set("Retry-After", "1")
    http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
    return
}

GraphQL: limit by query cost

With GraphQLGraphQLAn API style where the client sends a query describing exactly which fields and nested objects it wants (posts → author → name), and the server assembles the answer by calling a resolver function per field., one HTTP request can ask for almost anything: one field, or every user's every post's every comment. Counting requests is meaningless. Instead, the server computes a query costQuery cost (GraphQL)A number a GraphQL server computes for each query before running it, based on how many objects it could return. Limits are then counted in cost points instead of requests, so one huge query uses up more of your budget than a tiny one. before running the query, from how many objects it could return:

query {
  viewer {
    repositories(first: 50) {        ← up to 50 repositories
      issues(first: 20) {            ← up to 20 issues each → 50 × 20 = 1,000 issues
        title
      }
    }
  }
}

A typical scheme charges for the number of objects that could come back (here ~1,050) and scales it down to points. The client has a points budget per hour; a query that would cost more than what's left is rejected before any work is done. GitHub's GraphQL API uses a variant: it counts the connection requests the query needs (here 1 + 50 = 51), divides by 100, and charges at least 1 point, from a budget of 5,000 points per hour per user at the time of writing. Shopify's uses a leaky bucket of cost points. The principle is the cost-based limit above: charge for the work, not the request.

Saying no over HTTP: 429, Retry-After, polite clients

The status code: 429, not 503 or 400

429 Too Many Requests429 Too Many RequestsThe HTTP status code a server returns when you've exceeded its rate limit, often with a Retry-After header saying how long to wait. means "you are sending too much". It's about this client, and it's the client's job to slow down. Don't confuse it with:

• 503 Service UnavailableHTTP 429 / 503Status codes for “not now”. 429 Too Many Requests: you (this client) are over your limit. 503 Service Unavailable: the server itself is overloaded or shutting down. Both may carry a Retry-After header.: "I am overloaded or down". Nothing to do with this client; it's what Load sheddingLoad sheddingDeliberately rejecting some requests (e.g. HTTP 503) when overloaded, so the rest can still be served quickly. returns (§10).
• 400 / 403: "this request is wrong" or "you may not do this at all". Clients don't retry those, so using them for a temporary limit makes clients give up instead of waiting.

A rejection should be cheap. Check the limit before parsing the body, querying the database or calling anything else, so that a flood of rejected requests costs you almost nothing.

Telling the client when, and how much

A well-formed rejection looks something like this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 2
RateLimit-Policy: "default";q=100;w=60
RateLimit: "default";r=0;t=2

{"error": "rate_limited", "message": "100 requests per minute. Try again in 2 seconds."}

• Retry-After is the standard HTTP header (also used with 503) saying when to try again. It's either a number of seconds (2) or a full date (Fri, 30 Oct 2026 12:01:00 GMT). Clients must handle both; seconds is far more common. A token bucket can compute it exactly (§04).
• RateLimit headers let clients slow down before they hit the wall. They aren't standard yet. Many APIs send X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, but they disagree on details: GitHub's x-ratelimit-reset is a Unix timestamp, while others send "seconds until reset". The IETF is standardizing RateLimit headersRateLimit headersResponse headers that tell a client how much of its limit is left and when it resets, so it can slow down before it gets a 429. Many APIs use X-RateLimit-Limit / -Remaining / -Reset; the IETF is standardizing RateLimit-Policy and RateLimit. in an Internet-DraftInternet-DraftA work-in-progress document at the IETF, the group that writes internet standards. Drafts can still change or be dropped; only a published RFC is a finished standard.. In recent drafts, RateLimit-Policy describes the limit (q=100 requests per w=60 seconds) and RateLimit the current state (r=0 remaining, reset in t=2 seconds). The syntax has changed between drafts, so check the latest one before relying on it.

The client side: back off, with jitter

Being limited is normal; how a client reacts decides whether the limit calms things down or makes them worse. The rules:

1. Honor Retry-After when it's there.
2. Otherwise use exponential Exponential backoffExponential backoffWaiting longer after each failed retry, typically doubling: 1 s, 2 s, 4 s, 8 s… up to a cap. It gives a struggling server room to recover instead of hammering it.: wait 0.4 s, then 0.8, 1.6, 3.2… up to a cap, so a struggling server gets more breathing room each time.
3. Add JitterJitterA small random amount added to a delay or timer — a retry wait, a reconnect, a connection's maximum age — so that many clients don't all act at the same instant and stampede a server.: a random part of the wait, so many clients don't come back at the same instant.
4. Give up after a few attempts and report the error.
5. Only retry requests that are safe to repeat. A POST that charged a card may have half-worked; see Idempotency for how to make retries safe.

Rule 3 is the one people skip, because "the server told me 2 seconds" sounds precise. Here's what happens when one customer's 300 workers, all sharing one API key, are all told exactly that:

Waiting exactly 2 seconds turns 300 polite workers into a Thundering herdThundering herdMany clients hitting a server at the same moment — e.g. all retrying together right after an outage — which knocks it over again.: they come back in the same 100 ms, 270 are rejected again, and they keep returning in lockstep. After 12 seconds, 150 still haven't been served and 1,200 requests have been sent. With up to 1 second of jitter it's much better but still a little bunched; with up to 2 seconds (as much jitter as the hint itself) they spread out under the key's limit: everyone is done by 4 s, with exactly 300 requests. That's why the code below adds a random wait as long as the Retry-After hint, not a token few milliseconds.

func getPolitely(ctx context.Context, url string) (*http.Response, error) {
    const maxAttempts = 5
    base, maxWait := 200*time.Millisecond, 10*time.Second
    for attempt := 1; ; attempt++ {
        req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
        if err != nil {
            return nil, err
        }
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        if res.StatusCode != http.StatusTooManyRequests && res.StatusCode != http.StatusServiceUnavailable {
            return res, nil
        }
        if attempt == maxAttempts {
            return res, nil // give up: let the caller decide
        }

        backoff := min(maxWait, base<<attempt) // 400ms, 800ms, 1.6s, … capped
        wait := rand.N(backoff)                // "full jitter": anywhere in [0, backoff)
        if s, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil && s > 0 {
            hint := time.Duration(s) * time.Second
            wait = hint + rand.N(hint) // the hint, plus up to the hint again: spread out
        }
        fmt.Printf("attempt %d: %d, waiting %v\n", attempt, res.StatusCode, wait.Round(time.Millisecond))
        io.Copy(io.Discard, res.Body) // drain + close so the connection can be reused
        res.Body.Close()

        select {
        case <-time.After(wait):
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }
}

Run against a test server that answers 429 with Retry-After: 1 twice, then 200 (each wait lands somewhere between 1 and 2 seconds, and differs every run):

attempt 1: 429, waiting 1.484s
attempt 2: 429, waiting 1.593s
final: 200 "ok" after 3.1 s

Where to enforce, and which limit to use

A request passes through several layers before it reaches your code, and each one can reject it. The further out you reject, the cheaper it is, but the less you know about the request:

Most real systems use several: a generous per-IP limit at the edge to absorb floods, and precise per-customer limits in the gateway or app. The outer layers keep garbage away from the inner ones; the inner ones are where fairness happens.

nginx: a leaky bucket in config

nginx's limit_req is a leaky bucket (§04). The zone defines the key and rate; each location chooses how much burst to tolerate:

# key = client address (binary form, compact); 10 MB of shared memory holds the counters
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;  # tolerate 20 extra at once, don't queue them
        limit_req_status 429;                 # the default is 503!
    }
}

Without burst, any request faster than 10 per second (one per 100 ms) is rejected. With burst=20, up to 20 extra requests are queued and released at the steady rate: the leaky bucket as a queue. With nodelay, those 20 are served immediately instead, while the bucket still drains at 10/s: the leaky bucket as a meter, which behaves like a token bucket of size 21. And set limit_req_status, because nginx answers 503 by default, which tells clients "I'm broken" instead of "you're too fast".

Envoy: local and global

Envoy (a proxy often used as an API gateway and inside a Service meshService meshA setup where every service instance gets its own small proxy (a sidecar, often Envoy) that handles load balancing, retries, TLS and metrics for all its outgoing and incoming calls, configured centrally.) offers two kinds of limit:

• Local rate limit: a token bucket inside each Envoy instance. Fast, no network call, but per instance, with the same "3 instances = 3× the limit" effect as §07. Note that its bucket refills in steps (tokens_per_fill every fill_interval), not continuously.
• Global rate limit: for each request, Envoy asks a separate rate-limit service over gRPCgRPCA popular framework for one service to call functions on another service over the network. It passes deadlines along automatically. (the reference implementation stores counters in Redis). Exact across the fleet, at the cost of a network hop.

- name: envoy.filters.http.local_ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
    stat_prefix: http_local_rate_limiter
    token_bucket:
      max_tokens: 100        # bucket size (burst)
      tokens_per_fill: 100   # added every fill_interval…
      fill_interval: 1s      # …so 100 per second, refilled in one step each second
    filter_enabled:
      default_value: { numerator: 100, denominator: HUNDRED }
    filter_enforced:
      default_value: { numerator: 100, denominator: HUNDRED }

The two filter_* settings are a nice touch: with filter_enabled at 100% but filter_enforced at 0%, Envoy counts and reports who would be limited without rejecting anyone. That's the "shadow mode" from §11.

Which kind of limit is this, anyway?

"Rate limiting" gets used for four different tools. They answer different questions, and Worker pools covers the other three in depth:

The trap is using a rate limit as overload protection. It protects you from one client, but when 10,000 well-behaved clients each stay under their limit and your database slows down, every rate limiter still says yes. As Worker pools §08 shows, a fixed rate even makes things worse when responses slow down. Use rate limits for fairness and contracts, and concurrency limits plus load shedding for protection. Stripe's well-known write-up of its rate limiters describes exactly this layering: a per-user request-rate limiter and a concurrent-request limiter, plus fleet-wide load shedders that drop low-priority traffic first when the system is overloaded.

Adaptive limits

A fixed limit is a guess. When you're the client of someone else's API, you can let the 429s tell you the real number using AIMDAIMDAdditive increase, multiplicative decrease: grow a limit slowly (+1) while things go well, cut it sharply (e.g. halve it) when you see trouble. TCP uses it to find a network's speed.: speed up a little after each success, halve your rate after each 429. It's how TCP finds a network's speed, and it fits the x/time/rate limiter directly, since SetLimit can be called while it's in use:

// adjust adapts our own sending rate to what the partner API tolerates (AIMD).
func adjust(lim *rate.Limiter, got429 bool) {
    cur := lim.Limit()
    if got429 {
        lim.SetLimit(max(1, cur/2))    // multiplicative decrease: halve, but not below 1/s
    } else {
        lim.SetLimit(min(50, cur+0.5)) // additive increase: +0.5/s per success, capped at 50/s
    }
}

Starting from 20/s, two successes and a 429 give 20.5 → 21 → 10.5. On the server side the same idea powers adaptive concurrency limits (Netflix's concurrency-limits library, Envoy's adaptive concurrency filter), which watch latency instead of 429s.

Staff-level: limiters in production

A round trip on every request

A Redis-backed limiter (§07) adds a network round trip to the hot pathHot pathThe code that runs on every single request. Anything added there — an extra network call, a lock — costs you on every request, so it has to be fast.: usually a fraction of a millisecond inside one data center, often more across zonesAvailability zone (AZ)One of a few separate data centers inside a cloud region (e.g. us-east-1a, 1b, 1c), with its own power and network, so one can fail without taking down the others., and much more on a bad day. At 50,000 requests per second that's also 50,000 Redis calls per second, and one giant customer's key can overload the single Redis node that holds it (a Hot key / hot spotHot key / hot spotA single key (or a narrow range) that gets far more traffic than the rest, like a celebrity's profile. Because one key always lives on one shard, that shard overloads while the others idle.). Three common ways to pay less:

1. Local first, then global. Each server keeps a local token bucket per key and only asks Redis occasionally: for example, it pushes "I used 37 tokens for key X" every 100 ms in one batch, and reads back the fleet-wide total. Most decisions are local and instant. The price is accuracy: during each 100 ms gap, servers can overshoot together.
2. Cache the "no". Once Redis says a key is over its limit until 12:00:05, remember that locally and reject further requests for that key without asking again. Abusive clients, the ones sending the most, become the cheapest to handle.
3. Approximate, shared counters. Some systems skip the central store entirely: servers exchange their counts every few hundred milliseconds via a gossip protocolGossip protocolA way for servers to share information by periodically telling a few random peers what they know, who pass it on in turn — like a rumour spreading. Cheap and robust, but always a little out of date. and each enforces "global count ≈ sum of what I've heard". Always slightly stale, never a single point of failure.

The honest framing for design reviews: exact, fast, available, pick two. A per-request Redis call is exact but adds latency and a dependency; a local limiter is fast and available but per instance; hybrids are fast and available but approximate. For most APIs, "within 10% of the limit" is perfectly fine; say so explicitly.

When the limiter itself is down

If Redis is unreachable, every limiter call fails. You must choose the fail modeFail open / fail closed (limiter)What a limiter does when its own storage (say Redis) is unreachable. Fail open: let every request through, unlimited. Fail closed: reject everything. Most APIs fail open, often to a local fallback limit. in advance:

• Fail open (allow everything): your API stays up, but unprotected. The usual choice for general API limits, since the limiter shouldn't be the reason you're down.
• Fail closed (reject everything): safe but means an outage. Reserved for limits that guard something dangerous: login attempts, money transfers, sending SMS (each costs real money and can be abused).
• Fall back to local: switch to a per-instance limiter set to limit ÷ number of servers. Roughly right, still protected, no dependency. Often the best of both.

Whatever you pick, give the limiter call a short timeout (a few milliseconds). A slow Redis that makes every request wait 2 seconds is worse than a dead one.

Fairness and noisy neighbours

Per-customer limits stop one customer from exceeding their allowance. They don't stop the sum of everyone's allowances from exceeding your capacity: if 100 customers each have 100/s and your system handles 5,000/s, you've sold more than you have (often deliberately, since not everyone is busy at once). When one customer has a spike, they can still slow everyone down. That's the Noisy neighbourNoisy neighbourIn a shared (multi-tenant) system, one customer whose heavy usage slows things down for everyone else sharing the same servers. problem in Multi-tenantMulti-tenant (tenant)One app serving many customer organisations (tenants) from shared infrastructure, like Slack workspaces or Shopify stores. Each tenant only sees its own data. systems.

The fix is to make queueing fair, not just admission. With Weighted fair queuingWeighted fair queuingA scheduling method that gives each customer (or class of traffic) its own queue and serves the queues in proportion to their weights, so a busy customer can't push everyone else to the back., each tenant gets its own queue, and workers take from the queues in turn (weighted by plan), instead of from one shared first-come-first-served line. A tenant that dumps 10,000 jobs only delays its own jobs; a small tenant's single job is picked up in the next round. Combine it with the per-tenant bulkheadsBulkheadSeparate pools (or limits) for different kinds of work, so one slow kind can't use up all the capacity. Named after the walls that keep a ship from sinking when one compartment floods. from Worker pools §10.

Limits during incidents

Limits are also an incident tool, and you want the knobs ready before you need them:

• Change limits without a deploy. Keep limits in config that reloads at runtime, so you can tighten one noisy key or one expensive endpoint in seconds.
• Exempt what keeps you alive. Health checks, your own internal traffic, and admin tools shouldn't share a bucket with customers.
• Watch the recovery. After an outage, every client retries at once. Buckets that start full (§04) let all of them burst at the same moment, right when your caches are cold. Consider lower burst sizes or a gradual ramp-up for a few minutes after recovery.
• Retries count too. Your own services retrying a struggling dependency are clients as well. A Retry budgetRetry budgetA cap on how much extra load retries may add (say, 10% of normal traffic), so retries can't multiply an outage. is a rate limit on retries.

Seeing what your limits do

A limiter you can't observe is a source of mysterious support tickets ("we keep getting 429s and we're nowhere near our limit!"). Record:

• A counterCounter (metric)A metric that only ever goes up: total requests, total rejections. Dashboards usually show how fast it grows (per second). of decisions, labelled by limit name, route and plan tier, with outcome allowed / rejected.
• Not labelled by raw user or API key: a label with a million possible values (high CardinalityCardinalityHow many distinct values something can take. A user_id column has high cardinality (millions of values), a country column low. In metrics, high-cardinality labels (like user IDs) create one time series per value and can overwhelm the monitoring system.) can overwhelm your metrics system. Log the top rejected keys instead, or keep a small "top 20 rejected keys" list.
• How close keys get to their limit (for example, the remaining tokens at decision time). A big customer that sits at 95% every day is about to start failing.

Alert on sudden rejection spikes for a large customer: it's usually their incident (a retry loop) or your bug (a misconfigured limit), and either way someone should talk to them before they open a ticket.

Rolling out a new limit safely

Turn a new limit on in Shadow modeShadow mode (dry run)Running a new rule (a rate limit, a validation, a new service) in “log only” mode: it records what it would have done, but changes nothing. You switch it on once the logs show it behaves as intended. first: it logs who would be rejected but lets everyone through. After a week of logs you know exactly which customers it would hit, and you can raise their limits or warn them before enforcing. Envoy's filter_enforced (§10) does this out of the box.

Testing limiters

• Unit tests with a fake clock (§06): exact, fast, no sleeps.
• Concurrency tests for anything shared: hammer the limiter from many goroutines or connections, then assert the total admitted is never above the limit. That's how §07's GET-then-SET bug shows up; a single-threaded test passes.
• Failure tests: stop Redis in a test environment and check that the fail mode is the one you chose.
• Bursty load tests: real traffic arrives in bursts, not at an even rate. A load test that sends a perfectly even 100/s will never show you the boundary burst (§02) or a too-small bucket.

Pitfalls gallery

n, _ := rdb.Get(ctx, key).Int()
if n >= limit {
    return false
}
rdb.Set(ctx, key, n+1, time.Minute)
return true

n, _ := rdb.Incr(ctx, key).Result() // atomic
return n <= limit
// anything more complex: a Lua script

Check yourself

Predict the answer before you click. Q1 is covered in §02, Q2 in §04, Q3 in §03, Q4 in §07, Q5 in §06, Q6 in §07 and Q7 in §08.

n, _ := rdb.Get(ctx, key).Int()
if n >= limit {
    return false
}
rdb.Set(ctx, key, n+1, time.Minute)
return true

lim := rate.NewLimiter(1, 1) // 1 per second, burst 1
lim.Allow()                  // uses the only token
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
err := lim.Wait(ctx)

Cheat sheet