Worker Pools & Backpressure

The 30-second version

Sooner or later every program gets more work than it can do at once: 10,000 photos to resize, 5,000 requests per second, a million rows to sync. This page is about doing that work without falling over.

Three numbers define every pool, and most production incidents with pools come from choosing one of them badly:

What you already know (and where it's taught)

This page builds on two others. If any of this is new, read those first — this page won't repeat them:

• Async / Await §11 shows why Promise.all over 10,000 items is a bug, and how to cap concurrency with a tiny mapLimit function in JavaScript or errgroup.SetLimit(n) in Go.
• Channels §07 builds a Go worker pool from scratch: N goroutines range over one jobs channel, a feeder closes it, a WaitGroup closes the results channel. §06 shows how unbuffered channels slow a fast stage down to a slow one — backpressure for free — and §08 shows how to stop goroutines with ctx.Done().

In short, you already know how to build a pool. This page is about running one: how big to make it, why it gets slow long before it's “full”, what to do when there's more work than capacity, and how to shut it down without losing anything.

Why “start everything at once” falls over

In Go, starting a GoroutineGoroutineA very lightweight task in Go, started with the go keyword. The Go runtime runs thousands of them on just a few OS threads. costs a few kilobytes, so go process(photo) 10,000 times looks harmless. In JavaScript, photos.map(process) starts 10,000 promises in a millisecond. The goroutines and promises themselves are cheap. What they hold while they run is not.

Four things that run out

Let's resize 10,000 photos. Each job downloads one photo, decodes and resizes it, uploads the result and saves a row in the database. That takes about 200 ms, but only ~20 ms of it is the CPU resizing; the other ~180 ms is waiting on the network. Here's what each running job holds:

Drag the slider and watch which one breaks first — and what happens to the speed:

More concurrency stops helping

Here's the step that surprises people. With 1 job at a time, each job takes ~200 ms, so you do 5 photos per second. With 10 at a time, 50 per second. With 20, 100 per second. So far, more is better, because each job spends ~90% of its time waiting on the network, and other jobs can use the CPU meanwhile. (20 jobs × 20 ms of CPU = 0.4 s of CPU per 0.2 s, i.e. about 2 cores busy — fine on most machines.)

But every job needs one of the 20 database connections. Once 20 jobs are running, the 21st can't go faster — it waits for a connection. So 1,000 jobs at once is exactly as fast as 20, and on top of that it uses 50× the memory and breaks the other three limits.

How many workers? CPU cores and Little's Law

CPU-bound jobs: about one worker per core

A CPU-boundCPU-boundWork where the CPU itself is the bottleneck — calculating, compressing, parsing. Only more or faster cores help. job (resizing an image, hashing a password, parsing a big JSON file) needs the CPU the whole time. A machine with 8 coresCPU coreOne independent “brain” inside the CPU. Each core runs exactly one thread at any instant, so a 4-core machine runs at most 4 threads truly at the same time. can run exactly 8 of those at the same instant. A 9th worker doesn't add speed — it just takes turns with the others, and each switch between them (a Context switchContext switchWhen the OS pauses one thread and resumes another on the same CPU core. It's cheap, but not free — thousands per second add up.) costs a little time and memory.

So for CPU-bound work, N ≈ number of cores:

// GOMAXPROCS(0) reads the current setting without changing it: how many
// threads may run Go code at once. It defaults to the number of cores
// (since Go 1.25, it also respects a container's CPU limit on Linux).
workers := runtime.GOMAXPROCS(0)

I/O-bound jobs: Little's Law

An I/O-boundI/O-boundWork that spends most of its time waiting for I/O (network, disk). Making the CPU faster wouldn't help much. job (calling an API, querying a database) spends most of its time waiting. While it waits, it's still in flight and still holds a worker. So the question becomes: how many jobs are waiting at the same time? There's a beautifully simple law for that.

Start with the supermarket. 2 customers arrive per minute, and each spends 5 minutes in the store. How many are inside at any moment, on average? Every minute, 2 more come in, and each stays for 5 minutes, so at any moment the last 5 minutes' worth of arrivals are still inside: 2 × 5 = 10 customers.

That's Little's LawLittle's LawL = λ × W: the average number of things in a system equals the rate they arrive times how long each one stays. 100 requests/s × 0.2 s each = 20 in progress at any moment.:

(λ is the Greek letter lambda, the traditional name for the arrival rate.) Here's the same idea as a picture — watch the count under the moving line:

It holds for any stable system — one where, on average, things leave as fast as they arrive, so the line isn't growing forever — whatever the pattern of arrivals: stores, servers, queues, pools. Watch the units: if λ is per second, W must be in seconds.

Worked example: sizing a pool for an API

You need to make 300 calls per second to a payment API, and each call takes 80 ms.

1. L = 300/s × 0.080 s = 24 calls in flight on average.
2. So you need more than 24 workers. With fewer, more calls arrive than finish, and the backlog grows forever. With exactly 24, every worker is busy 100% of the time, with no spare moment to catch up after a burst — in practice the backlog still grows (§05 shows why).
3. So add HeadroomHeadroomSpare capacity you keep on purpose — e.g. workers that are only 80% busy on average — so random bursts can be cleared quickly instead of piling up.: about 30 workers. Then 24 ÷ 30 = 80% of them are busy on average, which is the usual target (§05).
4. Finally, check what's downstream: if the API allows only 20 concurrent calls per client, 30 workers just earns you 429 errors. Your real ceiling is then 20 ÷ 0.08 s = 250 calls/s — you need a bigger quota, not more workers.

Little's Law works backwards too. A database Connection poolConnection poolA set of open database (or HTTP) connections that your program keeps and reuses instead of opening a new one per request. Pools usually have a size limit; when every connection is busy, the next caller waits for one to be returned. of 20 connections, with 5 ms queries, can do at most 20 ÷ 0.005 s = 4,000 queries/s. If a missing index makes queries take 50 ms, the ceiling drops to 400/s — same pool, 10× less capacity.

Jobs that do both

Most real jobs compute a bit and wait a bit. A classic rule of thumb from Brian Goetz's Java Concurrency in Practice:

On 8 cores, a job that waits 90 ms and computes 10 ms gives 8 × (1 + 9) = 80 workers. It's a starting point, not an answer.

The queue: bounded, unbounded, and “full”

Back at the supermarket: a line is great when ten people show up at once and the tills clear them in a few minutes. But if customers arrive faster than the tills can serve them all day, the line grows all day. Making the store bigger doesn't fix that — it just means more people are unhappy at closing time.

Three choices when the line is full

A bounded queue has a maximum length. When it's full and another job arrives, you have to pick a policy:

And the fourth option, an unbounded queue, is really “none of the above”: it never says no, so overload turns into memory growth and ever-longer waits.

Try it

One number tells you whether a pool can keep up: its load = arrivals per second ÷ capacity, where capacity = workers ÷ job time. The simulator starts with 4 workers and 400 ms jobs, so capacity is 4 ÷ 0.4 s = 10 jobs/s. The presets send 5, 9 and 13 jobs/s, which is a load of 50%, 90% and 130%. Above 100%, more arrives than can possibly be finished.

The simulator also shows p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them. latency: the time that 99% of jobs beat — in other words, how slow the slowest 1% are.

Start on unbounded and press the overload preset. Then switch the policy to block and reject and compare. Also try busy · 90%: there's enough capacity on average, but watch the p99.

Things to notice:

• Well under 100% load (try 50%), the queue is mostly empty. Latency is about the job time; the queue only absorbs short random bursts. At 90% it's already a different story — §05 explains why.
• Over 100% load, the queue grows without limit — no queue size fixes that. Only more workers, faster jobs, or less work do.
• Block doesn't make the waiting disappear. It moves it to the callers (stuck in submit()). That's exactly what you want if they can slow down, and terrible if they're users holding open connections.
• Reject keeps latency low for the jobs you do accept. The price is that some get an error — on purpose, quickly, while they can still retry elsewhere.

A small pool with all three policies

Here's a reusable pool. Compared with the one in Channels §07, it takes jobs over time (instead of a fixed list), has a bounded queue, and offers both a waiting Submit and a non-waiting TrySubmit. The Go version is longer because it's production-shaped: each job runs with its submitter's context, and it is safe to shut down (§09).

// A Job must watch ctx: if ctx is already cancelled when it starts, return at once.
type Job func(ctx context.Context)

type task struct {
    ctx context.Context // the submitter's context: its deadline and cancellation
    fn  Job
}

type Pool struct {
    jobs     chan task          // the queue: a buffered channel, so it's bounded
    wg       sync.WaitGroup
    ctx      context.Context    // cancelled if shutdown runs out of time (§09)
    cancel   context.CancelFunc
    quit     chan struct{}      // closed when shutdown starts (§09)
    shutOnce sync.Once          // makes a second Shutdown call harmless (§09)
    mu       sync.RWMutex       // guards closed (§09)
    closed   bool
    busy     atomic.Int64       // workers running a job right now (§10)
}

var (
    ErrFull   = errors.New("pool: queue full")
    ErrClosed = errors.New("pool: shut down")
)

func NewPool(workers, queueSize int) *Pool {
    ctx, cancel := context.WithCancel(context.Background())
    p := &Pool{
        jobs:   make(chan task, queueSize),
        quit:   make(chan struct{}),
        ctx:    ctx,
        cancel: cancel,
    }
    p.wg.Add(workers)
    for range workers {               // Go 1.22+: repeat `workers` times
        go p.work()
    }
    return p
}

func (p *Pool) work() {
    defer p.wg.Done()
    for t := range p.jobs {           // ends when jobs is closed AND empty
        p.run(t)
    }
}

// run gives the job a context that ends when EITHER its submitter gives up
// OR the whole pool is cancelled.
func (p *Pool) run(t task) {
    ctx, cancel := context.WithCancel(t.ctx)
    defer cancel()
    stop := context.AfterFunc(p.ctx, cancel) // Go 1.21+: pool cancelled → cancel ctx
    defer stop()
    if p.ctx.Err() != nil {           // pool already cancelled? AfterFunc calls cancel
        cancel()                      // in its own goroutine, so do it now, before fn
    }

    p.busy.Add(1)
    defer p.busy.Add(-1)
    t.fn(ctx)
}

// Submit WAITS while the queue is full (backpressure)…
func (p *Pool) Submit(ctx context.Context, fn Job) error {
    p.mu.RLock()                      // shutdown safety, explained in §09
    defer p.mu.RUnlock()
    if p.closed {
        return ErrClosed
    }
    select {
    case p.jobs <- task{ctx, fn}:
        return nil
    case <-ctx.Done():                // …but not forever: the caller can give up…
        return ctx.Err()
    case <-p.quit:                    // …and shutdown wakes it up too (§09)
        return ErrClosed
    }
}

// TrySubmit NEVER waits: full queue → error right away (load shedding).
func (p *Pool) TrySubmit(ctx context.Context, fn Job) error {
    p.mu.RLock()
    defer p.mu.RUnlock()
    if p.closed {
        return ErrClosed
    }
    select {
    case p.jobs <- task{ctx, fn}:
        return nil
    default:                          // ← the only difference from Submit
        return ErrFull
    }
}

Why latency explodes near 100% busy

UtilizationUtilizationThe fraction of time workers are busy. 80% utilization means that, on average, 8 of 10 workers are working at any moment. is how busy the workers are: arrival rate ÷ capacity. One worker, 100 ms jobs (so capacity is 10 jobs/s), 8 jobs arriving per second → 80% utilization. LatencyLatencyHow long one request or job takes from the moment it arrives until it's finished — including any time spent waiting in a queue. is what the job's sender experiences: time waiting in the queue plus the time a worker spends on it (the Service timeService timeHow long a worker actually spends on one job, not counting the time the job waited in the queue before a worker picked it up.).

If jobs arrived like clockwork — exactly every 125 ms — the worker would finish each one before the next arrived, and nobody would ever wait. Real traffic is random: sometimes three requests land in the same millisecond, sometimes nothing for half a second. Some jobs are quick, some are slow. Compare the two at the same average load:

Why randomness creates waiting

Picture the checkout again. A cashier who's busy 50% of the time is idle half the time, so a little bunch of customers clears quickly. A cashier who's busy 95% of the time has almost no idle time to catch up: when a bunch arrives, the line it creates takes ages to shrink, and the next bunch often arrives before it has.

The standard queueing model (random arrivals, random job times, one worker) puts a number on it:

With 100 ms jobs (the last column is for comparison: 16 workers sharing one queue, same utilization):

(In this one-worker model the p99 is always about 4.6 × the average — 4.6 comes from the maths of random waits, it's ln 100.)

For one worker, going from 50% to 90% busy is less than double the traffic, but 5× the average latency. Many workers sharing a queue push the explosion much closer to 100% — but it still comes. That curve, flat and then nearly vertical, is called a hockey stick. Drag along it:

These are numbers from a model, so treat them as approximate: real traffic is usually burstier than the model assumes (making things worse), and real job times are often less random (making things a bit better). The shape is what holds everywhere.

Three lessons from the chart

1. Don't plan for 100%. Run pools at roughly 70–80% busy at peak. The “idle” 20–30% isn't waste; it's what lets bursts clear quickly.
2. The tail grows faster than the average. The p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them. line — the slowest 1% of requests, the Tail latencyTail latencyThe latency of the slowest few requests — usually reported as p99 or p99.9. It's what your unluckiest users feel, and it grows much faster than the average under load. — is ~4.6× the average for one worker. Users notice the slow ones, and they add up: if a page makes 100 backend calls, each one is fast with chance 0.99, so all 100 are fast with chance 0.99¹⁰⁰ ≈ 0.37. That means 63% of page loads hit at least one p99-slow call.
3. More workers sharing one queue help a lot. Switch to 16 workers: at the same 80% utilization the average is barely above the job time. With many workers, it's very unlikely that all of them are busy at once. That's why one shared line beats a separate line per till — and why one pool of 16 beats four pools of 4 (unless you separate them on purpose, §10).

Backpressure end to end: semaphores and streams

BackpressureBackpressurePushing back on whoever sends you work when you're full — making them wait or rejecting them — instead of piling it up in memory. takes its name from plumbing: when a pipe narrows, pressure builds up behind it and slows the flow upstream. In software, it means a slow consumer automatically slows down whoever feeds it — instead of the work piling up in memory in between.

You've already seen the simplest version: in a Go pipeline, an unbuffered send waits until the next stage is ready, so the whole line runs at the speed of its slowest stage. This section covers the other tools.

Tool 1: a semaphore — “at most N at a time”

A SemaphoreSemaphoreA counter of free slots. acquire() takes a slot (waiting if there are none left); release() gives it back. A semaphore with 10 slots lets at most 10 tasks through at once. is a counter of free slots. Before starting a job you acquire a slot (waiting if none are free); when the job ends you release it. Unlike a pool, there are no long-lived workers: you start one goroutine or promise per job, but never more than N at once — and the loop that starts them pauses when all slots are taken. That pause is the backpressure.

sem := make(chan struct{}, 10) // 10 slots: a buffered channel used as a counter
var wg sync.WaitGroup

for _, row := range rows {
    sem <- struct{}{}          // acquire: waits here while 10 jobs are running ←
    wg.Add(1)
    go func() {
        defer wg.Done()
        defer func() { <-sem }() // release: free the slot when done
        save(row)
    }()
}
wg.Wait()                      // wait for the last ones

Go also has a ready-made weighted semaphore, for when jobs have different sizes. Instead of “10 jobs”, the budget can be “512 MB of memory”, and each job takes as many units as it needs:

import "golang.org/x/sync/semaphore"

sem := semaphore.NewWeighted(512)           // budget: 512 MB in total

for _, job := range jobs {
    // Take job.SizeMB units (an int64). Waits until that much is free,
    // or returns an error if ctx is cancelled first.
    if err := sem.Acquire(ctx, job.SizeMB); err != nil {
        wg.Wait()                           // don't abandon the jobs already running
        return err
    }
    wg.Add(1)
    go func() {
        defer wg.Done()
        defer sem.Release(job.SizeMB)       // give the units back
        process(ctx, job)
    }()
}
wg.Wait()

Tool 2: Node streams

A Node StreamStream (Node.js)An object that hands data over in chunks instead of all at once — a file being read, an HTTP body, a zip compressor. Readable streams produce chunks, writable streams consume them. passes data along in chunks — reading a 10 GB file never means holding 10 GB in memory. A writable stream (a file, a socket) has a small internal buffer, and its highWaterMarkhighWaterMarkThe buffer size at which a Node stream says “I'm full”: write() starts returning false. It's a signal, not a hard limit — the stream still accepts more if you ignore it. says how big that buffer should get: 64 KiB by default since Node 22 (16 KiB before), or 16 objects in object mode.

The whole protocol is two signals:

1. stream.write(chunk) returns false once the buffer has reached the highWaterMark. That means “please stop for now”.
2. When the buffer has emptied, the stream emits a 'drain' event: “OK, go again”.

The catch: write() returning false is only a request. If you ignore it, Node keeps accepting chunks and buffering them in memory. Try both:

Here's the same copy, three ways. The by-hand version shows the protocol; in real code, use pipeline():

import { createReadStream, createWriteStream } from "node:fs";
import { once } from "node:events";
import { finished } from "node:stream/promises";

const out = createWriteStream("copy.bin");
for await (const chunk of createReadStream("huge.bin")) { // throws on read errors
  if (!out.write(chunk)) {      // false = "my buffer is full"
    await once(out, "drain");   // ← wait until it empties (rejects if out errors)
  }
}
out.end();                      // no more data
await finished(out);            // wait until it's flushed to disk (or throw its error)

End to end: one unbounded hop ruins it

Backpressure is a chain. A request to your service passes through several hops, and pressure only reaches the source if every hop has a bound:

If any one hop is unbounded — a go func() per request, a p-limit queue with no cap, an in-memory list “just for now” — the pressure stops there. That hop quietly fills with work, its memory grows, its latency climbs, and nothing upstream finds out until it falls over.

Load shedding: saying no on purpose

Load sheddingLoad sheddingDeliberately rejecting some requests (e.g. HTTP 503) when overloaded, so the rest can still be served quickly. means deliberately rejecting some work when you're overloaded, so that the rest can still be served well. It sounds like giving up. It's actually how you keep serving anyone at all.

The surprising failure: busy, but useless

Clients don't wait forever. A browser, a mobile app, or another service gives up after a timeout — say 2 seconds — and shows an error or retries. Your server usually doesn't find out. So if a request waits in your queue for 3 seconds, then gets processed perfectly, nobody is there to receive the answer. That work was wasted.

That's the difference between ThroughputThroughputHow many jobs get finished per second. For a worker pool, the maximum is roughly workers ÷ time per job. (jobs finished per second) and GoodputGoodputUseful throughput: responses that reached a client who was still waiting. Work finished after the client gave up counts as throughput but not as goodput. (answers that reached someone still waiting). In overload, an unbounded queue keeps throughput at 100% while goodput falls toward zero. This has a name: Congestive collapseCongestive collapseWhen an overloaded system spends its time on work nobody is waiting for any more, so it stays busy at 100% while useful output drops to nearly zero..

Watch it happen, then try the two fixes:

Look closely at the unbounded version after the spike ends at 40 s: traffic is back to normal, but goodput stays at zero. During the spike, 15 jobs/s arrived and only 10/s could be finished, so 5 extra jobs per second piled up — 150 of them, 15 seconds of work. The workers have to chew through all of them, late, before anything new is answered in time. Back at normal traffic the spare capacity is only 10 − 8 = 2 jobs/s, so the backlog takes more than a minute to clear. The outage outlasts the spike.

The three fixes, from the widget

1. Bound the queue and reject when full (a 503). The cheapest and most predictable. Everything you accept is fast; everything you reject learns that immediately, while it can still retry elsewhere.
2. Don't start work that can't finish in time. Before a worker starts a job, check its deadline. If the client will have given up before the job could finish, skip it.
3. Both. Real systems use a bound for memory safety and a deadline check for usefulness.

Shedding at the front door

The best place to shed is as early as possible, before you've spent anything on the request — no body parsing, no database lookup. A middlewareMiddlewareA function that wraps an HTTP handler to run code before and/or after it — logging, auth, limits — without changing the handler itself. that caps requests in flight is a few lines:

// limit lets at most maxInFlight requests run at once; the rest get a fast 503.
func limit(maxInFlight int, next http.Handler) http.Handler {
    slots := make(chan struct{}, maxInFlight) // a semaphore (§06)
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        select {
        case slots <- struct{}{}:              // got a slot
            defer func() { <-slots }()
            next.ServeHTTP(w, r)
        default:                               // all busy → say no, right now
            w.Header().Set("Retry-After", "1") // "try again in 1 second"
            http.Error(w, "overloaded, try again", http.StatusServiceUnavailable)
        }
    })
}

// http.ListenAndServe(":8080", limit(100, mux))

The Go version uses the semaphore from §06 with a default case — the same trick as TrySubmit in §04. The JS version needs no lock around inFlight: JavaScript runs one piece of code at a time, so the check and the ++ can't be interleaved with another request.

A short wait instead of none. Rejecting the very instant all slots are busy can be too strict: a slot often frees up a few milliseconds later. A common middle ground is a queue measured in time, not length — wait up to, say, 50 ms for a slot, then shed. This replaces the select inside limit() above:

timer := time.NewTimer(50 * time.Millisecond)
defer timer.Stop()
select {
case slots <- struct{}{}:      // a slot freed up within 50 ms
    defer func() { <-slots }()
    next.ServeHTTP(w, r)
case <-timer.C:                // still full → shed
    w.Header().Set("Retry-After", "1")
    http.Error(w, "overloaded, try again", http.StatusServiceUnavailable)
case <-r.Context().Done():     // the client already gave up → do nothing
}

Skipping work whose client is gone

The second fix lives in the worker: check the deadline before starting. In Go, the request's ctx already carries both “the client hung up” and (if you add one) a deadline, and the pool from §04 passes it through to the job. Node's req has no signal built in, so the JS version builds one:

var errTooLate = errors.New("deadline too close to start")

// In the handler:
//   ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second) // the client's budget
//   defer cancel()
//   errCh := make(chan error, 1)  // the handler waits on this for the outcome
//   err := pool.Submit(ctx, func(ctx context.Context) { errCh <- handleTask(ctx, task) })

// task.Expected is how long this kind of task usually takes (a time.Duration).
func handleTask(ctx context.Context, task Task) error {
    // ctx.Deadline() reports the deadline, if the context has one.
    if d, ok := ctx.Deadline(); ok && time.Until(d) < task.Expected {
        return errTooLate // can't finish in time: don't waste a worker on it
    }
    return task.Run(ctx)
}

429 or 503?

Both are “not now” status codesHTTP 429 / 503Status codes for “not now”. 429 Too Many Requests: you (this client) are over your limit. 503 Service Unavailable: the server itself is overloaded or shutting down. Both may carry a Retry-After header., and both can carry a Retry-AfterRetry-AfterAn HTTP response header telling the client how long to wait before trying again, e.g. Retry-After: 1 means “in 1 second”. header saying how many seconds to wait:

• 429 Too Many Requests — you (this client, this API key) are over your limit. Other clients are fine. Used by rate limiters (§08).
• 503 Service Unavailable — I (the server) am overloaded or shutting down. Nothing about you in particular.

Either way, shedding only works if clients back off instead of retrying instantly. Instant retries turn every rejected request into two or three more — the opposite of shedding. Clients should retry with exponential backoff and jitter, within a retry budget (Async / Await §11 shows how).

Rate limits vs concurrency limits

At the supermarket: a rate limit is the guard letting in at most 10 customers per minute. A concurrency limit is the rule “at most 50 people inside”. On a normal day they look the same. Now the tills slow down — a card reader is broken. The guard keeps letting in 10 a minute, and the store fills up. The “50 inside” rule automatically lets fewer people in, because people are leaving more slowly.

Little's Law connects them

In flight = rate × time per job (§03). So for a dependency that normally answers in 100 ms:

With the rate limit, the struggling service gets 20× more open requests just when it can least afford them. With the concurrency limit, it automatically gets less traffic — exactly the relief it needs. That's backpressure again.

So when do you want a rate limit?

Rate limits answer a different question: fairness and quotas, not resource protection.

• Someone else's rules. An API you call says “100 requests per second per key”. You must stay under it, however fast they answer.
• Protecting yourself from one client. On your own API, a per-client rate limit stops one script from using up everyone's capacity. Over the limit → 429HTTP 429 / 503Status codes for “not now”. 429 Too Many Requests: you (this client) are over your limit. 503 Service Unavailable: the server itself is overloaded or shutting down. Both may carry a Retry-After header..
• Money. Each call to a paid service costs something; a rate limit is a spending cap.

In practice you often want both: a rate limit for the contract, a concurrency limit for safety.

The token bucket

The most common rate-limit algorithm is the Token bucketToken bucketA rate-limit algorithm: a bucket refills with tokens at a steady rate (say 10 per second) up to a maximum. Each request takes one token; no token, no request. The maximum allows short bursts.. A bucket holds up to B tokens and refills at r tokens per second. Each request takes one token; if the bucket is empty, the request is rejected (or waits). The refill rate is the long-run limit; the bucket size is how big a burst you allow after a quiet period.

import "golang.org/x/time/rate"

lim := rate.NewLimiter(50, 10) // refill 50 tokens/s; bucket holds 10 (max burst)
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(5)                  // …and never more than 5 running at once

for _, id := range ids {
    // Wait for a token. Fails if ctx is cancelled first.
    if err := lim.Wait(ctx); err != nil {
        if gerr := g.Wait(); gerr != nil {
            return gerr        // a call failed (and cancelled ctx): report that
        }
        return err             // our caller cancelled
    }
    g.Go(func() error { return call(ctx, id) })
}
return g.Wait()

Graceful shutdown: stop, drain, cancel

Your pool will be stopped far more often than it crashes: every deploy, every scale-down, every machine replacement. A Graceful shutdownGraceful shutdownStopping a program without dropping work: stop taking new work, finish (or hand back) what's in progress, then exit — within a time limit. means none of those lose or corrupt work.

Here's how it usually starts. A process manager like Kubernetes, stopping your podPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy., sends your process the signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>. SIGTERM (“please stop”), waits a grace period — 30 seconds by default — and then sends SIGKILL, which ends the process instantly, mid-write if need be. Everything below has to fit in those 30 seconds.

The three steps

At the supermarket at closing time: lock the door (no new customers), serve everyone already in line, and if it's really late, ask the last few to come back tomorrow.

1. Stop accepting. Close the HTTP listener and the pool's intake. New work goes to another instance.
2. Drain. Workers finish the jobs that are running and the ones still queued.
3. Cancel at the deadline. If draining takes too long, cancel the remaining jobs through their context or AbortSignal, wait for workers to stop, and exit — before SIGKILL does it for you.

Step through both endings with the Shutdown method for the pool from §04:

1func (p *Pool) Shutdown(ctx context.Context) error {
2    p.shutOnce.Do(func() {
3        close(p.quit)      // wake Submits waiting for room
4        p.mu.Lock()        // wait until no Submit is mid-send
5        p.closed = true    // later Submits return ErrClosed
6        close(p.jobs)      // 1. no new jobs
7        p.mu.Unlock()
8    })
9 
10    done := make(chan struct{})
11    go func() {
12        p.wg.Wait()        // 2. workers drain the queue
13        close(done)
14    }()
15    select {
16    case <-done:
17        return nil         // 3a. everything finished
18    case <-ctx.Done():
19        p.cancel()         // 3b. out of time: cancel
20        <-done             //     wait for workers to exit
21        return ctx.Err()
22    }
23}

The one trap: a late Submit

In Go, sending on a closed channel panics, and that crashes the whole program. Normally the front door closes first (srv.Shutdown waits for running HTTP handlers), so nothing calls Submit after close(p.jobs). But srv.Shutdown has a deadline too. If one slow handler is still running when it expires, it may call Submit after the pool closed its channel:

That's why the §04 pool has a closed flag guarded by a sync.RWMutex (a read-write LockLock (mutex)A “do not disturb” sign for a piece of data: only the task holding the lock may touch it; others wait their turn.), plus a quit channel that tells waiting Submits to give up. Here are the two halves:

func (p *Pool) Submit(ctx context.Context, fn Job) error {
    p.mu.RLock()        // many Submits may hold the READ lock at the same time…
    defer p.mu.RUnlock()
    if p.closed {
        return ErrClosed // …and after shutdown they get an error, not a panic
    }
    select {
    case p.jobs <- task{ctx, fn}:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    case <-p.quit:      // shutdown started while we waited for room:
        return ErrClosed // give up, and let go of the read lock
    }
}

func (p *Pool) Shutdown(ctx context.Context) error {
    p.shutOnce.Do(func() { // only the first call closes things
        close(p.quit)   // 1a. wake every Submit waiting on a full queue
        p.mu.Lock()     // 1b. the WRITE lock waits until no Submit holds the read lock,
        p.closed = true //     so no send can be half-way through while we close
        close(p.jobs)   // 1c. no new jobs (queued ones are still handed out)
        p.mu.Unlock()
    })

    done := make(chan struct{})
    go func() {
        p.wg.Wait()     // 2. drain: workers finish the queue, then exit
        close(done)
    }()
    select {
    case <-done:
        return nil      // everything finished in time
    case <-ctx.Done():
        p.cancel()      // 3. out of time: cancel every job's context
        <-done          //    and wait for the workers to notice
        return ctx.Err()
    }
}

Two promises this code relies on. First, jobs watch their ctx — otherwise <-done waits until SIGKILL. Second, a job still in the queue when p.cancel() runs is handed to a worker with an already-cancelled context (that's the if p.ctx.Err() != nil { cancel() } line in §04's run), so it returns at once. But remember Submit already returned nil when the job was queued: its caller hears about the cancellation only if the job reports its result back (like the errCh in §07). Otherwise the job just vanishes — so log it, or put it back on a durable queue (see below).

Wiring it up

func main() {
    // ctx is cancelled when the process receives SIGTERM or Ctrl+C.
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
    defer stop()

    pool := NewPool(8, 100)
    srv := &http.Server{Addr: ":8080", Handler: api(pool)}
    go func() {
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
    }()

    <-ctx.Done() // ← wait here until SIGTERM
    log.Println("shutting down…")

    // One time budget for everything, safely inside Kubernetes' 30 s.
    shutCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutCtx); err != nil { // 1. front door: no new requests
        log.Println("http shutdown:", err)
    }
    if err := pool.Shutdown(shutCtx); err != nil { // 2 + 3. drain, cancel at deadline
        log.Println("pool shutdown:", err)
    }
}

What “cancel” should mean for a job

Cancelling stops the work, but the job still needs to end up somewhere. Pick one deliberately:

• Retry later. If jobs come from a durable queue (a database table, Kafka, SQS), don't mark the job as done; another instance picks it up. This only works if the job is IdempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly. — safe to run twice — because it may have been half-done.
• Save progress. A long job (a video encode, a big export) writes a checkpoint so the next attempt resumes instead of restarting.
• Fail loudly. If neither is possible, record it as failed so a human or a retry system sees it.

An in-memory queue is lost when the process dies — gracefully or not. If losing queued jobs is unacceptable, the queue must live outside the process.

In production: bulkheads, priority, fairness, metrics

Bulkheads: separate pools for separate work

Suppose one pool of 6 workers handles both checkout requests (50 ms each) and report exports (30 s each). A user requests six reports. Now every worker is busy for 30 seconds, and checkouts wait behind them:

The fix is a BulkheadBulkheadSeparate pools (or limits) for different kinds of work, so one slow kind can't use up all the capacity. Named after the walls that keep a ship from sinking when one compartment floods.: give each kind of work its own limit, so a flood of one kind can only exhaust its own share. It can be separate pools, separate semaphores, or separate connection pools for the same database. This is the one good reason to have several smaller pools instead of one big one (§05 showed why one big shared queue is otherwise better).

Priority, and its trap: starvation

Sometimes you'd rather keep one pool and let urgent jobs go first. The usual tool is a Priority queuePriority queueA queue that hands out the most important item first instead of the oldest. Usually built on a heap, so adding and removing are fast even with many items., or in Go, two channels and a “check the urgent one first” loop:

for {
    select {                   // 1. urgent work first, if there is any…
    case job := <-high:
        job(ctx)
        continue
    default:                   //    …none waiting → fall through
    }
    select {                   // 2. otherwise take whichever comes first
    case job := <-high:
        job(ctx)
    case job := <-low:
        job(ctx)
    case <-ctx.Done():
        return
    }
}

The first select with default peeks at high without waiting. Only when it's empty do we wait on both. (A single select on both channels wouldn't work: when both are ready, Go picks one at random.)

The trap: if urgent work never stops arriving, low-priority jobs never run. That's StarvationStarvationA task that is ready to run never gets its turn, because others keep getting the resource first. The system as a whole makes progress; that one task doesn't.: a job that is ready to run but never gets a turn, because something else is always chosen first. Two common fixes:

• Reserve capacity. Low-priority work may use at most 2 of 8 workers; or high-priority work always has 2 workers to itself.
• Aging. The longer a job waits, the higher its priority becomes, until it's served.

Fairness: one customer shouldn't fill the queue

On a service shared by many customers (tenants), one of them submits a 50,000-job batch. With one first-in-first-out queue, everyone else's jobs wait behind all 50,000 — Head-of-line blockingHead-of-line blockingOne slow item at the front of a queue holds up everything behind it, even items that would be quick.. Two fixes:

• Per-tenant limits: no tenant may have more than, say, 20 jobs queued or 5 running. Beyond that, they get 429s.
• Per-tenant queues served in turn: workers take one job from tenant A, then B, then C, then A again (round robin). A big batch just makes its own tenant's line long.

What to measure

Every pool should export a handful of numbers. Here they are, roughly in order of how early they warn you:

Performance engineer Brendan Gregg's USE method sums it up for any resource: Utilization (how busy it is), Saturation (how much extra work is waiting because it's busy — queue depth and wait time), Errors (rejections, failures).

// SubmitTimed records how long each job waited in the queue and how long it ran.
// `metrics` stands for your metrics library (Prometheus, OpenTelemetry, …).
func (p *Pool) SubmitTimed(ctx context.Context, fn Job) error {
    enqueued := time.Now()
    return p.Submit(ctx, func(ctx context.Context) {
        metrics.Histogram("pool.wait_ms", time.Since(enqueued).Milliseconds())
        start := time.Now()
        defer func() {
            metrics.Histogram("pool.service_ms", time.Since(start).Milliseconds())
        }()
        fn(ctx)
    })
}

// ReportGauges records the gauges every 5 s until ctx is cancelled.
// Start it once, next to NewPool:  go p.ReportGauges(ctx, 8)  // 8 = its workers
func (p *Pool) ReportGauges(ctx context.Context, workers int) {
    tick := time.NewTicker(5 * time.Second)
    defer tick.Stop()
    for {
        select {
        case <-tick.C:
            metrics.Gauge("pool.queue_depth", float64(len(p.jobs))) // items waiting
            metrics.Gauge("pool.utilization", float64(p.busy.Load())/float64(workers))
        case <-ctx.Done():
            return
        }
    }
}

Pitfalls gallery

app.get("/report", async (req, res) => {
  const pool = new Pool(10);       // a fresh limit per request…
  const rows = await Promise.all(
    ids.map((id) => pool.run(() => load(id)))
  );
  res.json(rows);
});

const pool = new Pool(10, 200);    // one limit for the whole process

app.get("/report", async (req, res) => {
  try {
    const rows = await Promise.all(
      ids.map((id) => pool.run(() => load(id)))
    );
    res.json(rows);
  } catch (err) {                  // e.g. "pool: queue full" under load
    res.status(503).json({ error: "busy, try again" });
  }
});

pool := NewPool(2, 100)
for _, order := range orders { // 2+ orders → both workers
    pool.Submit(ctx, func(ctx context.Context) {
        done := make(chan struct{})
        pool.Submit(ctx, func(ctx context.Context) {
            defer close(done)
            charge(ctx, order)  // same pool!
        })
        <-done // both workers wait here; the
               // sub-jobs never get a worker
    })
}

outer := NewPool(2, 100)
payments := NewPool(4, 100) // separate pool
for _, order := range orders {
    outer.Submit(ctx, func(ctx context.Context) {
        done := make(chan struct{})
        err := payments.Submit(ctx, func(ctx context.Context) {
            defer close(done)
            charge(ctx, order)
        })
        if err != nil {
            return // queue full, shut down or cancelled
        }
        select {
        case <-done:       // charged
        case <-ctx.Done(): // we were cancelled: stop waiting
        }
    })
}

sem <- struct{}{}                 // take a slot
for attempt := 0; attempt < 5; attempt++ {
    if err = call(ctx); err == nil {
        break
    }
    time.Sleep(backoff(attempt))  // sleeping… with the slot
}
<-sem

for attempt := 0; attempt < 5; attempt++ {
    sem <- struct{}{}             // hold the slot only
    err = call(ctx)               // while actually working
    <-sem
    if err == nil {
        break
    }
    time.Sleep(backoff(attempt))  // sleep without it
}

Check yourself

Predict the answer before you click. Q1 is covered in §03, Q2 in §05, Q3 in §04, Q4 in §11 (pitfall #4), Q5 in §06, and Q6 in §08.

// An API handles 400 requests per second.
// Each request takes 50 ms on average.
// On average, how many requests are in flight at once?

// One worker. Jobs take 100 ms on average (random).
// 9 jobs arrive per second, at random moments.
// Roughly what's the AVERAGE time from arrival to done?

jobs := make(chan Job, 2)
jobs <- a
jobs <- b       // the queue is now full; no worker is reading

select {
case jobs <- c:
    fmt.Println("queued")
default:
    fmt.Println("rejected")
}

g := new(errgroup.Group)
g.SetLimit(2)                 // pool of 2
for _, order := range orders {
    g.Go(func() error {
        sub := new(errgroup.Group) // a NEW group, with no limit
        sub.Go(func() error { return charge(order) })
        return sub.Wait()
    })
}
g.Wait()

const out = fs.createWriteStream("copy.bin");
for await (const chunk of hugeDownload) {
  out.write(chunk);   // return value ignored
}

// A downstream service normally answers in 100 ms.
// You allow 50 requests per second to it (a rate limit).
// It slows down to 10 s per request.
// How many of your requests are in flight now?

Cheat sheet