Load Balancing

The 30-second version

Your users type one address, like shop.example.com. Behind it there might be one server, or forty. They can't tell, and that's the point: the load balancer hides how many machines there are and which ones are up right now.

Try it. Requests (the dots) arrive on the left. Click a server to break it or fix it, and add a server to see the work spread out:

Why every real system has one

What you'll learn

The rest of this topic answers four questions, in this order:

1. Where does the balancing happen? In DNS, at the connection level (“L4”), at the HTTP level (“L7”), or inside the client itself (§03).
2. Which server gets the next request? Round robin, least connections, “power of two choices” and consistent hashing. You'll race them in a simulator and see why the simplest one can make your slowest users wait dozens of times longer (§04–§05).
3. How does it know who's alive, and how do you deploy safely? Health checks, draining, sticky sessions and long-lived connections (§06–§08).
4. What goes wrong in production? Client IPs, timeouts, retry storms, and the load balancer itself breaking (§09–§12).

Primer: just enough networking

Each of these has its own deep-dive in this area. Here's the 3-minute version, which is all this topic needs.

Four words, one analogy

Think of sending a parcel to a company in a big office building.

The verb is the HTTP methodHTTP methodThe verb at the start of an HTTP request saying what you want to do: GET (read), POST (create/submit), PUT/PATCH (update), DELETE (remove). (GET = read, POST = send data), and the labelled extra lines are headersHTTP headerA Name: value line sent before the body of an HTTP request or response, carrying metadata — Content-Type: application/json, Authorization: Bearer …, Cache-Control: max-age=60.. One connection can carry many requests, one after another. That detail will matter a lot in §08.

Two more pieces show up in every request:

• DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. is the phone book. Your browser asks “what's the address of shop.example.com?” and gets back an IP address.
• TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it. is a sealed envelope. With HTTPS, everything on the connection is encrypted, so only the two ends can read the request. Anything in between sees scrambled bytes.

Where the load balancer sits

Here's the whole trip for one request. Step through it:

The key detail: DNS gives the client the load balancer's address, not a server's. The client opens a connection to the load balancer. The load balancer then opens its own connection to one of the backends, which usually have private addresses (like 10.0.1.12) that the internet can't reach at all. A middleman like this is called a Reverse proxyReverse proxyA proxy that sits in front of your servers (nginx, Envoy, a cloud load balancer): clients connect to it, and it forwards each request to one of your backends..

See it for yourself with curl -v

curl is a command-line HTTP client; -v (verbose) prints the conversation. Lines starting with > are what you sent, < is what came back. Here it's talking to the tiny load balancer you'll build in §10, running on port 8080 (output trimmed; the ← notes are ours):

$ curl -v http://localhost:8080/cart
* Connected to localhost (127.0.0.1) port 8080      ← the connection (to the LB)
> GET /cart HTTP/1.1                                ← the request: method + path
> Host: localhost:8080                              ← headers…
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK                                   ← the response from the LB
< Content-Length: 45
< Content-Type: text/plain; charset=utf-8
<
hello from :9002 (X-Forwarded-For=127.0.0.1)        ← body, written by backend :9002

curl only ever talked to port 8080. It has no idea that a server on port 9002 wrote the answer. Run it again and a different backend replies.

Where the balancing happens: DNS, L4, L7 and the client

Network software is described in layers, each wrapped around the next like envelopes inside envelopes. The numbers come from a classic model (the TCP/IP topic explains it properly); for load balancing you only need three of them:

A load balancer is named after the deepest layer it reads. There are four common places to put one.

1. DNS: hand out several addresses

The simplest trick: when someone looks up shop.example.com, the DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. server answers with several IP addresses, often in a rotating order. Each client picks one, so traffic spreads out. This is DNS round robin.

It's nearly free, but it's coarse. DNS answers are cached for minutes (by browsers, phones and DNS servers along the way), so if a server dies, clients keep trying its address until their cached copy expires. And plain DNS has no idea how busy each server is. Use it to spread users between whole data centers, not between individual servers. More in DNS.

2. Layer 4: forward connections

A Layer 4 (L4) load balancerLayer 4 (L4) load balancerA load balancer that works at the TCP/UDP level. It only sees IP addresses and ports, picks a backend once per connection, and copies bytes back and forth without understanding them. Examples: AWS NLB, Linux IPVS. works at the connection level. When a new TCPTCPTransmission Control Protocol: delivers a stream of bytes between two programs reliably and in order, resending anything lost. HTTP/1.1, HTTP/2 and most databases run on it. connection arrives, it looks only at the outside of the envelope (source and destination IP and port, the 5-tuple5-tupleThe five values that identify one connection: protocol (TCP/UDP), source IP, source port, destination IP, destination port. An L4 load balancer hashes these so every packet of a connection goes to the same backend.), picks a backend once, and then shovels every byte of that connection to the same backend.

• ✅ Very fast: it never parses requests, so it can handle millions of packetsPacketOne small chunk of data sent across a network on its own, with a header saying where it's from and where it's going. Big messages are split into many packets. per second.
• ✅ Works for anything on TCP or UDPUDPUser Datagram Protocol: sends individual packets with no connection, no ordering and no resending. Fast and simple; used by DNS, video calls, games and QUIC (HTTP/3).: databases, game servers, video calls, not just HTTP.
• ❌ Blind to the content. With HTTPS it can't even read the path; it just sees encrypted bytes.
• ❌ One decision per connection. If a connection carries 10,000 requests, they all go to the same backend (§08 shows why that hurts).

Examples: AWS Network Load Balancer (NLB), Google's Maglev, the Linux kernel's IPVS (used by some KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call. setups), HAProxy in TCP mode.

3. Layer 7: forward requests

A Layer 7 (L7) load balancerLayer 7 (L7) load balancerA load balancer that understands the application protocol (usually HTTP). It reads each request's method, path, headers and cookies, and can pick a different backend for every request. Examples: nginx, Envoy, HAProxy, AWS ALB. goes all the way in. It accepts the connection itself, decrypts the TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it. (§09), reads each HTTP request, and picks a backend for every single request. It keeps its own pool of connections to the backends.

• ✅ Route by content: /api/* to the API servers, /images/* to a file server, Host: admin.example.com to the admin app, a canary=1 cookie to the new version (a canary releaseCanary releaseSending a small slice of real traffic (say 1%) to a new version first, watching its errors and latency, then increasing the share if it looks good.).
• ✅ Per-request tricks: retry a failed request on another backend, add headers, compress, cache, Rate limitRate limitA cap on how many requests are allowed per unit of time — e.g. 100 per second. It says nothing about how many may be running at once..
• ❌ More work per request, and it must hold your TLS certificates.
• ❌ HTTP-shaped: it has to understand the protocol (HTTP/1.1, HTTP/2, gRPC, WebSockets).

Examples: nginx, Envoy, HAProxy (HTTP mode), Traefik, Caddy, AWS Application Load Balancer (ALB).

Compare them on real traffic. Pick a request and a balancer type, and see which parts each one can read and where it sends the request:

4. Client-side: the caller balances itself

Sometimes there's no box in the middle at all. The client gets the list of backend addresses (from DNS or a service registryService registryA live list of which instances of each service are running right now and at which addresses. Clients and load balancers look it up instead of hard-coding addresses.), opens connections to several of them, and picks one per request itself. gRPC clients work this way inside many companies.

• ✅ No extra hop, no extra machine to run or to fail.
• ❌ Every client, in every language, needs the balancing logic and must keep its list up to date. And you have to trust the clients to behave.

Side by side

Picking a server: the algorithms

A worked example: round robin with one slow server

Round robinRound robinTaking turns in a fixed order: server 1, 2, 3, then 1, 2, 3 again. Simple and fair when every server and every request is the same size. takes turns: 1, 2, 3, 4, 5, 6, 1, 2, … It sounds perfectly fair. Let's put numbers on it.

• You have 6 servers. Five are new; one is an old machine that works at 30% of the speed of the others.
• Say a new server can handle 100 requests per second. The old one handles 30. Total capacity: 5 × 100 + 30 = 530 req/s.
• Traffic is 370 req/s, only 70% of capacity. Sounds safe.
• Round robin gives every server 1/6 of the traffic: 370 ÷ 6 ≈ 62 req/s each.
• The old server gets 62 requests per second but can finish only 30. Every second, 32 more pile up in its queue. The queue never shrinks, so waits grow from seconds to minutes.

One in six requests lands on that server, so your p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them. (how slow the slowest 1% of requests are) is decided entirely by its ever-growing queue. Five fast servers each get 62 of the 100 requests they could handle, so they sit almost 40% idle while users of the slow server time out. Round robin assumes every server is equally fast and every request is equally big. In real systems, neither is true.

The contenders

Race them

The simulator sends the same 5,000 requests through each algorithm. Arrivals are random, and so are request sizes: most are small but 1 in 20 is six times bigger, like real traffic. Each server works on 2 requests at a time and queues the rest. The bars show each algorithm's p99; the panel below shows how many requests each server was juggling over time.

Try these recipes:

1. 6 equal servers, 70% load. Even with identical machines, round robin's p99 is roughly 2–3× worse than least connections. Why? Request sizes vary. Round robin keeps feeding a server that's stuck on two big requests; least connections notices it's busy and skips it.
2. One slow server. Round robin and random explode (look at the scale: seconds, then tens of seconds) because they keep sending the slow server its full share. Weighted round robin is fine if you set the weights right. The load-aware algorithms adapt without being told.
3. Many load balancers, stale info. Now flip the switch. Big systems run dozens of balancer instances, and each one sees the servers' load late. (The simulator models this as balancers acting on counts that are 300 ms old.) Least connections and EWMA collapse: every instance sees the same “least busy” server and they all pile onto it at once. That's called HerdingHerdingMany senders all choosing the same “best” target at once because they share the same (often out-of-date) information — so the best target is instantly overloaded and the problem moves around.. Power of two choices gets somewhat worse (about +50% at 70% load) but stays far ahead of the rest.

Why “pick two at random” works so well

It looks like a compromise between random (cheap, dumb) and least connections (smart, needs to see everything). It's much better than a compromise.

• Compared to random: random's problem is the unlucky server that gets five requests in a row. With two choices, a request only lands on a busy server if both random picks are busy. That's rare, and it gets rarer very fast as the server gets busier. Mathematicians showed that the busiest server's overload drops exponentially compared to one random choice (Mitzenmacher, 1996).
• Compared to least connections with stale data: least connections sends every request to the one server that looked emptiest. With P2C, that server only wins when it happens to be one of the two picks, so with 6 servers it gets about 1 in 3 requests, not all of them. Herding can't happen.
• It's cheap. Two lookups instead of scanning every server, and no shared state to keep perfectly fresh.

That's why it's a default in a lot of modern software: Envoy's LEAST_REQUEST policy picks between two random hosts by default, nginx has random two least_conn, and HAProxy's random balancing makes two draws by default.

The code: counting in-flight requests, then P2C

Every load-aware algorithm needs one number per backend: how many requests are in flight right now. Add one when you send, subtract one when the response is done.

type Backend struct {
	active atomic.Int64            // requests in flight right now
	proxy  *httputil.ReverseProxy  // forwards requests to this backend (§10)
}

// p2c: pick two different backends at random, keep the less busy one.
func p2c(backends []*Backend) *Backend {
	if len(backends) == 1 {
		return backends[0] // only one choice (and IntN(0) below would panic)
	}
	i := rand.IntN(len(backends))     // 0 … n-1
	j := rand.IntN(len(backends) - 1) // 0 … n-2
	if j >= i {
		j++ // skip i, so the two picks are different
	}
	a, b := backends[i], backends[j]
	if a.active.Load() <= b.active.Load() {
		return a
	}
	return b
}

func (p *pool) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	b := p2c(p.backends)
	b.active.Add(1)        // ← one more in flight…
	defer b.active.Add(-1) // ← …one fewer when this function returns
	b.proxy.ServeHTTP(w, r)
}

In Node there's only one thread running your JavaScript, so a plain b.active++ is safe. The "close" event on the response fires when it has been fully sent or the client hung up, so the count always comes back down.

Which one should I use?

Same key, same server: consistent hashing

When “same key → same server” matters

• Caches. Each server keeps a local cache. If user 42's requests always reach server B, B's cache already has user 42's data: a fast hit. Spread randomly, every server would need its own copy of everyone.
• Stateful connections. A chat room whose members must all be on the same server, or a game match.
• Data that lives on one machine. Databases split their data the same way; that's sharding, a later topic that reuses everything on this page.

You can't use load to decide here, because the choice has to be the same every time. Instead you turn the key into a number with a hash function and let the number decide. (Here a hash is just a function that scrambles a string into a big, evenly spread number; the same string always gives the same number.)

The obvious way, and why it breaks

The obvious way is server = hash(key) % N: divide by the number of servers and take the remainder. With 4 servers, a key whose hash is 42 goes to server 42 % 4 = 2.

Now add a fifth server. The formula becomes % 5:

Over many keys, going from 4 to 5 servers moves about 80% of them. For a cache, that means 80% of lookups suddenly miss, all at once, and all those misses hit your database at the same moment. Adding capacity just caused an outage.

The ring

Consistent hashingConsistent hashingA way to map keys (user IDs, cache keys) to servers so that adding or removing one server moves only about 1/N of the keys, instead of reshuffling almost all of them as hash(key) % N does. fixes this with a picture. Bend the number line into a circle (a hash ring).

1. Put each server on the ring at the position hash("server-name").
2. Put each key on the ring at hash(key).
3. A key belongs to the first server you meet going clockwise.

Add a new server and it lands somewhere on the ring. It takes over only the keys on the arc just before it (keys that used to belong to its clockwise neighbour). Every other key stays put. Remove a server and only its keys move, to its neighbour.

One point per server gives lumpy arcs: one server might own 45% of the ring by bad luck. The fix is virtual nodesVirtual node (vnode)One of many points a single server gets on a consistent-hash ring. With ~100 points per server, each server's share of keys evens out, and a removed server's keys spread over all the others.: give each server many points (hash("B#0"), hash("B#1"), …). With ~100 points each, every server owns many small arcs that add up to a fair share. And when a server leaves, its keys spread over all the others instead of dumping on one neighbour.

Play with it. Each coloured tick on the ring is a server point; each dot is a key, coloured by the server it belongs to:

Things to try: with 1 point per server, look at the per-server counts (very uneven). Raise it to 100. Then add server E and compare “keys moved” with what % N would have moved.

The code

type Ring struct {
	points []uint32          // sorted positions on the ring
	owner  map[uint32]string // position → server name
}

func NewRing(servers []string, vnodes int) *Ring {
	r := &Ring{owner: map[uint32]string{}}
	for _, s := range servers {
		for i := range vnodes {                     // i = 0, 1, …, vnodes-1
			p := hash32(s + "#" + strconv.Itoa(i)) // "api-1#0", "api-1#1", …
			r.points = append(r.points, p)
			r.owner[p] = s
		}
	}
	slices.Sort(r.points)
	return r
}

// Get walks clockwise from the key's position to the next server point.
func (r *Ring) Get(key string) string {
	h := hash32(key)
	i := sort.Search(len(r.points), func(i int) bool { return r.points[i] >= h })
	if i == len(r.points) {
		i = 0 // fell off the end: wrap around to the start of the ring
	}
	return r.owner[r.points[i]]
}

// hash32: MD5 scrambles even near-identical names ("api-1#0", "api-1#1") well.
// (Not for security here, just spreading.) Take the first 4 bytes as a number.
func hash32(s string) uint32 {
	sum := md5.Sum([]byte(s))              // "crypto/md5": 16 bytes
	return binary.BigEndian.Uint32(sum[:4]) // "encoding/binary": bytes → uint32
}

We ran both versions with servers api-1 … api-4, 100 virtual nodes each, and 10,000 keys user:0 … user:9999. Both use MD5, so they agree exactly: each server got 22–27% of the keys (perfect would be 25%), and adding api-5 moved 18% of them (perfect would be 1/5 = 20%). Hashing is a little lumpy, so you get “about 1/N”, not exactly.

The catch: hashing ignores load

Hash routing is loyal, not smart. If one key is very popular (a celebrity's profile, a viral product), its server melts while the others idle, and the hash will keep sending it there. Two common fixes:

• Bounded loads (from a 2016 paper by Google researchers; used by HAProxy and others): each server gets a cap of, say, 125% of the average load. If a key's server is over its cap, walk further clockwise to the next one that isn't.
• Hash only where you need it. Use P2C for most traffic and hashing only for the routes that truly benefit from affinity.

Health checks: knowing who's alive

Two ways to find out

Active checks are the balancer knocking on each door. Every few seconds it sends a small request, like GET /healthz, to every backend and expects a quick 200 OK. A backend that fails is taken out of rotation; when it passes again, it's put back.

Passive checks, usually called Outlier detectionOutlier detection (passive health check)Watching the real traffic instead of sending test requests: if one backend returns, say, 5 errors in a row, the load balancer ejects it for a while, then lets it back in to try again., are the balancer watching real traffic. If a backend returns 5 errors in a row, or suddenly gets much slower than its peers, it's ejected for a while (say 30 seconds, longer each time it happens again) and then let back in on probation.

You want both. Active checks find dead servers even when no traffic is flowing. Passive checks catch the sneaky case where /healthz happily says “ok” but every real request fails, for example because one disk filled up or one dependency's credentials expired.

Thresholds: don't panic over one bad answer

Networks drop the odd packet, and a server can pause for a moment (a garbage-collection pause, a noisy neighbour on the same machine). If one failed check removed a server, you'd constantly yank healthy servers in and out. That's FlappingFlappingA server that keeps getting marked healthy, then unhealthy, then healthy again, every few seconds. Each switch moves traffic around and can reset connections., and it's bad: every flip moves traffic around and can cut open connections.

So checks use thresholds:

• Unhealthy threshold: mark down only after N failures in a row (e.g. 3).
• Healthy threshold: mark up only after M passes in a row (e.g. 2).
• Interval and timeout: check every few seconds; a check that takes longer than the timeout counts as a failure.

The trade-off is simple arithmetic: with a 5-second interval and a threshold of 3, a server that dies suddenly still gets traffic for up to ~15 seconds. That's what retries (§09) and passive checks are for.

Play with it. Server B is flaky: it fails about a third of its checks at random. Server C you can crash yourself. Watch the state changes (“flips”) and the number of user requests that hit a broken server:

Things to try: set both thresholds to 1 and watch B flap. Raise them to 3/2 and it settles. Crash C and count how many requests fail before it's removed; then turn on outlier detection and crash it again.

Liveness vs readiness

KubernetesKubernetesA system that runs your containers across many machines: it decides where each one runs, restarts them when they crash, rolls out new versions, and gives each group a stable name other services can call. and similar platforms ask two different questions, and mixing them up causes real outages:

var ready atomic.Bool // true once warmed up; false again while shutting down

http.HandleFunc("/livez", func(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("ok")) // if we can answer at all, we're alive
})
http.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		http.Error(w, "not ready", http.StatusServiceUnavailable) // 503 → take me out
		return
	}
	w.Write([]byte("ok"))
})

The health check that takes everything down

Here's a mistake that has caused many outages. Someone thinks: “if the database is down, this server can't really work, so /healthz should check the database too.”

1. The database has a 20-second hiccup (a failover, a slow query, a full connection pool).
2. Every instance's health check fails at the same moment, because they all share that one database.
3. The balancer removes every backend. Users now get errors for all pages, even the ones that never touch the database (the home page, static assets, cached data).
4. If it was a liveness check, it's worse: Kubernetes restarts every instance. They all start cold, all at once, and all open fresh connections to the database that was just recovering. Down it goes again.

A 20-second database blip became a full outage. In the simulator, switch the check to “process + database” and press database blip to watch it happen.

Balancers have a safety net for this too, called Fail openFail openWhen every backend looks unhealthy, the load balancer ignores the health checks and sends traffic to all of them anyway, on the theory that a broken check is more likely than a total outage.. If every backend looks unhealthy, AWS's Application Load Balancer sends traffic to all of them anyway. Envoy goes further with a panic threshold: once fewer than 50% of hosts look healthy (by default), it ignores health status and balances over all of them. Turn on “fail open” in the simulator and replay the database blip.

Sticky sessions, stateless servers and zero-downtime deploys

The logged-out user

A classic first bug with a load balancer: a user logs in, clicks a link, and is logged out again. What happened?

1. The login request went to server A. A stored “user 42 is logged in” in its own memory and gave the browser a session ID.
2. The next request went to server B. B has never heard of that session ID. Logged out.

The server was keeping state (information that has to survive between requests) in its own memory. There are two ways out.

Fix 1: sticky sessions

Sticky sessionsSticky session (session affinity)Sending all requests from the same user to the same backend, usually by setting a cookie that names the backend or by hashing the client's IP address. tell the balancer to send each user back to the same server:

• By cookie: on the first response, the balancer adds a cookie naming the backend (AWS ALB calls it AWSALB; HAProxy can insert a SERVERID cookie). Later requests carry the cookie, and the balancer obeys it.
• By client IP: hash the client's IP address to pick a server (nginx ip_hash). No cookie needed, but whole offices, mobile carriers and schools share one public IP (that's NATNATNetwork Address Translation: a router rewrites private addresses (like 192.168.1.20) to its one public address on the way out, and back on the way in. Why your laptop isn't directly reachable from the internet.), so thousands of users can land on one server.

It works, but it fights everything else in this topic:

• Uneven load. The balancer can't move a heavy user to an idle server.
• A dead server takes its users' sessions with it. Everyone on it is logged out.
• Deploys are slow or painful. To replace a server you must either wait for its sessions to expire or drop them.

Fix 2 (better): stateless servers

Make servers statelessStateless serverA server that keeps nothing about a user between requests in its own memory: sessions, carts and uploads live in a shared database, cache or object store. Any instance can serve any request.: move the state somewhere every instance can reach.

• Sessions go into a shared store, such as Redis (a fast in-memory key–value database) or your main database, keyed by session ID.
• Or into the client itself as a signed cookie or token (the server signs it so it can tell if anyone tampered with it).
• Uploaded files go to object storage (S3 and friends), not the local disk.

Now any instance can serve any request, and every algorithm from §04 works. Use stickiness only as a performance hint (warm caches; that's §05's consistent hashing), never as the thing that keeps users logged in.

Deploys: the race nobody tells you about

A Rolling deployRolling deployReplacing the instances of an app a few at a time: start a new one, wait until it's ready, drain and stop an old one, repeat. The service stays up throughout. replaces instances a few at a time: start a new one, wait until it's ready, stop an old one, repeat. The “stop an old one” step is where errors sneak in.

When an orchestratorOrchestratorA system (like Kubernetes) that starts, stops, restarts and moves your app's instances for you, and tells the load balancer which ones exist. like Kubernetes stops your instance (in Kubernetes, a podPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy.), two things start at the same time:

• your process receives the signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>. SIGTERM (“please shut down”), and
• the load balancer is told to remove the instance.

But the balancer finds out later: it has to notice a failed health check, or receive an update that travels through a few systems. For a few seconds, it keeps sending new requests to an instance that may already have stopped listening. Those users get connection errors.

The fix is Connection drainingConnection drainingTaking a backend out of rotation gently: the load balancer stops giving it new requests but lets the ones already running finish (up to a time limit) before the instance is stopped., done in the right order:

1. Fail readiness first. On SIGTERM, make /readyz return 503, but keep serving.
2. Wait long enough for the balancer to notice (at least check interval × unhealthy threshold, plus a margin).
3. Stop accepting new connections and finish in-flight requests, with a deadline.
4. Exit, before the orchestrator's hard kill (SIGKILL, by default 30 seconds after SIGTERM in Kubernetes).

var draining atomic.Bool
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
	if draining.Load() {
		http.Error(w, "draining", http.StatusServiceUnavailable) // ← LB sees 503
		return
	}
	w.Write([]byte("ok"))
})
srv := &http.Server{Addr: ":8080", Handler: mux}

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()
go func() {
	if err := srv.ListenAndServe(); err != http.ErrServerClosed {
		log.Fatal(err)
	}
}()

<-ctx.Done()                // SIGTERM arrived
draining.Store(true)        // 1. fail readiness
time.Sleep(10 * time.Second) // 2. give the LB time to notice

shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil { // 3. stop accepting, wait for in-flight
	log.Printf("forced shutdown: %v", err) // deadline hit: some requests were cut off
}
// 4. return from main → the process exits

We ran both against a request that takes 2 seconds: after SIGTERM, /readyz answers 503 straight away, the slow request still completes with 200, and only then does the process exit.

Long-lived connections: when balancing stops balancing

One connection, thousands of requests

Opening a connection costs time (a HandshakeHandshakeA short exchange of messages at the start of a connection where both sides agree it's open and settle the details (versions, keys, starting numbers) before any real data flows. or two, see §02), so modern clients keep connections open and reuse them:

• HTTP/1.1 Keep-aliveKeep-alive (connection reuse)Leaving a connection open after a request finishes so the next request can reuse it, skipping the cost of a new handshake. reuses one connection for requests one after another.
• HTTP/2 and gRPC go further with MultiplexingMultiplexingSending several independent conversations over one shared channel at the same time, by cutting them into small labelled pieces and interleaving them. HTTP/2 multiplexes many requests over one TCP connection.: many requests travel over one connection at the same time. A gRPC client typically opens one connection to a service and keeps it for its whole life.
• WebSockets turn a connection into a two-way channel that stays open as long as the user has the page open: minutes, hours, days.

Remember from §03: an L4 balancer picks a backend once per connection. So if a client opens one HTTP/2 connection and sends 10,000 requests per second down it, one backend gets all 10,000. The balancer never gets a say again.

The scale-up that does nothing

This bites hardest during a traffic spike:

1. Load goes up. Your autoscalerAutoscalingAutomatically adding instances when a metric (CPU, requests per second, latency) is high and removing them when it's low. adds two new backends. 🎉
2. Every existing client already has its connection open to an old backend. Nobody reconnects.
3. The new backends receive almost nothing: only the occasional brand-new client. The old ones stay overloaded.
4. Metrics still say “overloaded”, so the autoscaler adds even more idle servers.

Watch it. Six clients each hold one long-lived connection and send steady traffic. Press scale up, then try the two fixes:

The fixes

1. Balance per request with an L7 proxy. Put an HTTP/2-aware L7 balancer (Envoy, nginx, HAProxy, a cloud L7 balancer) in front. It accepts each client's connection, unpacks the individual requests, and spreads them over its own connections to all backends. New backends get their share the moment they're added.

2. Client-side balancing. The client opens a connection to every backend and picks one per request (§03). gRPC supports this with its round_robin policy, as long as it can find out the list of backends (usually from DNS).

3. Cap connection age. Have the server politely close each connection after a while, with some randomness, so clients reconnect and land on a fresh choice. This is Max connection ageMax connection ageA server or proxy setting that politely closes each long-lived connection after a set time (say 5 minutes, with some randomness), so clients reconnect and get re-balanced.:

srv := grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{
	MaxConnectionAge:      5 * time.Minute,  // ask clients to reconnect after ~5 min
	MaxConnectionAgeGrace: 30 * time.Second, // give in-flight calls time to finish
}))

grpc-go adds ±10% JitterJitterA small random amount added to a delay or timer — a retry wait, a reconnect, a connection's maximum age — so that many clients don't all act at the same instant and stampede a server. to MaxConnectionAge on its own, so thousands of connections opened at the same moment don't all reconnect at the same moment later. For plain HTTP/1.1 servers, the same trick is to send Connection: close on a response every so often, which tells the client to open a new connection for its next request.

WebSockets: you can't split a conversation

A WebSocket carries one user's live session, so you can't re-balance it message by message. What you can do:

• Pick well at connect time: least connections (§04) rather than round robin, because each connection is a long-term commitment.
• Make reconnecting cheap and normal. Clients should reconnect automatically, with a random delay (jitter), and resume where they left off.
• Drain on deploy. Before stopping a server, close its WebSockets with close code 1012 (“service restart”) a few at a time, so clients reconnect elsewhere gradually instead of all at once.

At the proxy: TLS, client IPs, timeouts and retries

TLS termination

With HTTPS, the bytes on the wire are encrypted (see TLS). An L7 balancer has to read the request, so it must decrypt it. That's TLS terminationTLS terminationDecrypting TLS at a proxy or load balancer in front of your app, so the app behind it receives plain HTTP (or a second, separate TLS connection).: the balancer holds the CertificateCertificate (TLS)A small signed file that says “this public key belongs to example.com”, vouched for by a trusted certificate authority. Browsers check it so they know they're talking to the real site. and private key, unwraps the encryption, and reads plain HTTP.

• ✅ One place to manage certificates (and renew them), instead of on every backend.
• ✅ The balancer can route by path and header, and backends skip the encryption work.
• ⚠️ Traffic from the balancer to the backends is now plain text unless you re-encrypt it. Inside a private network that's often accepted; for sensitive data, or where the rules require it, re-encrypt (often with each side proving its identity: mutual TLS).

The alternative is TLS pass-through at L4: the balancer forwards the encrypted bytes untouched and the backend decrypts. You keep end-to-end encryption, but lose all L7 features. (An L4 balancer can still peek at one unencrypted field in the TLS hello, the server name the client asked for, to choose a pool.)

Who is the client?

Every connection carries a source IP. But the backend's connection comes from the balancer, so every request seems to come from 10.0.0.5. Your logs, rate limits and fraud checks all see the same address.

At L7, the balancer adds a header: X-Forwarded-ForX-Forwarded-ForAn HTTP header a proxy adds to say who the original client was: X-Forwarded-For: 203.0.113.7. Each proxy appends the address it received the request from. Only trust the entries your own proxies added.. Each proxy along the way appends the address it received the request from, so after two proxies it might read X-Forwarded-For: 203.0.113.7, 10.0.0.5. (There's also a standard Forwarded: header, RFC 7239, with the same idea and a stricter format.)

At L4, a proxy can't add HTTP headers because it doesn't parse HTTP. Instead it can send the PROXY protocolPROXY protocolA small header (invented by HAProxy) that an L4 load balancer puts in front of a TCP connection's bytes, saying the original client IP and port, because an L4 balancer can't add HTTP headers.: one short line in front of the connection's first bytes, like PROXY TCP4 203.0.113.7 10.0.0.5 51234 443. The backend (nginx, HAProxy, Envoy all support it) reads that line first. Both sides must agree to use it, or the backend sees garbage.

Step through it with two balancers and a forged header:

Timeouts: two sides of the proxy

A proxy has two connections to manage, so it has timeouts on both sides: how long to wait for the client, how long to wait for a backend to accept a connection (connect timeout), and how long to wait for its response. Two in particular cause real trouble.

Idle timeouts that don't match. The balancer keeps a pool of idle Keep-aliveKeep-alive (connection reuse)Leaving a connection open after a request finishes so the next request can reuse it, skipping the cost of a new handshake. connections to each backend and reuses them. The backend also closes idle connections after a while. If the backend gives up first, the balancer can send a request down a connection at the exact moment the backend closes it. The request fails, and the user sees a seemingly random 502 Bad Gateway.

A concrete, very common case: AWS's ALB keeps idle connections for 60 seconds by default; Node's HTTP server closes them after 5 seconds (server.keepAliveTimeout). Rule: the backend's idle timeout must be longer than the balancer's.

const server = http.createServer(app);
server.keepAliveTimeout = 65_000; // > the LB's 60 s idle timeout
server.headersTimeout = 66_000;   // keep it above keepAliveTimeout (older Node versions misbehaved otherwise)

We checked the defaults on Node 22: keepAliveTimeout is 5000 ms and headersTimeout is 60000 ms.

In Go, the equivalent is http.Server{IdleTimeout: 65 * time.Second}. (Go's default, when IdleTimeout and ReadTimeout are both zero, is no idle limit at all.)

Request timeouts that don't nest. If the balancer gives up after 30 s but the backend keeps working for 60 s, the backend wastes 30 s on an answer nobody will read. Make timeouts shrink as you go deeper: the outer layer waits longest. gRPC's deadlines pass this budget along automatically (gRPC).

Retries: helpful once, deadly in layers

A balancer is a great place to retry (send a failed request again): if a backend refuses the connection, try another one; the user never notices. Retrying is always safe when the request never reached the backend. Retrying after it did reach it (a 503, a timeout) is only safe if the request is IdempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly., like a GET; retrying a POST /payments can charge someone twice.

The danger is multiplication. Suppose the browser makes up to 3 attempts, the balancer makes up to 3 attempts per request it receives, and the service makes up to 3 attempts at its database call: 3 × 3 × 3 = 27 queries for one click. When the database slows down, each user click can turn into those 27 database queries, at exactly the moment the database can least afford them. That's a Retry stormRetry stormWhen a struggling service gets even more traffic because every caller retries its failed requests — often at several layers at once — so a small slowdown becomes a full outage.:

The fixes, covered in depth in Worker Pools: load shedding:

• Retry at one layer, usually the one closest to the failure, not at every layer.
• Use a Retry budgetRetry budgetA cap on how much extra load retries may add (say, 10% of normal traffic), so retries can't multiply an outage.: retries may add at most, say, 20% extra traffic. When failures exceed that, stop retrying and fail fast. Envoy and Linkerd support budgets directly.
• Back off: wait longer before each new attempt, and add random JitterJitterA small random amount added to a delay or timer — a retry wait, a reconnect, a connection's maximum age — so that many clients don't all act at the same instant and stampede a server. to that wait, so retries don't arrive in synchronized waves.

Build one: a tiny load balancer in Go and Node

This is what you saw answering curl in §02. It does round robin (§04) and skips unhealthy backends, using an active check every 2 seconds plus a passive one: a failed request marks its backend down immediately (§06).

package main

import (
	"io"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
	"sync/atomic"
	"time"
)

type backend struct {
	url     *url.URL
	proxy   *httputil.ReverseProxy
	healthy atomic.Bool // read and written by many goroutines
}

type pool struct {
	backends []*backend
	next     atomic.Uint64
}

// pick does round robin, skipping backends marked unhealthy.
func (p *pool) pick() *backend {
	n := uint64(len(p.backends))
	for range n { // try each backend at most once
		b := p.backends[p.next.Add(1)%n]
		if b.healthy.Load() {
			return b
		}
	}
	return nil // everyone is down
}

// ServeHTTP makes *pool an http.Handler: it runs for every incoming request.
func (p *pool) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	b := p.pick()
	if b == nil {
		http.Error(w, "no healthy backends", http.StatusServiceUnavailable)
		return
	}
	b.proxy.ServeHTTP(w, r) // forward the request, stream the response back
}

func newBackend(raw string) *backend {
	u, err := url.Parse(raw)
	if err != nil {
		log.Fatal(err)
	}
	b := &backend{url: u}
	b.proxy = &httputil.ReverseProxy{
		Rewrite: func(pr *httputil.ProxyRequest) {
			pr.SetURL(u)       // send it to this backend
			pr.SetXForwarded() // add X-Forwarded-For / -Host / -Proto
		},
		ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
			log.Printf("%s failed: %v", u, err)
			b.healthy.Store(false) // passive check: a failed request marks it down
			http.Error(w, "bad gateway", http.StatusBadGateway)
		},
	}
	b.healthy.Store(true)
	return b
}

// healthLoop is the active check: GET /healthz on every backend, every interval.
func (p *pool) healthLoop(interval time.Duration) {
	client := &http.Client{Timeout: time.Second} // a hung backend counts as down
	for {
		for _, b := range p.backends {
			resp, err := client.Get(b.url.String() + "/healthz")
			ok := err == nil && resp.StatusCode == http.StatusOK
			if err == nil {
				io.Copy(io.Discard, resp.Body) // read it all, so the connection can be reused
				resp.Body.Close()
			}
			if was := b.healthy.Swap(ok); was != ok {
				log.Printf("%s healthy=%v", b.url, ok)
			}
		}
		time.Sleep(interval)
	}
}

func main() {
	p := &pool{}
	for _, raw := range []string{"http://127.0.0.1:9001", "http://127.0.0.1:9002", "http://127.0.0.1:9003"} {
		p.backends = append(p.backends, newBackend(raw))
	}
	go p.healthLoop(2 * time.Second)

	srv := &http.Server{Addr: ":8080", Handler: p, ReadHeaderTimeout: 5 * time.Second}
	log.Fatal(srv.ListenAndServe())
}

Run it

You need three backends. Any server that answers /healthz will do; this one prints its port and the X-Forwarded-For header it received:

import http from "node:http";
const port = Number(process.argv[2]);
http.createServer((req, res) => {
  if (req.url === "/healthz") return res.end("ok\n");
  res.end(`hello from :${port} (X-Forwarded-For=${req.headers["x-forwarded-for"]})\n`);
}).listen(port, "127.0.0.1");

Start it on ports 9001, 9002 and 9003, start the balancer, then (Go version shown; we ran both):

$ for i in 1 2 3 4; do curl -s localhost:8080/; done
hello from :9002 (X-Forwarded-For=127.0.0.1)
hello from :9003 (X-Forwarded-For=127.0.0.1)
hello from :9001 (X-Forwarded-For=127.0.0.1)
hello from :9002 (X-Forwarded-For=127.0.0.1)    ← round robin

# now stop the backend on :9003 …
$ curl -s -w " [%{http_code}]\n" localhost:8080/
bad gateway
 [502]                                          ← one unlucky request finds out (passive check fires)
$ curl -s localhost:8080/
hello from :9001 (X-Forwarded-For=127.0.0.1)    ← :9003 is skipped from now on

The Go version starts at :9002 because next.Add(1) returns 1 the first time; the Node version starts at :9001. With Node, the request that hits the dead backend may come one or two requests later, depending on where the round robin is.

That one 502 is the cost of finding out. A real balancer would retry that request on another backend, because the connection was refused, so it never reached the dead server and is safe to retry (§09).

What's missing compared with nginx or Envoy

Plenty, and it's a good checklist of everything in this topic: health-check thresholds instead of one-strike (§06), a smarter algorithm such as P2C (§04), retries with a budget, TLS termination and timeouts on both sides (§09), slow start for new backends (§11), metrics per backend, and stripping hop-by-hop headers in the Node version. In production, use a battle-tested proxy; write your own only to learn, or when you need logic no proxy offers.

Staff-level: load balancing at scale

Who balances the balancer?

You added a load balancer so no single server could take you down. But now every request goes through the balancer, so it is the Single point of failureSingle point of failure (SPOF)One component that, if it breaks, takes the whole system down with it — because everything depends on it and nothing can take its place.. Three standard answers, from small to huge:

1. Active-passive pair. Two balancer machines share a Virtual IPVirtual IP (VIP)An IP address that isn't tied to one machine. It can move to a standby machine in a second or so if the active one dies, so clients keep using the same address.. The active one owns it and keeps announcing “I'm alive” using VRRPVRRPVirtual Router Redundancy Protocol: two or more machines share a virtual IP. The active one sends “I'm alive” messages; if they stop, a standby takes over the IP. keepalived is the common Linux tool. (on Linux, usually with keepalived). If those announcements stop, the standby claims the IP within a second or two. Simple, but half your hardware sits idle, and connections in progress on the dead machine are lost.
2. Active-active with ECMP. The router in front knows several balancers can reach the same IP and spreads connections across all of them with ECMPECMPEqual-Cost Multi-Path routing: a router that knows several equally good next hops for an address spreads connections across all of them (by hashing the 5-tuple). Used to run many L4 balancers behind one IP. (it hashes the 5-tuple5-tupleThe five values that identify one connection: protocol (TCP/UDP), source IP, source port, destination IP, destination port. An L4 load balancer hashes these so every packet of a connection goes to the same backend., so a connection's packets all take the same path). Lose one balancer and the router stops using it. To keep existing connections alive when the set of balancers changes, every balancer uses the same consistent hash (§05) to map a connection to a backend, so whichever balancer a packet lands on picks the same backend. That's the design of Google's Maglev and similar systems at large companies.
3. Let the cloud do it. AWS ALB/NLB, Google Cloud Load Balancing and friends are already large distributed fleets behind one name. You still own their configuration (timeouts, health checks, draining), and their failures are rare but very wide.

Global: sending users to the nearest region

Above the data center sits global load balancing:

• GeoDNSGeoDNSAn authoritative DNS server that gives different answers depending on where the question seems to come from — e.g. a European IP to resolvers in Europe — to send users to a nearby data centre.: the DNS server answers shop.example.com with a European IP for European users. It's simple, but subject to DNS caching (§03), and it guesses location from the user's DNS resolver, which can be far from the user.
• AnycastAnycastAnnouncing the same IP address from many places at once. Routers send each packet to the nearest copy, so 1.1.1.1 or a CDN address is answered by a server in your region.: the same IP is announced from dozens of locations. Internet routing (BGP, the protocol internet providers use to exchange routes) delivers each user's packets to the nearest one. Failover happens at routing speed, without waiting for DNS caches. CDNs, public DNS resolvers and global cloud balancers work like this.

A typical big setup: anycast or GeoDNS → nearest region's L4 fleet → L7 proxies → services, with each tier health-checking the one below it.

Service mesh: a balancer next to every instance

In a Service meshService meshA setup where every service instance gets its own small proxy (a sidecar, often Envoy) that handles load balancing, retries, TLS and metrics for all its outgoing and incoming calls, configured centrally., every service instance gets its own small proxy (a SidecarSidecarA helper process (or container) that runs right next to your app and does one job for it, such as a proxy that handles TLS or a log shipper. They usually talk over a local socket., usually Envoy). All calls between services go through these proxies, which do client-side balancing (§03), retries, timeouts, TLS between services and metrics, all configured from one central place (the “control plane”).

• ✅ Consistent behaviour in every language; P2C and outlier detection for every call, with no extra hop to a central balancer.
• ❌ Two extra proxy hops per call (out of the caller's sidecar, into the callee's), memory and CPU for every sidecar, and a powerful new system to understand and debug. Newer designs move the proxy out of each pod and onto each machine to cut that cost.

Slow start: don't drown the new server

A freshly started instance has cold caches, cold connection pools and, on runtimes with a JIT compiler (one that optimises the hottest code while the program runs, as in Node and Java), code that hasn't been optimised yet. Its first requests are slow.

It gets worse with load-aware algorithms. Least connections sees the new server with zero in-flight requests and sends it everything. The cold server slows down, requests pile up, and you get a latency spike on every deploy and every scale-up.

Slow startSlow start (load balancer)Giving a freshly started backend only a small share of traffic at first and ramping it up over, say, 30–60 seconds, while its caches and JIT warm up. fixes this: ramp a new backend's weight from near zero to full over 30–60 seconds. Envoy, HAProxy (slowstart), nginx Plus and AWS target groups all offer it.

Zone-aware routing

Cloud regions are split into availability zonesAvailability zone (AZ)One of a few separate data centers inside a cloud region (e.g. us-east-1a, 1b, 1c), with its own power and network, so one can fail without taking down the others.. Sending a request to another zone adds a little latency and, on most clouds, costs money per gigabyte. So balancers prefer backends in the caller's own zone.

The catch is spill-over: if zone A has 10 callers but only 2 healthy backends, “stay local” would melt those 2. Good zone-aware routing (Envoy and Kubernetes' topology-aware routing both do this) stays local only while local capacity matches local demand, and spreads the excess to other zones otherwise.

Feedback loops

Balancers, health checks and autoscalers all react to each other, and some combinations spiral:

• The death spiral. Load rises → backends slow down → their health checks time out → the balancer removes them → the remaining backends get even more load → their checks time out too. Health checks must test “broken”, not “busy”. Busy servers should shed load (Worker Pools: load shedding), not disappear.
• The idle scale-up. Covered in §08: long-lived connections don't move to new instances, so the autoscaler keeps adding servers that get no traffic.
• The slow scale-up. New instances take a minute to boot, plus slow start. If the autoscaler looks at CPU every 60 s and adds 10% each time, a sudden 3× spike takes many minutes to absorb. Scale on a leading signal (requests per second, queue length) and keep HeadroomHeadroomSpare capacity you keep on purpose — e.g. workers that are only 80% busy on average — so random bursts can be cleared quickly instead of piling up..
• The cold-start herd. After a big failure every instance restarts at once, all with cold caches, all hitting the database. Stagger restarts and use slow start.

Pitfalls gallery

Check yourself

Predict the answer before you click. Q1 is covered in §03, Q2–Q3 in §04, Q4 in §05, Q5 in §06, Q6 in §08 and Q7 in §07 and §09.

Clients → L4 load balancer (TCP, port 443, HTTPS) → web servers

Goal: send /images/* requests to a separate image pool.

6 servers; one runs at 30% of the speed of the others.
Traffic = 70% of total capacity. Algorithm: round robin.

20 load-balancer instances. Each sees backend
connection counts refreshed once per second.
Algorithm: least connections.

cache := servers[hash(key) % len(servers)]
// we go from 10 cache servers to 11

// every instance:
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    if err := db.PingContext(r.Context()); err != nil { // shared Postgres
        http.Error(w, "db down", 503)
        return
    }
    w.Write([]byte("ok"))
})
// The database fails over, which takes 15 seconds.

gRPC clients → L4 balancer → 3 pods
Autoscaler adds 3 more pods. Ten minutes later…

<-ctx.Done()                         // SIGTERM arrived
srv.Shutdown(context.Background())   // stop listening right away
// the load balancer health-checks every 5 s, threshold 2

Cheat sheet