Deployment Strategies

The 30-second version

You've probably merged a pull request and seen it "go live" a few minutes later. Between those two moments, something had to stop the old program and start the new one on real servers, while real users were clicking. That swap is where most outages begin: Google's Site Reliability Engineering book reports that roughly 70% of their outages came from changes to a live system. The strategies in this topic exist to make that swap boring.

The one idea to keep

Deploying means the new code is running on your servers. Releasing means users actually get the new behavior. Old-school deploys do both at once, so every deploy is a gamble. Most of this topic is about pulling the two apart: deploy quietly, then release to a few users, watch, and widen, with a fast way back at every step.

The rest of the topic, in order: the words you need (§02), the strategies one by one (§03–§05), a simulator that runs them against a buggy release (§06), shadow traffic (§07), feature flagsFeature flagA runtime on/off switch (a config value) that decides whether new code runs, so you can ship code turned off and enable it gradually — or turn it off instantly if it breaks. (§08), automatic rollback (§09), the database and API rules that make all of this possible (§10–§11), and how to choose (§12).

Primer: what's running, and what can go wrong

The picture to keep in your head

Six words carry the whole topic:

• InstanceInstance (replica)One running copy of your app. A service usually runs several identical instances behind a load balancer, so one can be replaced or crash while the others keep answering.: one running copy of your app (a process, often inside a ContainerContainerA normal Linux process (or a small group of them) that the kernel fences off: it sees only its own files, processes and network, and gets a cap on how much CPU and memory it may use. Docker and Kubernetes run containers.). Running four of them means one can die, or be replaced, while three keep answering.
• Load balancerLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped.: the single address users connect to. It forwards each request to one of the instances. It's how you add, remove and swap instances without users noticing. Load Balancing explains how it picks one.
• Readiness checkReadiness check“Should this instance receive traffic right now?” If it fails, the instance is taken out of the load balancer but not restarted — e.g. while it starts up, warms caches or shuts down.: a URL like /readyz that each instance answers with 200 OK once it's started and able to serve. The load balancer only sends traffic to instances that pass it, so a half-started instance gets no users.
• Version: one specific, unchangeable build of your code, usually a Container imageContainer imageThe packaged, read-only file system a container starts from (built from a Dockerfile). It's shipped to registries and servers, so anyone who can pull it can read every file inside. tagged with the git commit, like shop:3f9c2a1. "v1" and "v2" in this topic mean two such builds. Because the build never changes, "go back to v1" means "run that exact image again". CI/CD §05 explains why you build once and deploy that same Build artifactBuild artifactThe output of a build that gets shipped, such as a binary, a zip of static files or a container image. Build once, deploy that same artifact everywhere. everywhere.
• RollbackRollback (deploy)Putting the previous version back after a bad deploy: redeploying the last good build, switching traffic back to the old copy, or turning a feature flag off.: putting the previous version back after a bad deploy. How fast you can do it is the main thing that separates the strategies.
• Error rateError rateThe share of requests that failed — a network error, a timeout or an error status like 500. 3 failures out of 1,000 requests = 0.3%.: the share of requests that fail. If v1 fails 0.3% of requests and v2 fails 5%, v2 is broken, even if its home page loads fine. This number is how you (or a robot) decide whether a deploy is going well.

Three ways a deploy hurts users

1. A gap. For a moment, nothing is ready to answer. Users see errors or a spinner. (§03 shows a real one: 527 failed requests in 3 seconds.)
2. A bad version. v2 starts fine but has a bug: 5% of checkouts fail, or one page is slow. The question is how many users meet it before you notice, and how fast you undo it.
3. Two versions at once. Unless you stop everything first, v1 and v2 will both be serving for a while. The same user might hit v2 on one click and v1 on the next. Both read and write the same database. This has a name, Version skewVersion skewDifferent versions of the same program (or of a client and its server) running at the same time — normal during deploys, and for months with mobile apps that users don't update., and §10–§11 are about surviving it.

Every strategy in this topic trades these three against each other and against cost. Keep that in mind and the rest is detail.

Recreate and rolling deploys

Recreate: turn it off and on again

A recreate deployRecreate deployThe simplest deploy: stop every instance of the old version, then start the new one. Easy and cheap, but users get errors in the gap while nothing is running. is what you'd do by hand on a single server: stop the old process, start the new one. Nothing could be simpler, and nothing is answering in between. How long is "in between"? As long as v2 takes to start: load its config, connect to the database, warm its caches.

We measured it. A Go program sent requests from 1,000 fake users to one v1 server, about 170 per second, and printed what came back each second. Each output line counts the answers received during that second: map[v1:176] means 176 requests were answered by v1, error:155 means 155 got no answer at all. At second 4 we stopped v1 and started v2, which takes 3 seconds to boot (measured on this machine):

t= 3s map[v1:176]
t= 4s map[error:155 v1:17]     ← v1 stopped; v2 still booting
t= 5s map[error:181]
t= 6s map[error:174]
t= 7s map[error:17 v2:160]     ← v2 ready
t= 8s map[v2:166]

527 failed requests in 3 seconds, every one of them a real user's click. On a busy service that's thousands of errors per deploy, and some of those requests were checkouts.

Recreate is still the right choice sometimes:

• Nobody is waiting: a dev or test environment, a nightly batch job, an internal tool used during office hours.
• Two versions must never run together: v2 changes a file format or a database in a way v1 can't handle, or only one instance may ever do a job (a single scheduler). Recreate is the one strategy with no "both at once" phase (§02).

In Kubernetes it's one line in the Deployment, the object that keeps N copies of your app running (more in Kubernetes):

spec:
  strategy:
    type: Recreate   # kill all old pods first, then create the new ones

Rolling: one at a time

A Rolling deployRolling deployReplacing the instances of an app a few at a time: start a new one, wait until it's ready, drain and stop an old one, repeat. The service stays up throughout. keeps the service up by never replacing everything at once:

1. Start one v2 instance.
2. Wait until its readiness checkReadiness check“Should this instance receive traffic right now?” If it fails, the instance is taken out of the load balancer but not restarted — e.g. while it starts up, warms caches or shuts down. passes, so the load balancer starts sending it traffic.
3. Stop one v1 instance, letting it finish the requests it's already handling (that's called draining).
4. Repeat until no v1 is left.

This is Kubernetes' default. It runs your app in podsPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy. (one pod ≈ one instance), and two numbers control the pace: maxSurgemaxSurge / maxUnavailableThe two knobs of a rolling update. maxSurge: how many extra pods above the desired count may exist during the update. maxUnavailable: how many of the desired pods may be missing or not ready. (how many extra pods may exist during the rollout) and maxUnavailablemaxUnavailableA Kubernetes rolling-update setting: how many pods may be missing (not ready) below the desired count during a rollout. 0 means capacity never drops. (how many may be missing). Step through a 4-replica rollout with maxSurge: 1, maxUnavailable: 0:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: shop
5spec:
6  replicas: 4
7  strategy:
8    type: RollingUpdate
9    rollingUpdate:
10      maxSurge: 1        # at most 4 + 1 = 5 pods
11      maxUnavailable: 0  # never fewer than 4 ready
12  template:
13    spec:
14      containers:
15        - name: shop
16          image: example.com/shop:v2  # was :v1
17          readinessProbe:
18            httpGet: { path: /readyz, port: 8080 }

The two knobs trade speed against spare capacity:

Two commands you'll see in every runbook (illustrative, not run here):

kubectl rollout status deployment/shop   # waits and prints progress until the rollout finishes or fails
kubectl rollout undo deployment/shop     # roll back: a rolling deploy to the previous version

What rolling deploys don't solve

Rollback is just as slow. rollout undo is another rolling deploy, pod by pod. If v2 is broken, users keep hitting it until the last v2 pod is gone.

A bad version reaches everyone. Kubernetes only checks that new pods pass readiness. A v2 that starts fine but fails 5% of checkouts passes every check and rolls out to 100%. Rolling deploys protect you from crashes, not from bugs.

v1 and v2 serve at the same time. During the rollout, consecutive requests from one user can land on different versions:

If v2 renamed a JSON field, or stores a new kind of value in the database, the v1 pods still running will choke on it. §10 and §11 show how to make each change safe for both versions.

Blue-green: two copies and a switch

The recipe

1. Blue runs v1 and gets all the traffic. Green is idle (or doesn't exist yet).
2. Deploy v2 to green. Take as long as you like: no user can reach it.
3. Test green through a private address, for example a Smoke testSmoke testA quick, broad check that the app starts and its main pages load and respond without crashing. It doesn't check that results are correct, only that nothing is obviously on fire. that logs in and loads the main pages.
4. Switch: tell the router to send everything to green.
5. Keep blue running for a while (minutes to hours). If anything looks wrong, switch back.
6. Once you trust v2, shut blue down. Next deploy, the colors swap roles.

Try the recipe, including the shortcuts that hurt:

Where the switch lives

The "switch" is whatever decides which servers get traffic:

• A load balancer's target list. A cloud load balancer has listeners (a port plus rules, like "HTTPS on 443") that forward to a target group (a list of servers). Pointing the listener at green's target group instead of blue's is one API call.
• A Kubernetes Service. A Service is a stable address that forwards to every pod carrying certain labels (more in Kubernetes). Point it at pods labelled color: green instead of color: blue, and traffic moves within seconds.
• DNS. Change shop.example.com to point at green's address. It's the easiest switch to set up and the worst one to rely on: resolvers and browsers keep the old answer until its TTLTTL (time to live, DNS)How many seconds a DNS answer may be cached before it must be asked for again. Set by the domain owner on each record. A TTL of 300 means “you may reuse this answer for up to 5 minutes”. runs out, and some ignore the TTL. Traffic drifts over for minutes or hours, and so does your rollback.

Build one: a version-switching proxy

A Reverse proxyReverse proxyA proxy that sits in front of your servers (nginx, Envoy, a cloud load balancer): clients connect to it, and it forwards each request to one of your backends. is a server that sits in front of your servers and forwards each request to one of them. Here's a complete one in Go that can send any share of users to v2. Blue-green is the special case where that share jumps from 0 to 100; §05 reuses the same proxy for canaries.

// (package main and imports omitted; the full file is ~60 lines)

// bucket maps a user ID to a number from 0 to 99. Same user, same bucket, every time.
func bucket(userID string) int64 {
	h := fnv.New32a()
	h.Write([]byte(userID))
	return int64(h.Sum32() % 100)
}

type router struct {
	stable, next *httputil.ReverseProxy
	percent      atomic.Int64 // share of users (0–100) sent to the new version
}

func (rt *router) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	if bucket(r.Header.Get("X-User-ID")) < rt.percent.Load() { // ← the whole strategy
		rt.next.ServeHTTP(w, r)
		return
	}
	rt.stable.ServeHTTP(w, r)
}

func proxyTo(addr string) *httputil.ReverseProxy {
	u, _ := url.Parse(addr) // fixed, known-good address; error ignored for brevity
	return httputil.NewSingleHostReverseProxy(u)
}

func main() {
	rt := &router{stable: proxyTo("http://127.0.0.1:9001"), next: proxyTo("http://127.0.0.1:9002")}

	admin := http.NewServeMux() // a second, private port for operators
	admin.HandleFunc("POST /percent/{n}", func(w http.ResponseWriter, r *http.Request) {
		n, err := strconv.Atoi(r.PathValue("n"))
		if err != nil || n < 0 || n > 100 {
			http.Error(w, "want 0-100", http.StatusBadRequest)
			return
		}
		rt.percent.Store(int64(n)) // takes effect on the very next request
		fmt.Fprintf(w, "now sending %d%% of users to the new version\n", n)
	})
	go func() { log.Fatal(http.ListenAndServe("127.0.0.1:8081", admin)) }()

	log.Fatal(http.ListenAndServe(":8080", rt))
}

One shortcut to know about: the demo trusts an X-User-ID header, which any client can set, and every request without one hashes to the same bucket. A real router takes the ID from the logged-in session, or gives anonymous visitors a random ID in a CookieCookieA small name=value string a server asks the browser to store (with a Set-Cookie response header). The browser then sends it back on every later request to that site in a Cookie header — that's how a stateless protocol remembers you're logged in. on their first visit.

We started v1 on port 9001, v2 on 9002 and the proxy on 8080, pointed the same 1,000-user load loop at the proxy, then flipped it from a second terminal:

$ curl -X POST localhost:8081/percent/100      # at t ≈ 4 s: switch to green
now sending 100% of users to the new version
$ curl -X POST localhost:8081/percent/0        # at t ≈ 8 s: switch back
now sending 0% of users to the new version

t= 3s map[v1:174]
t= 4s map[v1:33 v2:142]   ← switched mid-second
t= 5s map[v2:177]
t= 7s map[v2:175]
t= 8s map[v1:143 v2:33]   ← rolled back mid-second
t= 9s map[v1:176]

Zero errors, both ways, because both versions were already running and ready. Compare that with the 527 errors of the recreate deploy in §03. (Measured on this machine; a real proxy like nginx or Envoy does the same job with a config reload instead of an admin endpoint.)

The fine print

• Double the servers. Green must handle full traffic the moment you switch, so for a while you pay for two complete copies. In the cloud that's usually only minutes of cost; on your own hardware it can be a real reason not to do it.
• One database. Blue and green almost always share the database, so v2 must not change it in a way v1 can't handle, or switching back will break v1. Same rule as rolling deploys (§10).
• Cold start. Green has empty caches and no warmed-up connections, so the first seconds after the switch are slower. Send it some test traffic first, or ramp up instead of jumping (that's a canary, §05).
• Long-lived connections stay put. A WebSocketWebSocketA long-lived two-way connection between browser and server, used for chat and live updates. It starts as a normal HTTP request, then the server takes over (“hijacks”) the raw connection. or streaming connection opened on blue stays on blue after the switch. Blue must keep running (or close them politely) until they're gone.
• Background jobs run twice. If the app also runs scheduled jobs, idle green will run them too, and both colors send the 9 a.m. email. Turn jobs off in the idle color.
• "All at once" means every user meets a bug at once. Blue-green makes undo instant; it doesn't make the Blast radiusBlast radiusHow much of the system (and how many users) a single failure or bad deploy can affect. Smaller is better. small. It works best combined with a canary step or a feature flag.

Canary: a few users first

The recipe

A typical Canary releaseCanary releaseSending a small slice of real traffic (say 1%) to a new version first, watching its errors and latency, then increasing the share if it looks good. schedule looks like this. The numbers vary by team; the shape doesn't:

The waiting at each step is called bake timeBake timeHow long a new version runs at a given step (say 5% of traffic) before you trust it enough to go further. Long enough for slow problems (memory leaks, nightly jobs) to show up.: long enough for problems to show up, short enough that deploys don't take all day. If any check fails, set the share back to 0%. Rollback takes one second, and only the users in the slice ever saw v2.

Running it

The proxy from §04 already does this: its percent setting is the canary share. We pointed the same load loop at it and stepped from 0% to 5% to 25% (measured on this machine):

t= 1s map[v1:173]
t= 2s map[v1:146 v2:6]     ← POST /percent/5
t= 3s map[v1:158 v2:9]     ≈ 5% of requests
t= 5s map[v1:161 v2:9]
t= 6s map[v1:127 v2:41]    ← POST /percent/25
t= 8s map[v1:130 v2:40]    ≈ 24%

Why split by user, not by request

The proxy doesn't pick v2 at random for each request. It runs the user's ID through a hash function to get a bucket from 0 to 99, and sends buckets below the percentage to v2. At 5%, users in buckets 0–4 always get v2 and everyone else always gets v1. Raise it to 25% and buckets 0–24 get v2: the first 5% stay in, and 20% more join.

That stickiness matters:

• A user doesn't flip between versions on every click, which would look broken (a button appears, then disappears).
• The damage is contained. If v2 breaks checkout, 5% of users have a bad time, rather than every user failing one checkout in twenty.
• You can find the affected users afterwards, because the bucket is reproducible.

For a purely internal API with no user-facing state, splitting per request is fine and gives you smoother statistics. Feature flags (§08) use the exact same bucket trick.

Three ways to split traffic

1. By instance count. Run 19 v1 instances and 1 v2 instance behind the same load balancer: about 5% of requests go to v2. A bare Kubernetes ServiceService (Kubernetes)A stable name and virtual IP inside a Kubernetes cluster that load-balances traffic to whichever pods currently match its label selector. can do only this (two Deployments behind one Service, sketched below); weights need an extra component such as a gateway or service mesh. It's coarse: with 4 instances, your smallest step is 25%.
2. By weight at a proxy. A proxy, cloud load balancer or Service meshService meshA setup where every service instance gets its own small proxy (a sidecar, often Envoy) that handles load balancing, retries, TLS and metrics for all its outgoing and incoming calls, configured centrally. sends exactly 1% to v2, however many instances each side has. This is Traffic splittingTraffic splitting (weights)Telling a load balancer or proxy to send a set share of requests to each version, e.g. 95% to v1 and 5% to v2, independent of how many instances each has., and it's what our Go proxy does.
3. By who the user is. Send a header or cookie value (employees, beta testers, one small country) to v2 first. Running the new version on your own staff first is called dogfooding.

The instance-count version in Kubernetes, for reference (illustrative):

# Two Deployments; one Service selects pods by "app: shop" only, so it
# spreads requests over all 20 pods: 19 stable + 1 canary ≈ 5% to v2.
kind: Deployment
metadata: { name: shop-stable }
spec:
  replicas: 19
  template:
    metadata: { labels: { app: shop, track: stable } }
    spec: { containers: [{ name: shop, image: "registry.example.com/shop:v1" }] }
---
kind: Deployment
metadata: { name: shop-canary }
spec:
  replicas: 1                     # ← the canary share is 1 / (19 + 1)
  template:
    metadata: { labels: { app: shop, track: canary } }
    spec: { containers: [{ name: shop, image: "registry.example.com/shop:v2" }] }

What a canary can't catch

• Too little traffic. 1% of 20 requests a minute is one request every five minutes. You'll learn nothing from it. §09 shows how to tell a real problem from noise.
• Slow problems. A memory leak that kills the process after six hours won't show in a 15-minute step. Keep watching after 100%, and keep the old version easy to bring back.
• Damage that sticks. If v2 writes corrupted data, even 1% of users means corrupted rows that stay corrupted after rollback. Canaries limit how many users meet a bug; they don't undo what the bug wrote.
• Version mixing. v1 and v2 serve together for the whole canary, which can be hours. Everything in §10–§11 applies even more than with a rolling deploy.

Side by side: ship a buggy release four ways

The simulator below runs a service with 10 instances and about 100 requests per second. Each tick is 30 seconds of simulated time. v1 fails 0.4% of requests (every real service has a little background noise). Flip v2 to buggy and v2 fails about 7%.

Without automatic rollback, someone has to notice: the simulator assumes the On-callOn-callTaking turns being the engineer who gets woken up (“paged”) by alerts and must respond, usually for a week at a time. engineer gets paged (an alert wakes them) and rolls back about four minutes after the errors start. With auto-rollback on, a robot watches v2's own error rate (the yellow line) and rolls back after two ticks in a row above 2%. (§09 shows how that robot works.)

The graph has two lines on purpose. Red is the error rate of all traffic, the number a dashboard shows by default. Yellow is the error rate of v2's requests only. In a canary, yellow jumps to ~7% while red barely moves, because v2 gets only 5% of the traffic. Judge the canary by the red line and you'd never roll back (that's Pitfall 04 in §13).

Things to try

The numbers below are what the simulator prints in its summary line; your runs will match.

1. Healthy v2, each strategy. Compare "finished at" and "peak instances". Recreate finishes first (2:00) but is the only one with an outage (60 s). Blue-green takes 7 minutes and needs 20 instances at the peak. Rolling (11 min) and canary (13.5 min) are the slowest, because they wait at every step.
2. Buggy v2, no auto-rollback. Recreate gives the bug to everyone, after an outage, and rolling back means a second outage. Blue-green also gives it to 100% of traffic, but switching back is instant. Rolling hands it to more and more users while the human reacts: here 40% of traffic was on v2 by the time the rollback started (747 extra failed requests). Canary keeps it to a slice, though notice that the schedule doesn't wait for the human: it widens from 5% to 25% just before the rollback (111 extra failures).
3. Buggy v2, auto-rollback on. Canary's extra failures drop to 19: the robot reacts within a minute, while only 5% of traffic is at risk. Blue-green's robot is just as fast, but 100% of traffic was at risk for that minute (383). Rolling's robot stops the rollout at the first bad pod (76), so there's almost nothing to undo.

That pairing, a small slice plus a fast robot, is why most large teams end up with canaries and automated checks.

What the numbers say

Shadow traffic: test with real requests, risk nothing

Canaries still let a few users meet the bug. Sometimes even that is too much: you're replacing the search engine, the pricing code, or rewriting a service in another language, and you want to see v2 handle a week of real traffic before anyone depends on it. That's what shadow trafficShadow traffic (dark launch)Sending a copy of real requests to a new system and comparing its answers with the old one's, while users still get the old system's answers. (also called mirroring, or a dark launch) is for.

A shadowing handler in Go

func shadow(stable, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.Method != http.MethodGet { // ← only copy reads: a copied POST could charge a card twice
			stable.ServeHTTP(w, r)
			return
		}
		live := httptest.NewRecorder()
		stable.ServeHTTP(live, r) // the user's answer always comes from stable

		// The copy gets its own 2-second deadline, detached from the user's request.
		ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 2*time.Second)
		dup := r.Clone(ctx)
		go func() {
			defer cancel() // ← frees the timer when the copy is done
			dark := httptest.NewRecorder()
			next.ServeHTTP(dark, dup)
			if dark.Code != live.Code || !bytes.Equal(dark.Body.Bytes(), live.Body.Bytes()) {
				log.Printf("shadow mismatch %s: stable=%d %q next=%d %q",
					dup.URL.Path, live.Code, live.Body.String(), dark.Code, dark.Body.String())
			}
		}()

		for k, v := range live.Header() { // copy v1's recorded answer to the real user
			w.Header()[k] = v
		}
		w.WriteHeader(live.Code)
		w.Write(live.Body.Bytes())
	})
}

We wired it to a v1 that always answers price=1000 and a v2 with a bug for item 7, then sent three requests (real output):

/items/1 → user got price=1000
/items/7 → user got price=1000
/items/9 → user got price=1000
shadow mismatch /items/7: stable=200 "price=1000" next=200 "price=1100"

The user never saw the wrong price; the log told us exactly which request v2 got wrong.

Why the 2-second deadline matters: without it, a v2 that hangs keeps every copy waiting forever. We sent 500 requests through the handler to a v2 that never answers: right afterwards 501 goroutines were alive (one per stuck copy), and 2.5 seconds later, with the deadline, just 1. Without the deadline they pile up until the proxy runs out of memory, and then it does hurt real users. (This relies on v2 respecting its request's context, as Go's httputil.ReverseProxy and any well-written handler do. Busy proxies also cap how many copies can be in flight and drop the rest.) In production you'd count mismatches as a metric and look at a sample of them, not log every one.

In practice you rarely write this yourself. nginx has a mirror directive, and Envoy and Istio can mirror a percentage of traffic to another cluster:

location / {
    mirror /shadow;                 # copy each request to /shadow… (ALL methods, by default!)
    proxy_pass http://shop_v1;      # …and answer the user from v1
}
location = /shadow {
    internal;                       # not reachable from outside
    if ($request_method !~ ^(GET|HEAD)$) {
        return 204;                 # ← drop copies of POST, PUT, DELETE… without calling v2
    }
    proxy_pass http://shop_v2$request_uri;   # v2's answer is ignored
}

The if block is not optional. nginx's mirror copies every request, POST bodies included. We ran this config in an nginx 1.27 container with two logging backends and sent GET /items/1, POST /orders, GET /items/2 and DELETE /orders/9. v1 got all four; v2 got only the two GETs. With the if block removed, the POST /orders showed up at v2 too. (mirror doesn't accept variables, so you can't pick the target with a map; filter inside the shadow location like this.)

The hard part: side effects

A copied request that only reads is harmless. A copied request that does something happens twice:

• A copied POST /orders creates a second order. A copied payment charges the card twice. A copied signup sends a second welcome email.
• Both versions write to the same database, so v2's writes can overwrite v1's.

Ways teams handle it:

1. Mirror reads only, as the Go code does. Simple and safe, but writes go untested.
2. Give v2 its own fake world: its own copy of the database, and stubs instead of the payment and email services. More work, and the copy drifts from reality.
3. Run v2 in dry-run mode: it computes everything but skips the final write, and reports what it would have written. This is Shadow modeShadow mode (dry run)Running a new rule (a rate limit, a validation, a new service) in “log only” mode: it records what it would have done, but changes nothing. You switch it on once the logs show it behaves as intended., the same idea used to test a new rate limit or validation rule.

Comparing is harder than it looks

Two correct answers are rarely byte-for-byte equal: timestamps differ, IDs are random, JSON keys come out in a different order, floating-point sums differ in the last digit. Real comparison tools parse both answers and ignore the fields that are allowed to differ. Expect to spend more time tuning the comparison than building the mirror.

Feature flags: deploy is not release

Canaries and blue-green work on a whole version: every change in the build goes out together. A Feature flagFeature flagA runtime on/off switch (a config value) that decides whether new code runs, so you can ship code turned off and enable it gradually — or turn it off instantly if it breaks. works on one feature inside a version. That changes the timeline:

What a flag check looks like

Here's a complete flag in Go and JavaScript: a kill switch, an allow-list for your own team, and a percentage rollout using the same hashing trick as the canary proxy in §05.

type Flag struct {
	Name    string
	Enabled bool     // kill switch: false = off for everyone
	Percent int      // 0–100: share of users who get it
	Allow   []string // always on for these users (e.g. your own team)
}

// bucket turns any string into a number from 0 to 99, the same one every time.
func bucket(s string) int {
	h := fnv.New32a()
	h.Write([]byte(s))
	return int(h.Sum32() % 100)
}

func (f Flag) On(userID string) bool {
	if !f.Enabled {
		return false // ← kill switch wins over everything
	}
	if slices.Contains(f.Allow, userID) {
		return true
	}
	return bucket(f.Name+":"+userID) < f.Percent // ← flag name in the hash
}

// Using it in a handler (userID reads the logged-in user from the request):
func checkout(w http.ResponseWriter, r *http.Request) {
	if checkoutFlag.On(userID(r)) {
		newCheckout(w, r)
		return
	}
	oldCheckout(w, r)
}

We ran both versions with name: "new-checkout", percent: 10 and dev-ana on the allow-list. They print exactly the same thing (real output), which matters when a Go backend and a JS frontend must agree on who sees the feature:

alice    bucket= 2 on=true
bob      bucket=93 on=false
carol    bucket=99 on=false
dev-ana  bucket=30 on=true        ← not in the 10%, but on the allow-list
on for 1002 of 10000 users        ← ≈ 10%
after kill switch, alice: false dev-ana: false

Why the flag name goes into the hash: if you hashed only the user ID, alice would be in bucket 2 for every flag, so the same unlucky 10% of users would get every half-finished feature at once. Mixing in the flag name gives each flag its own random-looking 10%. Try both in the playground:

Where the flag values come from

A flag service keeps the rules (enabled, percent, allow-list) in one place with a UI and an audit log. Your app downloads the rules at startup, refreshes them every few seconds in the background, and evaluates every flag in memory. Calling the flag service on every request would add latency and make it a single point of failure. OpenFeature is a vendor-neutral standard API for flag checks, so you can switch providers without rewriting every if.

Two rules for production:

• Every flag needs a safe default for when the rules can't be loaded (the flag service is down, a new instance hasn't fetched yet). Usually that's "off", the old behavior.
• Test both sides. Your test suite should run the new path with the flag on and the old path with it off, for as long as both exist.

Four kinds of flags

A Kill switchKill switchA feature flag (or setting) whose only job is to turn something off instantly in production, without a deploy, when it misbehaves. is the ops kind worth adding before you need it: wrap anything optional and expensive (recommendations, a third-party widget, a heavy report) so that during an IncidentIncidentAn unplanned event that hurts users or puts them at risk, such as an outage, data loss or a security breach, handled by an on-call responder. you can shed it in seconds instead of deploying under pressure. An A/B testA/B test (experiment)Showing two variants to two random groups of users at the same time and measuring which does better on a business metric (sign-ups, purchases). Uses the same plumbing as feature flags. uses the same bucketing, plus statistics on a business metric to decide the winner.

Flag debt

Every flag doubles the number of ways your code can run. Ten independent flags is 1,024 combinations, and nobody tests most of them. Old release flags that stay at 100% for months are Flag debtFlag debtOld feature flags left in the code after they've been fully on for months. Each one is a branch nobody tests the other side of, and a switch someone might flip by accident.: dead branches that still compile, and switches someone might flip by mistake.

The most expensive example on record: in 2012, the trading firm Knight Capital reused an old flag name for a new feature. One of its eight servers hadn't received the new code, and on that server the flag woke up long-dead trading logic. According to the SEC's report, the firm lost more than $460 million in about 45 minutes.

Habits that keep flag debt down:

• Give every release flag an owner and an expiry date when you create it.
• Make removing the flag part of the feature's "done": a ticket created with the flag, closed when the if and the old path are deleted.
• Never reuse a flag name. Create a new one.
• Alert on flags that have been at 100% (or 0%) for longer than, say, 30 days.

Progressive delivery: let the metrics roll back

In the simulator in §06, auto-rollback beat a human by minutes, and every minute of a bad canary is failed requests. A human has to get paged, open a laptop, find the right dashboard and decide. A robot checks every minute and never hesitates to press undo. Progressive deliveryProgressive deliveryReleasing a change to more and more users in steps (canary, flag percentages), checking metrics automatically at each step and rolling back without a human if they look bad. means building that robot into the deploy itself.

What the robot needs

1. Metrics split by version. Every request counter and latency histogram (a metric that counts requests in duration buckets, so you can compute percentiles) needs a version label (v1, v2), or the canary's 5% vanishes into the average. Observability §05 covers labels.
2. Something to compare against. Not yesterday's numbers: traffic, time of day and other deploys all change. Compare with a baselineBaseline (canary analysis)A group of old-version instances that gets the same share of traffic as the canary, started at the same time, so you compare like with like instead of against yesterday's numbers.: v1 instances getting the same share of traffic over the same minutes.
3. A rule that turns two sets of numbers into "continue", "wait" or "roll back".
4. The power to act: setting the canary weight to 0, or redeploying the previous version.

"Clearly worse" with little data

At 5% of a small service, the canary might handle 200 requests in ten minutes. If the baseline had 1 error and the canary had 3, is v2 three times worse? Probably not: that difference happens by pure luck all the time. Small counts bounce around; that's noiseNoise (small samples)Random ups and downs in a measured number that aren't caused by any real change. With 200 requests, 2 errors vs 4 errors is noise; with 200,000, the same ratio is a real signal.. Watch it happen:

So a sensible rule has two guards: wait for enough requests, and ignore differences too small to matter:

type Stats struct{ Requests, Errors int }

func rate(s Stats) float64 { return float64(s.Errors) / float64(s.Requests) }

// judge compares the canary with a baseline that got the same traffic in the same minutes.
func judge(baseline, canary Stats) string {
	if baseline.Requests < 1000 || canary.Requests < 1000 {
		return "wait" // too few requests to tell a real problem from luck
	}
	b, c := rate(baseline), rate(canary)
	if c > 2*b && c-b > 0.005 { // ← twice as bad AND at least 0.5 percentage points worse
		return "rollback"
	}
	return "continue"
}

Three cases, run in both languages with identical results:

judge({200 req, 1 err},   {200 req, 3 err})    → wait       (too little data)
judge({5000 req, 18 err}, {5000 req, 21 err})  → continue   (0.36% vs 0.42%: noise)
judge({5000 req, 20 err}, {5000 req, 350 err}) → rollback   (0.4% vs 7%: real)

The thresholds (1,000 requests, 2×, 0.5 points) are examples; pick yours from your service's normal error rate and traffic. Real canary-analysis tools go further: Netflix's open-source Kayenta, for example, runs a statistical test on each metric and combines the results into a score. The idea is the same.

Which metrics to judge on

• Error rate: 5xx responses and timeouts, per version.
• Latency: p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them., not the average (Observability §06 explains why). A v2 that's 300 ms slower for 1% of requests is a regression.
• Saturation: CPU, memory, and restarts. A slow memory leak shows up here first.
• A business number: checkouts per minute, sign-ups, searches with a click. Some bugs return 200 OK with an empty cart.

Doing it in Kubernetes

Plain Kubernetes Deployments can't pause, measure or roll back by themselves. Add-ons like Argo Rollouts / FlaggerArgo Rollouts / FlaggerTwo open-source Kubernetes add-ons for progressive delivery: they shift traffic step by step to a new version, query your metrics, and roll back automatically if a check fails. add that. The shape of an Argo Rollouts canary, which queries PrometheusPrometheusThe most common open-source metrics system. Every few seconds it fetches (“scrapes”) the /metrics page of each program and stores the numbers so you can graph and alert on them. at each step (illustrative; check the field names against the version you install):

apiVersion: argoproj.io/v1alpha1
kind: Rollout                      # replaces the Deployment (selector and pod template left out)
metadata: { name: shop }
spec:
  replicas: 10
  strategy:
    canary:
      analysis:                    # runs in the background for the whole rollout
        templates: [{ templateName: error-rate }]
        startingStep: 1
      steps:
        - setWeight: 5             # 5% of traffic to the new version
        - pause: { duration: 10m } # bake time
        - setWeight: 25
        - pause: { duration: 20m }
        - setWeight: 50
        - pause: { duration: 20m } # then 100%
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: error-rate }
spec:
  metrics:
    - name: error-rate
      interval: 1m                 # ask Prometheus every minute
      failureLimit: 2              # more than 2 failed checks (i.e. the 3rd) → abort and roll back
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="shop", version="canary", code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{app="shop", version="canary"}[5m]))

That rule compares the canary with a fixed 1% rather than a baseline: simpler, and fine when your normal error rate is stable. Flagger (another add-on) works similarly. For exact weights, both need something that can split traffic by percentage: a Service meshService meshA setup where every service instance gets its own small proxy (a sidecar, often Envoy) that handles load balancing, retries, TLS and metrics for all its outgoing and incoming calls, configured centrally., or an ingress controller (the proxy, often nginx or Envoy, that carries out a cluster's IngressIngress (Kubernetes)A Kubernetes object with rules for routing outside HTTP traffic (hostnames, paths) to services inside the cluster, carried out by an ingress controller. rules). Without one, Argo Rollouts falls back to pod counts: setWeight: 5 with 10 replicas really means 1 pod, so 10%.

Roll back or roll forward?

Rolling back (going to the last good version) is the default: it's fast, rehearsed and doesn't need anyone to understand the bug. Rolling forwardRoll forwardFixing a bad deploy by shipping a new version with the fix (or a revert commit) through the normal pipeline, instead of going back to the old build. (shipping a fix) is right when going back is riskier than going on: v2 already migrated data that v1 can't read, or the fix is a one-line revert that the pipeline ships in minutes. The rule of thumb: mitigate first, investigate later. Get users back to working, then find out why.

Database changes during a deploy

The rule: N−1

During every strategy except recreate, v1 and v2 run at the same time against the same database, and after a rollback v1 runs against whatever v2 left behind. So:

Every migrationDatabase migrationA versioned script that changes a database's structure — add a column, rename a table — so every environment's database can be updated the same way. must work with the code version before it and the version after it.

That's N−1 compatibilityN−1 compatibilityThe rule that version N must work alongside version N−1 (the one before it): same database, same messages, same API. It's what makes rolling deploys and quick rollbacks safe.. It rules out the obvious one-step changes:

Expand, then contract, in deploy order

The safe way is expand / contractExpand / contractChanging a schema in safe steps: add the new shape alongside the old (expand), move code and data over, then remove the old shape (contract). No step breaks running code.: first add the new shape next to the old one, move the code over in small deploys, and remove the old shape last. SQL vs NoSQL §12 runs the full six steps on Postgres for splitting a name column; here is the same recipe seen from the deploy pipeline, for renaming total to total_cents:

1. Migration: add total_cents, nullable. v1 doesn't know it exists and keeps working.
2. Deploy v2: writes both columns, still reads total. Rolling back to v1 is safe.
3. Backfill: a job copies total into total_cents for old rows, in small batches (a BackfillBackfillA background job that copies existing rows from the old location to the new one in small batches, while new writes are handled separately.).
4. Deploy v3: reads total_cents, still writes both. Rolling back to v2 is safe, since total is still filled in. (If you rolled back from v2 to v1 after the backfill, v1 wrote rows without total_cents: re-run the backfill before deploying v3.)
5. Deploy v4: stops writing total. (If total was NOT NULL, drop that rule first, or v4's inserts fail.)
6. Migration: drop total, once you're sure you'll never roll back to v3 or earlier.

Four releases instead of one sounds slow, but each one is small, boring and reversible, and that's the point.

Where migrations run

Run migrations as their own pipeline step, before the deploy, not from the app's startup code. If ten instances all try to migrate as they boot, they race each other, and every instance is stuck "starting" while a long migration runs. Only expand steps go before a deploy; contract steps go in a later release.

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v7
      - run: ./scripts/migrate.sh          # only additive (expand) migrations here
  deploy:
    needs: migrate                         # ← code ships only after the schema is ready
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v7
      - run: ./scripts/deploy.sh

Most teams keep "down" migrations for local development only. In production, undoing a schema change usually means losing data that v2 already wrote, so the plan is: keep the schema compatible, and roll back only the code.

Not every ALTER is instant

The SQL can be fine and still take your site down, because many schema changes lock the table while they run. We timed a few on a 2-million-row orders table in Postgres 16 (measured on this machine):

So: use CREATE INDEX CONCURRENTLY on live tables (it's slower and can't run inside a Database transactionDatabase transactionA group of database changes that happen all together or not at all (COMMIT keeps them, ROLLBACK undoes them). How much other users can see or interfere with the half-done state depends on the isolation level., but it doesn't block writes). Add a constraint with NOT VALID first (it applies to new rows immediately) and run VALIDATE CONSTRAINT later; validating only blocks schema changes, not reads and writes. And check your migration tool's docs for the version-specific details.

The lock queue trap

Even an instant change like ADD COLUMN note text needs an ACCESS EXCLUSIVE lockACCESS EXCLUSIVE lockPostgres's strongest table lock, taken by most ALTER TABLE commands. While it is held — or even just waiting in line — no one else can read or write that table. for a moment. If a long query is reading the table, the ALTER has to wait for it. That sounds harmless. It isn't, because of the Lock queueLock queueThe line of sessions waiting for a lock on the same table. In Postgres a newcomer waits behind anyone already waiting for a conflicting lock, so one waiting ALTER TABLE blocks every later SELECT.:

1-- session A: a slow report, transaction still open
2BEGIN;
3SELECT count(*) FROM orders;  -- holds a read lock
4 
5-- session B: the migration
6-- SET lock_timeout = '2s';   -- the fix: steps 7-8 only
7ALTER TABLE orders ADD COLUMN note text;
8 
9-- sessions C, D, E: normal user requests
10SELECT total_cents FROM orders WHERE id = 1;
11 
12-- session A, 8 seconds after it started
13COMMIT;

Here is the real run: session A held a read open for 8 seconds, B sent the ALTER, C sent a primary-key lookup one second later.

  pid  | blocked_by |                    query
-------+------------+----------------------------------------------
 23058 | {23052}    | ALTER TABLE orders ADD COLUMN note text;
 23089 | {23058}    | SELECT total_cents FROM orders WHERE id = 1;

-- ALTER TABLE:  Time: 6942.043 ms (00:06.942)
-- the SELECT:   Time: 6008.473 ms (00:06.008)    ← normally 0.2 ms

pg_blocking_pids(pid) lists who a session is waiting for: the SELECT (23089) is blocked by the ALTER (23058), which is blocked by the long read (23052). On a busy table, a few seconds of that is hundreds of piled-up requests, timeouts, and a full Connection poolConnection poolA set of open database (or HTTP) connections that your program keeps and reuses instead of opening a new one per request. Pools usually have a size limit; when every connection is busy, the next caller waits for one to be returned. (every database connection the app keeps open is stuck waiting, so even requests that never touch orders can't get one).

The fix: give up quickly and retry. SET lock_timeout = '2s' makes the ALTER abandon its place in line after 2 seconds instead of blocking everyone behind it:

SET
ERROR:  canceling statement due to lock timeout     ← the ALTER gave up after 2.0 s
-- the SELECT: Time: 1001.543 ms                    ← waited 1 s instead of 6

Wrap it in a retry loop in your migration script:

#!/usr/bin/env bash
set -euo pipefail

# Run one migration statement, giving up on locks after 2 s, up to 5 times.
migrate() {
  for attempt in 1 2 3 4 5; do
    if PGOPTIONS='-c lock_timeout=2s' psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -c "$1"; then
      return 0                                   # ← success: stop retrying
    fi
    echo "attempt $attempt failed, retrying in 10 s" >&2
    sleep 10
  done
  return 1                                       # ← all 5 failed: fail the pipeline
}

# Build an index without blocking writes. No lock_timeout here: CONCURRENTLY doesn't
# block other queries, it waits for them. A failed build leaves an INVALID index, so
# drop any leftover first and check the result afterwards.
build_index() {  # usage: build_index NAME "TABLE (COLUMNS)"
  psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -c "DROP INDEX CONCURRENTLY IF EXISTS $1;"
  psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -c "CREATE INDEX CONCURRENTLY $1 ON $2;"
  valid=$(psql "$DATABASE_URL" -tA -c "SELECT indisvalid FROM pg_index WHERE indexrelid = '$1'::regclass;")
  if [ "$valid" != "t" ]; then
    echo "index $1 is INVALID" >&2
    return 1                                     # ← never go green with a broken index
  fi
}

migrate "ALTER TABLE orders ADD COLUMN IF NOT EXISTS note text;"
build_index orders_note_idx "orders (note)"

Run against a table with a 12-second read in progress (real output):

ERROR:  canceling statement due to lock timeout
attempt 1 failed, retrying in 10 s
ALTER TABLE
DROP INDEX
CREATE INDEX

Run alone while an 8-second transaction was open, build_index simply waited for it (7.6 s) and then built a valid index. While it waits it holds only a lock that doesn't block reads or writes.

Why PGOPTIONS instead of putting SET lock_timeout = '2s'; in front of the statement: psql sends a multi-statement -c string as one transaction, so SET …; ALTER … would work, but some statements refuse to run inside a transaction. PGOPTIONS sets the timeout for the whole connection instead.

Why the index gets its own function: CREATE INDEX CONCURRENTLY can't run inside a transaction, it waits for every older transaction to finish, and if it's cancelled halfway (by a lock_timeout, a deploy timeout, or Ctrl+C) it leaves an INVALID index behind. We reproduced exactly that with the first version of this script, which sent the index through migrate():

ERROR:  canceling statement due to lock timeout             ← attempt 1: an old transaction was open
NOTICE:  relation "orders_note_idx" already exists, skipping ← attempt 2, with IF NOT EXISTS
CREATE INDEX                                                ← exit code 0: the pipeline goes green…

   indexrelid    | indisvalid
-----------------+------------
 orders_note_idx | f                                          ← …with a broken index Postgres won't use

So build_index skips the lock timeout (letting the build wait is harmless, since it doesn't block reads or writes), drops any leftover first, and fails the pipeline if the result isn't valid.

Backwards compatibility: old and new must talk

Where old meets new

§10 covered the database. The same Version skewVersion skewDifferent versions of the same program (or of a client and its server) running at the same time — normal during deploys, and for months with mobile apps that users don't update. shows up in five more places:

• Browsers. A user opened your web app before the deploy; their tab is still running v1's JavaScript, calling your v2 API. If v2 deleted v1's old hashed JS files, their next lazy-loaded page fails to download (Build Tools explains hashed assets). Keep old assets around for a while.
• Mobile apps. Users update when they feel like it. An API change hits app versions from months ago.
• Other services. Your service is v2; the service calling it may still be v1 (or the other way round), because each team deploys on its own schedule.
• Messages in a queue. v2 writes a message in a new shape to a queue; a v1 worker, or a rolled-back service, picks it up. Messages can sit in a queue for hours.
• Caches and sessions. v2 stores a value in a shared cache (like Redis, an in-memory key–value store) or a cookie in a new format; v1 reads it after a rollback.

Rule 1: add, don't change

API Design §08 goes through these for public APIs, and gRPC §05 for protobuf. They're all the same rule.

Rule 2: readers must be tolerant

"Old readers ignore it" only holds if they're written to ignore unknown fields. That's a Tolerant readerTolerant readerA client that reads only the fields it needs and ignores everything else, so the server can add fields without breaking it. Go's encoding/json and JSON.parse already behave this way by default.. Go's encoding/json and JavaScript's JSON.parse are tolerant by default, but it's easy to make them strict by accident:

// v1's view of an order. v2 added "currency"; v1 has never heard of it.
type Order struct {
	ID    string `json:"id"`
	Total int    `json:"total"`
}

fromV2 := []byte(`{"id":"o-17","total":1250,"currency":"EUR"}`)

var o Order
err := json.Unmarshal(fromV2, &o) // ← unknown fields are skipped
fmt.Printf("%+v err=%v\n", o, err)
// {ID:o-17 Total:1250} err=<nil>

dec := json.NewDecoder(bytes.NewReader(fromV2))
dec.DisallowUnknownFields() // ← a "strict" reader
var strict Order
fmt.Println(dec.Decode(&strict))
// json: unknown field "currency"     ← one new field broke v1

The comments show the real output. Notice the two directions: Go shows an old reader facing new data; JS shows a new reader facing old data. You need both: v2 must handle what v1 wrote (old rows, queued messages), and v1 must handle what v2 writes (after a rollback).

Rule 3: deploy the side that copes with both first

During a rollout, each side meets the other side's old and new version. So the order rule is: first deploy the side that can cope with both shapes; deploy the side that depends on the new shape last.

• If the new code is a tolerant reader (it understands the old and the new format), the reader goes first. Example: consumers that read message formats 1 and 2, deployed before any producer writes format 2.
• If the new code depends on something new (a new client that needs a new response field), whoever provides it goes first: the server, even though it's the writer.

Pick a change and an order, and see which old/new pairs meet during the rollout:

For stored data and messages, the rule usually reads "readers before writers": deploy consumers that understand format 2 while producers still write format 1, and flip producers to format 2 only once every consumer (and every version you might roll back to) can read it. A feature flag is the usual way to do that flip, which turns "start writing the new format" into a switch you can turn off again.

Choosing a strategy (and combining them)

A quick decision guide

Answer five questions about your service:

Beyond the web service

The four strategies assume stateless instances behind a load balancer. Other things you deploy need their own versions of the same ideas:

• Background workers (queue consumers, cron jobs) don't get traffic from a load balancer, so "send 5% of traffic" means "run 1 of 20 workers on v2" and watch its failure and retry counts. Make sure v1 and v2 workers can process each other's messages (§11), and that jobs are safe to retry if a worker is stopped mid-job (Idempotency).
• Stateful systems (databases, message brokers, caches with data) are rolled one node at a time, by hand or by a Kubernetes operator (a program that runs in the cluster and knows how to manage that particular system): replicas first, the leader last after handing leadership over. Nobody canaries a database by traffic share.
• Frontends and static sites get blue-green almost for free. This site's own pipeline (CI/CD §03) deploys to Vercel, which builds every deploy at its own URL and then points the production domain at it in one step; rolling back points the domain at the previous build. The one thing to add: keep old hashed JS files available, for tabs that loaded the old version (§11).
• Mobile apps can't be rolled back at all: once a phone has the binary, it keeps it. Both app stores offer staged rollouts (Google Play lets you pick a percentage; Apple's phased release spreads an update over 7 days), and teams put every risky feature behind a remotely controlled flag, so the kill switch works even on old installs.

Small deploys beat clever strategies

The biggest lever isn't which strategy you choose but how much changes per deploy. A deploy with one small change is easy to watch, easy to blame and easy to undo. A deploy with three weeks of changes fails in ways nobody can untangle. The DORA metricsDORA metricsFour measures of delivery performance from the DevOps Research and Assessment program: how often you deploy, lead time from commit to production, the share of deploys that cause a failure, and time to restore service. research has found, year after year, that the teams deploying most often also tend to have the lowest change failure rates (the share of deploys that cause a problem): speed and safety go together. Frequent, small, automated deploys make every strategy in this topic work better.

Putting it all together

A mature setup for a typical web service combines everything in this topic:

1. Small PRs merge to main behind release flags (CI/CD §09); the pipeline builds one image.
2. Additive migrations run first, with lock_timeout and retries (§10).
3. The image goes out as a canary, 5% → 25% → 50% → 100%, judged automatically against a baseline (§05, §09).
4. Rollback at any step is instant and automatic.
5. After the deploy, the feature itself is released through its flag: staff, then 1%, then everyone (§08).
6. Weeks later, cleanup: the flag is deleted, and the contract migration removes the old column.

Pitfalls gallery

Check yourself

Predict the answer before you click. Q1 is covered in §03, Q2 in §04, Q3 in §05, Q4 in §08, Q5 in §09, Q6 and Q7 in §10, and Q8 in §11.

if (baseline.requests < 1000 || canary.requests < 1000) return "wait";
const b = rate(baseline), c = rate(canary);
if (c > 2 * b && c - b > 0.005) return "rollback";
return "continue";

Cheat sheet