Reliability & SRE

The 30-second version

You've pushed code that someone else deployed. This topic is about what that "someone else" does once the code is running: how they decide whether the service is healthy, who gets woken up when it isn't, and what happens the morning after. The discipline has a name, SRESRE (site reliability engineering)Running production with software-engineering methods: measured reliability targets (SLOs), error budgets, automation, and blameless learning from incidents., coined at Google in the early 2000s. Today many companies use it, whether or not anyone there has "SRE" in their job title.

The words you'll meet

• An SLISLI (service level indicator)A measured number that captures what users experience, like “fraction of requests answered successfully in under 300 ms”. is a measurement of what users experience, e.g. "the share of checkouts that succeed".
• An SLOSLO (service level objective)A reliability target you measure, like “99.9% of requests succeed in under 300 ms over 30 days”. The measured number (here, the share of fast successful requests) is the SLI; the 0.1% you're allowed to miss is the error budget. is the target for it: "99.9% over 30 days".
• The Error budgetError budgetHow much failure an SLO allows: at 99.9% over 30 days, about 43 minutes of bad requests. While budget remains you can ship risky changes; when it's spent you slow down and fix reliability. is the 0.1% you're allowed to miss. It's a budget because you spend it, on risky releases, experiments and the occasional bad day.
• An IncidentIncidentAn unplanned event that hurts users or puts them at risk, such as an outage, data loss or a security breach, handled by an on-call responder. is an unplanned event that hurts users. It's handled by whoever is on call, using a RunbookRunbookA short written guide linked from an alert: what it means, how to check it, and what to do first. as a starting point.

Before you start

This topic builds on Observability. You don't need all of it, but §09 SLIs, SLOs and error budgets and §10 Alerts that deserve to wake someone are the foundation; we'll recap them briefly and then go further. Load testing has its own topic, and so do deployment strategies such as canaries and feature flags. We'll link to them rather than repeat them.

“Reliable” is a number, not a feeling

Start from the user, not the server

Ask a team "is the checkout service reliable?" and you'll hear "CPU is fine", "no errors in the logs", "it felt slow yesterday". None of those answers the question. The question users care about is: when I press Pay, does it work, and quickly?

So reliability is measured from the outside, as a ratio of good experiences to all experiences:

SLI = good events ÷ valid events

For checkout that might be "checkouts that returned success in under 1 second ÷ all checkouts". That number is the SLISLI (service level indicator)A measured number that captures what users experience, like “fraction of requests answered successfully in under 300 ms”.. The target you set for it, say "at least 99.9% over the last 30 days", is the SLOSLO (service level objective)A reliability target you measure, like “99.9% of requests succeed in under 300 ms over 30 days”. The measured number (here, the share of fast successful requests) is the SLI; the 0.1% you're allowed to miss is the error budget.. The Observability topic shows how to compute it from metrics; here we focus on using it.

Why not 100%?

100% sounds like the right goal. It's the wrong one, for four reasons:

1. Users can't tell the difference. Their phone's Wi-Fi, their mobile network and their browser fail far more often than 0.01% of the time. Above a certain point your extra reliability disappears in their noise.
2. Your dependencies aren't 100%. If your card-payment provider promises 99.95%, checkout can't do better than that without extra work, like a second provider.
3. Each nine costs about ten times more. Going from 99.9% to 99.99% means redundant everything, slower and more careful releases, and people on call who must react within minutes.
4. 100% leaves no room to change anything. Every deploy carries some risk. A 100% target, taken seriously, means never deploying.

That last point is the heart of SRE. The gap between your SLO and 100% is the Error budgetError budgetHow much failure an SLO allows: at 99.9% over 30 days, about 43 minutes of bad requests. While budget remains you can ship risky changes; when it's spent you slow down and fix reliability., and it is meant to be spent: on shipping features, on experiments, on the occasional bad deploy. We'll make that concrete in §04.

What the nines look like

Toggle the SLO and watch how many failures it allows. Each square is one request:

The same arithmetic over a 30-day month (43,200 minutes):

Four nines means the whole month's budget is gone in a single 5-minute outage. Only a page that wakes someone within a minute or two, plus systems that fix themselves, can defend that. Most products don't need it, and that's fine.

SLO, SLA: which one has lawyers?

An SLASLA (service level agreement)A contract with paying customers that promises a reliability level, with a penalty (usually money back) if you miss it. Set looser than your internal SLO, so you notice trouble before it costs money. is a contract with paying customers: "99.9% or you get 10% of your money back". An SLO is an internal target your team uses to make decisions. Keep the SLO stricter than the SLA (say, SLO 99.95% against an SLA of 99.9%). Then you start fixing things while missing the target only hurts your pride, not your revenue.

Choosing SLIs and writing the SLO down

What to measure depends on what the service does

Different kinds of systems disappoint users in different ways, so they need different SLIs:

Notice that a latency SLI is also written as a ratio: not "p99 under 300 ms" but "99% of requests complete in under 300 ms". The ratio form works the same way as availability, so the error budget and burn-rate math from §04–§05 apply to both. (If percentilesPercentileThe value below which a given share of measurements fall: p50 is the median; p99 is the value that 99% of requests are at or under. Used for latency because averages hide slow outliers. are fuzzy, Observability §06 explains them.)

Where you measure changes what you see

The same request can be counted in several places, and each one misses something:

A common, sensible start: SLIs from load balancer logs or server metrics, plus a synthetic check for your most important journey, so you notice outages even when traffic is too low to measure.

"Good" and "valid" need fine print

The ratio is easy. Agreeing on what goes on each side of it is the real work. Some questions every SLO has to answer:

• Which requests are valid? Do health checksHealth checkA small automatic request (e.g. GET /healthz every 5 s) a load balancer or orchestrator sends to each server to decide whether it should still receive traffic. (the load balancer's automatic "are you alive?" requests) count? Bots? Requests from your own load tests? Usually not, since they aren't users.
• Is a 4xx "bad"? A 404 because the user typed a wrong URL is the user's mistake, not an outage. But a sudden flood of 401s after a deploy is your bug. Most teams count 5xx as bad and watch 4xx rates separately.
• What does "fast enough" mean? Pick the threshold from what users notice (a page that loads in under 1 s feels instant), not from what you happen to achieve today.
• Over what window? A rolling 30 days (always "the last 30 days") is the usual choice: it moves smoothly instead of resetting on the 1st of the month.

Write it down

An SLO that lives only in an alert rule gets forgotten. Teams keep a short document next to the code, reviewed like code. Here's a plain example. The exact format matters less than the fields:

service: checkout
owner: team-payments            # who answers for this SLO
slos:
  - name: checkout-availability
    description: Users can complete a checkout.
    sli:
      good: responses to POST /checkout with status < 500     # ← what counts as success
      valid: all POST /checkout requests, excluding health checks and synthetic load tests
      measured_at: load balancer logs                          # ← where we count
    objective: 99.9%            # the target
    window: rolling 30 days

  - name: checkout-latency
    description: Checkout feels instant.
    sli:
      good: successful POST /checkout answered in under 1000 ms
      valid: all successful POST /checkout requests
      measured_at: load balancer logs
    objective: 99%
    window: rolling 30 days

error_budget_policy: docs/error-budget-policy.md   # what we do when budget runs low (§04)
review: every quarter, with product                # targets aren't forever

Fewer, better SLIs

A service with fifteen SLOs has none, because nobody can keep fifteen numbers in their head during an incident. Start with one availability and one latency SLI for the most important user journey. Add more only when users complain about something your SLIs didn't catch. That complaint is a sign your SLIs measure the wrong thing.

Error budget policies: spending on purpose

The argument it ends

Without an SLO, every release turns into the same argument. Product wants to ship the new feature this week. The engineers who get paged want to slow down because "things feel fragile". Both sides are guessing, and the louder voice wins.

With an Error budgetError budgetHow much failure an SLO allows: at 99.9% over 30 days, about 43 minutes of bad requests. While budget remains you can ship risky changes; when it's spent you slow down and fix reliability., the question becomes arithmetic: how much budget is left? If plenty is left, ship; the users are clearly fine. If it's nearly gone, users are already feeling pain, and the next risky release should wait. The Error budget policyError budget policyA short document, agreed in advance by engineering and product, that says what the team does at each level of remaining error budget — e.g. “under 20% left: feature freeze until it recovers”. writes down that rule before anyone is upset.

What a policy says

It's usually one page, signed by the engineering and product leads. A typical shape:

And a few rules for the edge cases:

• Exceptions: who can approve an urgent launch during a freeze (e.g. a security fix, or a launch with a legal deadline)?
• Not our fault: if a cloud provider's outage burned the budget, does that count? Most policies say yes for the SLO, since users felt it, but the freeze might be relaxed if your own changes weren't involved.
• Who decides when it's disputed: usually a named director, so the decision doesn't drag on for days.

Play a quarter

The game below compresses this into 13 weeks. Each week you either ship a feature or spend the week on reliability. Shipping piles up risk; each feature makes the next incident a bit more likely. The policy above is enforced: in the red zone, the ship button turns off.

Play it two or three times with different rhythms. We simulated 200 quarters of this game. Shipping every week averaged about 9 features, because the freeze stops you partway, and missed the SLO in over a third of quarters. Two weeks of features, then one week of reliability also averaged about 9 features, and met the SLO in every one of the 200 quarters. That's the claim SRE makes: reliability work and feature work aren't enemies; the budget tells you the right mix.

Rolling windows give budget back

In the game, the quarter is fixed, so spent budget never comes back. Real SLOs usually use a rolling 30-day window: the SLI always covers "the last 30 days". A bad day 31 days ago falls out of the window, and its cost is returned to the budget. That's how a freeze ends by itself. After a few clean weeks, the old incident ages out and the budget climbs back above the threshold.

Spending budget on purpose

A leftover budget at the end of the month isn't a trophy. It's risk you could have taken and didn't. Teams with plenty of budget left can spend it deliberately:

• release more often, or with smaller canaries and shorter waits between steps,
• run planned maintenance (a database upgrade with a few seconds of errors),
• run chaos experimentsChaos engineeringDeliberately injecting failures (killing servers, adding latency) in a controlled way to check the system really survives them. in production (§12),
• or lower the SLO, if you keep beating it by a mile and users don't notice the difference.

Alerting on burn rate, in practice

The idea in one division

Observability §10 builds burn-rate alerts step by step, including the Prometheus rules. Here's the core again, because everything in this section rests on it:

burn rate = current error rate ÷ error rate the SLO allows

A 99.9% SLO allows 0.1% of requests to fail. If 1.44% are failing now, the Burn rateBurn rateHow fast you're using up the error budget compared with the allowed pace. 1 = you'd use exactly the whole budget by the end of the window; 10 = ten times too fast. is 1.44 ÷ 0.1 = 14.4. Two numbers follow from it:

• Budget used per hour = burn rate ÷ 720 (a 30-day window has 720 hours). 14.4 ÷ 720 = 2% of the month's budget every hour.
• How long a full budget lasts = 720 hours ÷ burn rate = 720 ÷ 14.4 = 50 hours.

A burn rate of 1 means "exactly on pace to use the whole budget by the end of the window", which is fine. Anything well above 1, sustained, is trouble, and the higher it is, the sooner someone needs to look.

The standard tiers

These starting points come from Google's SRE Workbook (chapter "Alerting on SLOs"). Each rule checks a long window, to ignore short blips, and a short window, so the alert stops soon after the problem does:

A page interrupts a human right now (an PagePage (alert)An alert that interrupts a human right now — a phone call or push notification — as opposed to a ticket someone looks at during work hours.); a ticket waits for work hours.

Why two windows, and why both must agree

A "1 h window" means "the error rate averaged over the last hour", recomputed every minute. A rule fires only while the burn rate is above its threshold in both windows at the same moment. Each window covers the other's weakness:

• The long window ignores blips. One bad minute barely moves a 1-hour average, so it can't page on its own.
• The short window ends the page quickly. After the errors stop, the 1-hour average still contains the bad minutes and stays high for a long time. The 5-minute average forgets them within minutes. Because the rule needs both, the page clears as soon as the short window recovers.

Worked example, with a 99.9% SLO: from 14:20 to 14:40, 5% of requests fail, a burn rate of 50.

1. At 14:21 the 5-minute average is already above 14.4, but the 1-hour average isn't: one bad minute in sixty is too little. No page.
2. By 14:36 the last hour holds 17 bad minutes: 17 × 50 ÷ 60 ≈ 14.2, plus the background errors, puts it just over 14.4. Both windows are above the line, so the rule pages.
3. At 14:40 a rollback stops the errors. By 14:43 the 5-minute window holds mostly good minutes, so it falls below 14.4, and the page clears.
4. A 1-hour-only rule would keep paging until 15:23, when the bad 20 minutes have mostly slid out of the hour.

Switch between the three rules and watch the shaded "page firing" area:

Try the numbers yourself

Two things to try. First, set the SLO to 99.99% and the error rate to 0.2%: a failure rate most people would call "fine" is a burn rate of 20, and the month's budget is gone in a day and a half. Stricter SLOs make alerts far more sensitive. Second, look at the low-traffic check at 10 requests per minute. A single failed request makes the 5-minute window look like a fast burn. More on that below.

The same math in code

You'll rarely compute this by hand, since your monitoring system does it. But writing it once makes it stick, and it's handy for scripts and dashboards. (New to Go? The 90-second Go primer covers the basics used here.)

package main

import (
	"fmt"
	"time"
)

// SLO is the promise: e.g. 99.9% of requests good over 30 days.
type SLO struct {
	Target float64       // 0.999
	Window time.Duration // 30 days
}

// BurnRate: how many times faster than allowed we are using the budget.
func (s SLO) BurnRate(bad, total int) float64 {
	if total == 0 {
		return 0 // no traffic, nothing burned
	}
	errorRate := float64(bad) / float64(total)
	allowed := 1 - s.Target   // 0.001 for a 99.9% SLO
	return errorRate / allowed // ← the whole idea is this one division
}

// BudgetUsed: the share of the whole window's budget spent during `lookback`.
func (s SLO) BudgetUsed(burn float64, lookback time.Duration) float64 {
	return burn * lookback.Hours() / s.Window.Hours()
}

// TimeLeft: how long a full budget lasts at this burn rate.
func (s SLO) TimeLeft(burn float64) time.Duration {
	return time.Duration(float64(s.Window) / burn)
}

func main() {
	slo := SLO{Target: 0.999, Window: 30 * 24 * time.Hour}

	// The last hour: 72,000 requests, 1,037 of them failed.
	burn := slo.BurnRate(1037, 72000)
	fmt.Printf("burn rate:          %.1f\n", burn)
	fmt.Printf("budget used (1 h):  %.1f%%\n", 100*slo.BudgetUsed(burn, time.Hour))
	fmt.Printf("full budget lasts:  %v\n", slo.TimeLeft(burn).Round(time.Hour))
}
// burn rate:          14.4
// budget used (1 h):  2.0%
// full budget lasts:  50h0m0s

(Both run as shown with Go 1.24 and Node 22. The output comments are real.)

Low-traffic services: when one request is 2%

Burn-rate alerts assume lots of traffic. At 10 requests per minute, a 5-minute window holds 50 requests; one failure is 2% errors, a burn rate of 20 at 99.9%. The short window alone would page for a single unlucky request. The long window helps (one failure in an hour of 600 requests is "only" a burn rate of 1.7), but quiet services still page more often than they should. Common fixes:

• Require a minimum number of failures in the rule, e.g. "and at least 5 failed requests in the last hour". It's one extra condition in the alert expression (using the recording rules from Observability §10):

  YAML

  Copy

  expr: |
    slo:checkout_errors:ratio_rate1h > (14.4 * 0.001)
    and slo:checkout_errors:ratio_rate5m > (14.4 * 0.001)
    and sum(increase(http_requests_total{route="/checkout", code=~"5.."}[1h])) >= 5   # ← the new line

  increase(…[1h]) is the number of failed requests counted during the last hour.

• Add Synthetic monitoringSynthetic monitoringRunning a few scripted user journeys against production on a schedule (say every 5 minutes) and alerting if one fails — an end-to-end test used as a smoke alarm for the live site.. A script that does a fake checkout every minute gives you steady traffic, so you notice an outage at 3 a.m. even when no real user does.

• Group small services that share a user journey into one SLO, so there are enough requests to measure.

• Accept a looser SLO or a longer window. A service with 100 requests a day can't meaningfully promise 99.9%: a single failure is already 1%.

What a good alert contains

When a burn-rate alert fires at 3 a.m., the person reading it is half asleep. The notification itself should answer the first questions:

annotations:
  summary: "Checkout is burning its 30-day error budget 14× too fast"
  impact: "About 1 in 70 checkouts is failing; users see 'payment failed'."
  dashboard: "https://grafana.example.com/d/checkout-slo"
  runbook_url: "https://wiki.example.com/runbooks/checkout-burn"   # ← what to do first

The runbook_url is the most important line. §06 shows what goes in a RunbookRunbookA short written guide linked from an alert: what it means, how to check it, and what to do first..

On-call without burnout

The rotation

An On-call rotationOn-call rotationThe schedule that decides who is on call when — usually each person takes one week in turn, with a primary (paged first) and a secondary (backup). spreads the job across the team. The common setup:

• Primary: gets paged first, for one week at a time.
• Secondary: the backup, paged if the primary doesn't respond, and available to help in big incidents.
• Weekly turns: with 8 engineers, each person is primary roughly one week in eight. Google's SRE book recommends at least 8 people for a rotation at one site (or 6 per site when two sites share it), so that nobody spends more than about 25% of their time on call. With fewer, on call comes around so often that people burn out.
• Follow-the-sunFollow-the-sunAn on-call setup where teams in different time zones each cover their own daytime hours, so nobody is paged at night.: teams in, say, Europe and the US each cover their own daytime, so nobody is woken at night. It needs a team in more than one time zone.

The paging tool (PagerDuty, Opsgenie, Grafana OnCall and similar) holds the schedule and an Escalation policyEscalation policyThe rule for who gets paged next if nobody responds: e.g. primary on-call, then after 5 minutes the secondary, then after 10 more the team lead.: if the primary doesn't acknowledgeAcknowledge (ack)Pressing “I've got it” on a page. It stops the escalation to the next person and tells everyone someone is looking. the page within, say, 5 minutes, it pages the secondary, then a manager. Pages never vanish just because one phone was on silent.

What "being on call" asks of you

• Be reachable and able to act within the agreed time (often 5–15 minutes to acknowledge): laptop nearby, network access, not on a plane.
• You don't have to know everything. Your job is to respond, judge how bad it is, follow the runbook, and pull in help when you're stuck. Escalating early is a skill, not a failure.
• Hand off cleanly. At the end of your week, a short On-call handoffOn-call handoffThe short meeting or note at the end of an on-call shift where the outgoing engineer tells the next one what happened, what's still open and what to watch. (a note or a 15-minute call) tells the next person what fired, what's still open and what to watch.
• Rest afterwards. Good teams give time off after a night of pages, and pay or compensate on-call time. A tired engineer makes the next incident worse.

Paging hygiene: every page must earn its noise

The fastest way to wreck on call is too many alerts. People who are paged for nothing learn to ignore pages, which is called Alert fatigueAlert fatigueWhen so many alerts are false or useless that people stop reacting to them — including to the real ones.. Sooner or later they'll ignore a real one. The rule of thumb:

A page must be urgent, about something users feel (or will feel very soon), and actionable by the person who receives it. Everything else is a ticket for work hours, or a graph on a dashboard.

You've just joined a team and inherited these alert rules. Decide what each one should do:

Measure the pager itself

Teams that keep on call healthy track a few numbers every week and review them together:

Runbooks: the first five minutes, written down

A RunbookRunbookA short written guide linked from an alert: what it means, how to check it, and what to do first. is a short page linked from the alert. It turns "I've never seen this alert before, at 3 a.m." into a checklist. A good one fits on one screen:

# CheckoutBudgetBurnFast

## What it means
Checkout is failing ≥ 14× faster than the 99.9% SLO allows. Users can't pay.

## Check first (2 minutes)
1. Dashboard: https://grafana.example.com/d/checkout-slo
   Is the error rate still high? Which status codes? All regions or one?
2. What changed? Deploys and flag flips are the vertical lines on the graph.
3. Payment provider status: https://status.payments.example

## Mitigate
- Bad deploy in the last hour?  → roll back (see below)
- Payment provider down?         → switch to backup provider (flag `payments.backup`)
- Traffic spike?                 → scale out; reject low-priority traffic (/recommendations)

## Commands
    kubectl rollout undo deployment/checkout -n shop      # back to the previous version
    kubectl rollout status deployment/checkout -n shop    # wait until it's done

## Escalate
Not better in 15 minutes, or unsure → declare an incident (§07) and page #payments-oncall.

(The kubectl commands are what you'd run on a Kubernetes cluster; the Kubernetes topic explains them. They weren't run for this page.)

Notice what's there: what it means, how to confirm, what to do, when to escalate. Notice what isn't: the full architecture, the history of the service, a theory of the bug. And update the runbook after every incident where it was wrong. A stale runbook is worse than none, because people trust it.

Incident response: roles, severity, communication

What counts as an incident

An IncidentIncidentAn unplanned event that hurts users or puts them at risk, such as an outage, data loss or a security breach, handled by an on-call responder. is anything unplanned that hurts users or puts them at risk: an outage, a slow or broken feature, data loss, a security breach. The key decision is declaring one: saying out loud, "this is an incident, we're switching to incident mode." That switch brings in a process: a dedicated chat channel, roles, regular updates.

Declare early. It costs almost nothing to declare an incident and close it 10 minutes later as "false alarm". It costs a lot to have three engineers quietly poking at production in different chat threads for an hour, each assuming someone else is telling the support team.

Severity: how big is it?

A SeveritySeverity (SEV level)A label for how bad an incident is, usually SEV1 (worst: many users can't use the product) to SEV3 or SEV4 (minor). It decides who gets pulled in and how often you update people. level tells everyone how much to drop. Every company defines its own, but most look like this:

When unsure, pick the higher severity. Downgrading later is easy; finding out an hour in that it was a SEV1 is not.

Roles: one job each

In a small incident, one person may do everything. Once more than two people are involved, split the work. The structure is borrowed from how fire services run emergencies:

Bigger incidents add a scribe, who keeps a timestamped log of what was seen, tried and decided. That log becomes the backbone of the postmortem (§09). The commander can hand over the role ("I'm handing incident command to Ben") if they're tired or needed elsewhere. They just have to say it explicitly.

Watch one unfold

Step through a real-shaped SEV2. Watch the roles appear, the first status update go out, and when the team stops investigating and starts fixing:

Communicating: short, honest, regular

Users and colleagues mostly want to know three things: is it being handled, what should I do, and when will I hear more? A good status update answers exactly that:

Investigating — Payments failing for some customers (14:12 UTC) Some card payments are failing at checkout. Orders are not charged when this happens, so it's safe to retry. We're investigating and will update by 14:45 UTC.

Notice what's missing: guesses about the cause, internal names, blame. Then keep the promise. Update by 14:45 even if the update is "still investigating". Silence feels worse to users than slow progress.

Ending an incident on purpose

An incident is mitigated when users are no longer hurt, and resolved when the commander declares it over, usually after a quiet period to make sure the fix holds. Before everyone goes back to normal work:

1. post a final status update ("resolved, here's what happened in one sentence"),
2. make sure any risky temporary changes are tracked (a disabled feature, an extra 20 servers),
3. assign an owner and a date for the postmortem.

Mitigate first, root cause later

Why mitigation comes first

Every minute of an incident burns error budget and hurts real users. Finding a Root causeRoot causeThe underlying reason an incident happened. In practice there are usually several contributing causes rather than one, which is why many teams say “contributing factors” instead. can take an hour of reading code under pressure. A MitigationMitigationAnything that stops users from being hurt — rolling back, failing over, turning a feature off, adding servers — even if you don't yet know why the problem happened. is anything that stops the harm without that understanding: put back the old version, send traffic somewhere healthy, switch the broken thing off.

It feels wrong to engineers. We want to know why. But the bug will still be there tomorrow, in the old version's diff, ready to be studied. Users won't wait.

Ask "what changed?" first

Most incidents are caused by a change: a deploy, a config edit, a feature flag flip, a certificate expiring, a traffic pattern shifting. So the cheapest question in any incident is "what changed just before it started?" Good dashboards draw deploys and flag changes as vertical lines on every graph, so the answer takes seconds (Observability §11 puts "what changed?" on its list of questions every dashboard should answer).

The mitigation toolbox

A RollbackRollback (deploy)Putting the previous version back after a bad deploy: redeploying the last good build, switching traffic back to the old copy, or turning a feature flag off. is the workhorse. It's fast, because the previous version was already built and tested, and safe, because it ran fine an hour ago. The Deployment strategies topic covers how blue-greenBlue-green deploymentRunning the new version (green) next to the old one (blue) and switching all traffic at once, so going back is just switching back. releases (old and new versions running side by side, traffic switched in one step) and canary releases make rollbacks nearly instant.

Try it: race the clock

Three incidents, eight possible moves. Every move takes time, and the red area is budget burned while you made it. Try the "deploy" scenario twice: once by digging for the root cause first, once by rolling back first.

Notice that the right mitigation depends on the situation, and only the 3-minute "what changed?" look tells you which situation you're in. Rolling back does nothing for a provider outage, and adding servers does nothing for a bad deploy. Restarting is the move people reach for when they don't know what's wrong. It helps only when a process is stuck (a memory leak, a deadlock, connections that never close); it does nothing for bad code or too much traffic, and under heavy load it makes things worse by removing servers for a while.

What a rollback looks like

How you roll back depends on how you deploy. Two common forms:

kubectl rollout undo deployment/checkout -n shop
kubectl rollout status deployment/checkout -n shop   # waits until every copy (pod) runs the old version

$ git log --oneline
7654ba1 Turn on pricing cache
7d5609e Add pricing service config
$ git revert --no-edit HEAD
[main 61828e1] Revert "Turn on pricing cache"
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git log --oneline
61828e1 Revert "Turn on pricing cache"
7654ba1 Turn on pricing cache
7d5609e Add pricing service config

git revert doesn't delete history. It adds a new commit that undoes the bad one, so the normal pipeline can test and deploy it like any other change (CI/CD covers that pipeline). The kubectl route is faster because it skips the build. The git route is slower but leaves the repository matching what's running. Many teams do the fast one first, then the git revert, so the next deploy doesn't bring the bug back.

Mitigations that make things worse

Under pressure, some moves feel productive but backfire:

• Restarting everything at once. You lose all capacity for a minute, and every client reconnects at the same moment (a Thundering herdThundering herdMany clients hitting a server at the same moment — e.g. all retrying together right after an outage — which knocks it over again.).
• Turning up retries. If a service is overloaded, more retries add load. That's a Retry stormRetry stormWhen a struggling service gets even more traffic because every caller retries its failed requests — often at several layers at once — so a small slowdown becomes a full outage., and it turns a slowdown into an outage.
• Two people changing things at once. If both a rollback and a config change happen, nobody knows which one helped, or which one broke things further. The operations lead announces each change in the channel before making it.
• "Quick fix" deploys under pressure. A hand-written fix that skips review and tests has a real chance of making things worse. Roll back first; fix forward only when rollback is impossible.

Blameless postmortems

Why blameless?

Imagine an engineer ran a script that deleted a production table. If the postmortem concludes "Sam made a mistake; Sam has been told to be more careful", what changes? Nothing. The script still accepts production credentials without asking. The next tired person will do the same thing. And everyone has learned to keep quiet the next time they break something, so you'll understand less about your next incident.

A blameless PostmortemPostmortem (incident review)A written review after an incident: what happened, why, how it was handled and what will change. Blameless: it looks for weak systems, not people to punish. assumes everyone acted reasonably given what they knew at the time, and asks why the system let a reasonable action cause harm. People talk openly when they aren't afraid, and open talk is the only way to learn what really happened.

When to write one

Most teams write a postmortem when an incident:

• hurt users noticeably (any SEV1 or SEV2),
• used a big chunk of error budget (the §04 policy might say "more than 20%"),
• needed a human to fix data or intervene by hand,
• or was a near miss: nothing broke, but only by luck.

What goes in it

A postmortem is a document of a few pages, written within a week while memories are fresh. A common structure:

1. Summary: two or three sentences a manager can read in 20 seconds. What broke, for whom, for how long.
2. Impact: numbers. "About 1 in 5 checkouts failed from 13:58 to 14:17, then fewer during the rollback until 14:24; ~3,100 users affected; about 10% of the monthly error budget." (Check it the §05 way: roughly 19 minutes × 19% + 7 minutes × ~10% ≈ 4.3 bad minutes, out of the 43.2 a 99.9% SLO allows.)
3. Timeline: timestamped facts, mostly copied from the scribe's log: when it started, when the alert fired, when someone acknowledged it, each action, when it was mitigated. The gaps are often the most useful part: why did 4 minutes pass between the first errors and the page, and another 10 after the acknowledgement before anyone asked what had changed?
4. Contributing factorsContributing factorOne of the conditions that together made an incident possible or worse — a missing check, a confusing dashboard, an untested script. Postmortems list several, not a single “root cause”.: everything that made the incident possible or worse.
5. What went well: the rollback took 4 minutes; the runbook was right. These are worth keeping.
6. Action itemsAction itemA concrete follow-up task from a postmortem, with an owner and a due date — e.g. “add a confirmation step to the cleanup script (Sam, by 14 May)”.: concrete changes, each with an owner and a due date.

Here's the §07 incident's timeline as a strip. The bands are the gaps worth asking about:

Contributing factors, not the root cause

Real incidents almost never have a single cause. The table-deletion incident above might have all of these:

• the cleanup script accepted production credentials without a confirmation step,
• production and staging credentials were stored in the same file,
• the backup restore had never been tested, so recovery took 3 hours instead of 20 minutes,
• the alert for "table missing" didn't exist; a customer noticed first.

Remove any one of them and the incident is smaller or doesn't happen. That's why many teams write "contributing factors" instead of "root cause". The popular Five whysFive whysAsking “why?” repeatedly (about five times) to dig from a symptom to deeper causes. Useful, but it tends to follow one chain and stop at a person, so use it with care. technique is useful for digging, but it follows a single chain of "why?" and tends to stop at a person ("why did the script run? Because Sam ran it."). When you reach a person, ask one more question: why was it easy for a person to do that?

Practise the rewrite

Each sentence below is the kind of thing that shows up in first drafts. Click to see a blameless version and the question it opens up:

Action items that actually happen

The point of the document is the action items. Too often they're vague ("improve monitoring") and quietly die in a backlog. Good ones are:

Prefer fixes that make the mistake impossible or harmless (a guard in the script) over fixes that rely on people remembering (a new rule in the wiki). And keep the list short. Three action items that get done beat twelve that don't.

Share it

Postmortems are most valuable when people outside the team read them. Many companies hold a weekly or monthly review where a postmortem is presented and discussed, and keep all of them in one searchable place. The same contributing factors (untested restores, missing alerts, manual deploy steps) show up again and again across teams. Spotting that pattern is how an organization, not just a team, gets more reliable.

Toil, and automating it away

What counts as toil

Not all operations work is toil. Google's SRE book gives a checklist. ToilToilManual, repetitive operational work that a machine could do, that has no lasting value and that grows as the service grows — like restarting a stuck job by hand every morning. is work that is:

• manual: a human runs the commands,
• repetitive: you've done it before and you'll do it again,
• automatable: a script could do it, since no human judgment is needed,
• tactical: done because something interrupted you, not because you planned it,
• without lasting value: afterwards the service is exactly as good as before,
• growing with the service: twice the traffic, twice the work.

Restarting a stuck worker every morning is toil. So is resizing a disk by hand whenever it hits 80%, or copying a customer's data between systems on request. Writing a postmortem, designing a new cache, or reviewing a pull request aren't toil; they need judgment, or they leave something better behind.

Google's teams aim to keep toil under 50% of each engineer's time, so the other half goes into engineering that removes future toil. Without a cap like that, toil grows until there's no time left to automate anything.

When does automating pay off?

Automation costs time up front and some upkeep afterwards. Toil costs a little time, forever, and more each month as traffic grows. Play with the numbers:

With the defaults, a 20-minute task done 5 times a week (about 1.7 hours a week, growing to about 3 hours) pays back a 24-hour automation project in about 16 weeks. And the chart doesn't count the hidden costs: the mistakes people make when they're bored, and the context switch every time it interrupts real work.

A real example: certificate expiry

An expired CertificateCertificate (TLS)A small signed file that says “this public key belongs to example.com”, vouched for by a trusted certificate authority. Browsers check it so they know they're talking to the real site. is one of the most embarrassing outages, because it's completely predictable. Browsers refuse the connection the moment a TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it. certificate passes its expiry date. Many teams have a calendar reminder and a person who checks by hand. That's toil, and it fails the one time the person is on holiday.

The fix is a tiny program that connects to each host, reads the certificate's expiry date, and fails if any expires within 14 days:

package main

import (
	"crypto/tls"
	"fmt"
	"net"
	"os"
	"time"
)

const warnBefore = 14 * 24 * time.Hour // renew at least two weeks ahead

func main() {
	failed := false
	dialer := &net.Dialer{Timeout: 10 * time.Second} // never hang on an unreachable host
	for _, host := range os.Args[1:] {               // e.g. go run . api.example.com:443 shop.example.com:443
		conn, err := tls.DialWithDialer(dialer, "tcp", host, nil) // nil config = normal certificate checks
		if err != nil {
			fmt.Printf("FAIL %s: %v\n", host, err)
			failed = true
			continue
		}
		cert := conn.ConnectionState().PeerCertificates[0] // the server's own certificate
		conn.Close()

		left := time.Until(cert.NotAfter)
		status := "ok  "
		if left < warnBefore {
			status, failed = "WARN", true // ← a human gets a ticket, weeks before users notice
		}
		fmt.Printf("%s %s expires %s (in %.0f days)\n", status, host, cert.NotAfter.Format(time.DateOnly), left.Hours()/24)
	}
	if failed {
		os.Exit(1) // non-zero exit fails the scheduled CI job
	}
}

We ran both against a local HTTPS server with a certificate made to expire in 5 days, then against the same server without trusting that certificate, against a port with nothing listening, and against a port that accepts the connection but never answers:

$ SSL_CERT_FILE=cert.pem go run . localhost:18443
WARN localhost:18443 expires 2026-11-04 (in 5 days)
exit status 1

$ go run . localhost:18443          # certificate not trusted
FAIL localhost:18443: tls: failed to verify certificate: x509: certificate signed by unknown authority (…)

$ NODE_EXTRA_CA_CERTS=cert.pem node check.mjs localhost:18443
WARN localhost:18443 expires 2026-11-04 (in 5 days)

$ node check.mjs localhost:18443 localhost:18444
FAIL localhost:18443: self-signed certificate
FAIL localhost:18444: connect ECONNREFUSED 127.0.0.1:18444

$ node check.mjs localhost:18445     # accepts, then says nothing
FAIL localhost:18445: timeout after 10 s
$ go run . localhost:18445
FAIL localhost:18445: context deadline exceeded

(SSL_CERT_FILE and NODE_EXTRA_CA_CERTS tell Go and Node to trust our home-made test certificate. Against real public hosts you don't need them. An already-expired certificate also shows up as FAIL, because the HandshakeHandshakeA short exchange of messages at the start of a connection where both sides agree it's open and settle the details (versions, keys, starting numbers) before any real data flows. (the opening exchange where the client checks the certificate) refuses it.)

Run it on a schedule

A checker nobody runs is still toil. Let CI run it every morning. In GitHub Actions, a workflow can be triggeredTrigger (event)What starts a workflow: a push to a branch, a pull request being opened or updated, a manual button press (workflow_dispatch), a schedule, and so on. Written under on:. by a schedule instead of a push:

name: cert-expiry
on:
  schedule:
    - cron: "17 7 * * *"      # every day at 07:17 UTC (minute hour day month weekday)
  workflow_dispatch:           # also adds a "Run workflow" button

permissions:
  contents: read
  issues: write                # needed to open an issue below

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with:
          go-version: "1.26"
      - run: go run ./tools/certcheck api.example.com:443 shop.example.com:443
      - if: failure()          # only runs when the check above failed
        env:
          GH_TOKEN: ${{ github.token }}
          GH_REPO: ${{ github.repository }}
        run: |
          # one open issue is enough: don't open a new one every morning
          open=$(gh issue list --label cert-expiry --state open --json number --jq length)
          if [ "$open" -eq 0 ]; then
            gh issue create --label cert-expiry --title "TLS certificate expiring soon" \
              --body "See $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
          fi

The cert-expiry label must exist in the repository once (create it in the Issues tab, or with gh label create cert-expiry). Because the step checks for an open issue with that label first, a certificate that stays unrenewed for a week produces one issue, not seven.

A ticket (here, a GitHub issue) is the right severity: it's real but not urgent, exactly like the "certificate expires in 10 days" alert in §06. The CI/CD topic explains workflows, jobs and permissions in depth.

The automation ladder

Most toil isn't removed in one step. Teams climb a ladder, and every rung is better than the one below:

1. Written down: a runbook says exactly what to do, so anyone can do it.
2. Scripted: one command runs all the steps.
3. Self-service: the people who need it (another team, support) can trigger it without you.
4. Automatic: the system detects the condition and runs it, and tells you afterwards.
5. Designed away: the condition can't happen any more. Certificates renew themselves (e.g. with an ACMEACMEThe protocol Let's Encrypt and tools like cert-manager and certbot use to prove you own a domain and get (and renew) TLS certificates automatically, with no human involved. client such as cert-manager or Let's Encrypt's certbot), and disks grow automatically.

The certificate checker is rung 4 for noticing. Automatic renewal is rung 5, and the checker then becomes a safety net that should never fire.

Capacity: having enough before you need it

What capacity planning is

Capacity planningCapacity planningWorking out how many servers (or how much database, cache, bandwidth) you need for the traffic you expect, plus a safety margin — using numbers measured in load tests. answers one question: will we have enough of everything for the busiest moment coming up, even if something breaks? "Everything" means servers, but also database connections, disk, queue throughput, third-party rate limits and cloud quotas.

The method has three inputs:

1. How much one server can safely handle. You measure this with a Load testLoad testSending a planned amount of traffic to a system to measure its latency, throughput and errors, and find where it breaks., not by guessing. The Load testing topic walks through finding that number with k6k6A free, open-source load-testing tool from Grafana Labs. You write the test in JavaScript; k6 itself is a single program written in Go that runs it and measures every request., and keeping HeadroomHeadroomSpare capacity you keep on purpose — e.g. workers that are only 80% busy on average — so random bursts can be cleared quickly instead of piling up. below it.
2. How big the peak will be. Today's busiest hour, grown by your monthly growth rate, plus any known events (a sale, a launch, a TV ad).
3. What failure you must survive. One server? A whole Availability zoneAvailability zone (AZ)One of a few separate data centers inside a cloud region (e.g. us-east-1a, 1b, 1c), with its own power and network, so one can fail without taking down the others.?

Plan for a zone, not just a server

The load testing topic adds one spare server ("N+1 redundancyN+1 redundancyHaving one more unit than you need (N needed, N+1 running), so the system still copes when any single unit fails. N+2 survives two failures, or one failure during maintenance.") so a single crash doesn't hurt. In the cloud, servers are spread across a few availability zones: separate data centers in the same region. A zone can fail as a unit (power, network, a bad config pushed to one zone). If you run in three zones and lose one, the other two must carry everything.

Worked example: peak needs 12 servers' worth of capacity. Spread over 3 zones that's 4 per zone. Lose a zone, and 8 servers are left to do the work of 12, running at 150% of their safe limit. To survive, each zone needs 12 ÷ 2 = 6 servers, so 18 in total. That's 50% more than the bare minimum. With only 2 zones, surviving one means each zone must carry the whole load alone: double.

Try it, and press "lose zone B":

The limits that aren't servers

Adding app servers is usually the easy part. Incidents often come from the parts that don't scale with a button:

• The database. More app servers means more database connections. A database that allows 500 connections, with 30 servers each keeping a Connection poolConnection poolA set of open database (or HTTP) connections that your program keeps and reuses instead of opening a new one per request. Pools usually have a size limit; when every connection is busy, the next caller waits for one to be returned. of 20, has already run out (600).
• Cloud quotas. Accounts have limits on instances, IP addresses and API calls per second. Autoscaling that hits a quota at peak simply stops adding servers.
• Third-party rate limits. Your payment or email provider might allow 100 requests per second on your plan. Your own scaling doesn't change theirs.
• Things that fill up: disks, log volume, message queues. Forecast these like traffic: "at the current rate, the disk is full on 14 March".

Autoscaling is a tool, not a plan

AutoscalingAutoscalingAutomatically adding instances when a metric (CPU, requests per second, latency) is high and removing them when it's low. adds servers when load rises. It's great for daily ups and downs, but it doesn't remove the need to plan:

• it takes minutes to notice the load, start servers and warm them up, so a sudden spike hits before help arrives,
• it has a maximum (which you set, or a quota sets for you),
• and it only scales what it controls. It won't add database capacity or raise a partner's rate limit.

So you still decide the minimum (enough for a zone loss at normal peak) and the maximum (below every downstream limit), and you check both against forecasts.

Before a big day: test it

Before a launch, a sale or a marketing event, teams do a capacity review. They estimate the peak (often 3–10× normal for a big sale), check every limit above against it, and run a load test at that level in an environment like production. Then they pre-scale on the day instead of trusting autoscaling to react in time. The Load testing topic covers testing somewhere that resembles production.

Chaos engineering in brief

The idea

You've added health checksHealth checkA small automatic request (e.g. GET /healthz every 5 s) a load balancer or orchestrator sends to each server to decide whether it should still receive traffic., timeouts and failover. But have you ever seen them work? Chaos engineeringChaos engineeringDeliberately injecting failures (killing servers, adding latency) in a controlled way to check the system really survives them. means deliberately injecting failures (kill a server, slow down a dependency, cut a network link) and watching whether the system copes. Netflix made it famous around 2011 with Chaos Monkey, a tool that randomly shut down production servers during work hours, so every team had to build services that survived it.

It's not random destruction. It's a scientific experiment:

1. Define the Steady stateSteady stateA measurable picture of “the system is fine” — e.g. checkout success ≥ 99.9% and p99 under 800 ms. A chaos experiment checks that the steady state survives a failure.: a measurable "everything's fine", usually your SLIs. "Checkout success ≥ 99.9%, p99 under 800 ms."
2. Form a hypothesis: "If one database replica dies, the steady state holds."
3. Inject the failure, starting with a small Blast radiusBlast radiusHow much of the system (and how many users) a single failure or bad deploy can affect. Smaller is better.: one server, 1% of traffic, or a staging environment.
4. Compare the steady state during the experiment with normal. Stop automatically if the SLIs dip below a limit.
5. Learn: if the hypothesis was refuted, you found a weakness cheaply. Fix it, then rerun the experiment.

Try some experiments

The system below has a load balancer, six app servers across three zones, a database with a replica, and a recommendations service on the checkout page. Pick which defenses exist, then break something:

The "add latency to recommendations" experiment is the classic lesson. A slow dependency is often worse than a dead one. A dead service fails fast, while a slow one makes every caller wait, until callers run out of connections or memory and fail too. A TimeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever. plus a fallback (show "popular items") is Graceful degradationGraceful degradationDesigning a system so that when a part fails, it keeps working with less — showing a default list instead of personalized recommendations — instead of failing completely.: the page keeps working with a bit less.

A fault injector in a few lines

You don't need special tools to start. A small MiddlewareMiddlewareA function that wraps an HTTP handler to run code before and/or after it — logging, auth, limits — without changing the handler itself. that fails a share of requests on purpose, switched on by environment variables, lets you check how callers behave when your service misbehaves:

package main

import (
	"log"
	"math/rand/v2"
	"net/http"
	"os"
	"strconv"
	"time"
)

// chaos wraps a handler and breaks a fraction of requests on purpose.
// CHAOS_ERROR_RATE=0.1 → 10% get a 503. CHAOS_DELAY_MS=300 → every request waits 300 ms.
func chaos(next http.Handler) http.Handler {
	errRate, _ := strconv.ParseFloat(os.Getenv("CHAOS_ERROR_RATE"), 64) // unset → 0, so chaos is off
	delayMS, _ := strconv.Atoi(os.Getenv("CHAOS_DELAY_MS"))
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(time.Duration(delayMS) * time.Millisecond)
		if rand.Float64() < errRate { // ← the injected fault
			http.Error(w, "chaos: injected failure", http.StatusServiceUnavailable)
			return
		}
		next.ServeHTTP(w, r)
	})
}

func main() {
	hello := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok\n"))
	})
	log.Fatal(http.ListenAndServe(":18081", chaos(hello)))
}

We ran both with CHAOS_ERROR_RATE=0.1 CHAOS_DELAY_MS=300 (Go on port 18081, Node on 18082) and sent 200 requests to each (measured on this machine):

$ for i in $(seq 200); do curl -s -o /dev/null -w '%{http_code}\n' localhost:18081/; done | sort | uniq -c
    187 200
     13 503          # Go: 6.5% failed (random, so not exactly 10%)
$ curl -s -w 'time: %{time_total}s\n' localhost:18081/
ok
time: 0.301007s      # the 300 ms delay, on every request

$ for i in $(seq 200); do curl -s -o /dev/null -w '%{http_code}\n' localhost:18082/; done | sort | uniq -c
    181 200
     19 503          # Node: 9.5% failed
$ curl -s -w 'time: %{time_total}s\n' localhost:18082/
ok
time: 0.300438s

Now point a caller at it. Does the caller time out, retry with Retry with backoffRetry with backoffTrying a failed call again after waiting, and waiting longer each time (100 ms, 200 ms, 400 ms…, usually with random jitter), so retries don't hammer a struggling service., show a fallback, or hang?

Real tools, real environments

Once the basics work, dedicated tools inject failures at the infrastructure level, without code changes:

• Toxiproxy: a proxy you put between a service and a dependency to add latency, drop connections or limit bandwidth. Handy in tests.
• Chaos Mesh and LitmusChaos: kill pods, partition networks and stress CPU inside a Kubernetes cluster.
• AWS Fault Injection Service, Azure Chaos Studio: managed experiments against cloud resources, with built-in stop conditions.

And start with people, not tools. A Game dayGame dayA planned session where a team deliberately causes a failure (or simulates an incident) and practises responding to it, like a fire drill. is a scheduled session where the team runs one experiment together, everyone watching dashboards, the runbook open, a rollback ready. It tests the system and the humans at once.

Staff-level: reliability trade-offs

Chains multiply

A checkout request might pass through a CDN, a load balancer, the app, a database and a payment API. It succeeds only if every step succeeds, so their availabilities multiply. Five parts at 99.9% each give 0.999⁵ ≈ 99.5%, five times the failures of any single part. A system is less reliable than its least reliable dependency, often a lot less.

Two tools fight back:

• Redundancy (parallel copies): if any one of n copies can serve the request, it fails only when all of them fail at once. Two app servers at 99.5% each: 1 − 0.005² = 99.9975%.
• Graceful degradationGraceful degradationDesigning a system so that when a part fails, it keeps working with less — showing a default list instead of personalized recommendations — instead of failing completely.: take a non-essential part off the critical path. If recommendations fail, show a default list instead of an error page. Then it drops out of the multiplication entirely.

The catch: failures aren't independent

That replica math assumes the copies fail independently, like two coins flipped separately. In real life they often fail together: same buggy deploy, same expired certificate, same zone, same overloaded database underneath. That's a Correlated failureCorrelated failureSeveral parts failing at the same time for the same reason — same bad deploy, same zone, same expired certificate — which breaks the “they fail independently” assumption behind redundancy math., and it's why "we have three replicas" doesn't mean 99.9999%. The defenses are about breaking the correlation:

• spread replicas across zones (and for the highest tiers, regions),
• deploy gradually (canariesCanary releaseSending a small slice of real traffic (say 1%) to a new version first, watching its errors and latency, then increasing the share if it looks good., one zone at a time), so a bad version never reaches every copy at once,
• avoid shared single points: one config service, one DNS provider, one certificate for everything, one person who knows how the system works. Each is a Single point of failureSingle point of failure (SPOF)One component that, if it breaks, takes the whole system down with it — because everything depends on it and nothing can take its place..

Every nine has a price

Moving from 99.9% to 99.99% doesn't make things a bit better. It cuts the allowed downtime from 43 minutes a month to about 4. In practice that requires most of:

Before asking for another nine, ask what the business loses per hour of downtime and compare it with the cost of the nine. Sometimes the honest answer is that 99.5% is right, and the money is better spent elsewhere.

Your SLO depends on your dependencies'

You can't promise users more than your critical dependencies give you, unless you design around them. If the payment provider is 99.95% and on your critical path, a 99.99% checkout SLO is fiction. Options: add a second provider (and fail over), make the dependency asynchronous (accept the order, charge the card a minute later from a queue), or set your SLO to what's achievable. The same applies inside a company. Ask internal teams for their SLOs, and give yours to the teams that depend on you.

Reliability vs. speed is mostly a false choice

It sounds like teams must choose between shipping fast and being reliable. Research on software delivery (the DORA reports behind the DORA metricsDORA metricsFour measures of delivery performance from the DevOps Research and Assessment program: how often you deploy, lead time from commit to production, the share of deploys that cause a failure, and time to restore service.) has repeatedly found the opposite: teams that deploy most often also tend to have the lowest change failure rates and the fastest recovery. The mechanism is the toolbox from this topic. Small, frequent changes are easier to review, canary and roll back than big monthly releases. The error budget tells you when the balance is off. It shouldn't feel like a permanent tug of war.

Don't forget the humans

A system that meets its SLO by paging the same three people every night isn't reliable. It's borrowing from their health, and that loan comes due when they quit. Treat on-call load (pages per week, night pages, time spent on toil) as a reliability metric with its own targets, and fund reducing it like any other reliability work.

Pitfalls gallery

Check yourself

Work out your answer before you click. Every question is covered above, and each explanation names the section.

Cheat sheet

Status update template: what users see · what they should do · when the next update comes.

Runbook template: what it means · check first · mitigate · commands · when to escalate.

Chaos experiment: steady state → hypothesis → small blast radius → automatic stop → learn and fix.