Famous Outages, Part 2: The Cloud Edition

A customer saves a setting. An engineer tidies up a routing policy. A maintenance job starts in one building. A team adds a few servers. A network blips for a moment. A traffic shift goes onto the wrong network. A service scales up. A power company does repairs. Each of these happens somewhere every day without anyone noticing. On these eight days, each one took down services that a large part of the internet relies on.

Part 1 replayed six famous outages, and its lessons were mostly about people's actions: a mistyped command, a deploy that missed a server, a backup nobody had restored. This sequel looks at eight incidents at cloud and CDN providers, the companies many other companies build on. Their lessons are about systems: limits nobody was watching closely enough, retries that made things worse, and dependencies nobody had drawn.

The stories are in an order that builds, not in date order:

1. One change goes everywhere (§02–§04): a valid setting meets a sleeping bug; a few re-ordered lines withdraw 19 data centers; maintenance automation reaches further than it should.
2. A limit nobody was watching (§05–§06): one more server means one more thread on every server; answers grow too big to arrive in time.
3. Retries make it worse (§06–§08, with DynamoDB in both groups): servers that time out ask again and again; disks hunt for space that isn't there; congestion feeds on its own retries.
4. The dependency nobody drew (§09): services built to survive losing a data center turn out to need it.

Each story is told from the company's own postmortem, published after the incident:

Every story has the same beats: a one-line hook, the setup (what the system was and what normal looked like), the trigger, the spread with a timeline you can step through, why it was hard to fix, the way back, and what changed. Most also have an interactive model of the mechanism, and every one ends with its sources. After the eight, §10 compares them, §11 turns them into design rules, and the page ends with a quiz and a cheat sheet.

Three words you'll need

These incidents happen inside big, shared systems. Three ideas come up again and again:

• A fleet is all the servers that run one service. "Adding capacity" means adding servers to the fleet.
• The data plane is the part of a system that handles each request: it forwards your packets, serves your file, answers your API call.
• The control plane is the part that manages the data plane: it decides which servers exist, where traffic should go, and what the settings are.

An everyday picture: in a city's traffic, the cars and roads are the data plane, and the traffic lights' timing system is the control plane. If the timing system crashes, the lights can keep running their last pattern for a while; if they all switch off at once, traffic stops. Hold on to that picture. In several stories below, the control plane broke, and what mattered most was how long the data plane could keep going without it. Try it:

Keeping on with the last settings when the manager disappears is called failing static (AWS engineers write about the same idea as static stability, though none of these postmortems uses the term). §04 shows what happens when the grace period is shorter than the outage.

This page builds on Famous Outages, Part 1 (especially How to read a postmortem) and on Cloud Basics for regions and zones. You don't need either to follow along.

The setup

Fastly is a CDN: websites put Fastly's servers in front of their own, so visitors get pages and files from a Fastly server in a nearby city. Each customer controls how Fastly treats their traffic with a configuration: which servers to fetch from, what to cache, which rules to apply. Customers change their configurations all the time, and the changes reach Fastly's servers quickly.

Four weeks earlier, on 12 May, Fastly had begun deploying new software. It worked. Nobody knew it contained a bug that only one specific kind of configuration could set off.

The trigger

At 09:47 UTC on 8 June 2021, one customer pushed a configuration change. It was valid. It also contained, in Fastly's words, "the specific circumstances that triggered the bug", and 85% of Fastly's network started returning errors. Fastly's customers, and everyone who relies on them, were hit.

The spread

Fastly's monitoring spotted the disruption within a minute. Step through the morning:

Inside the bug: it waited 27 days

The timeline starts four weeks before the outage, because that's when the cause arrived:

A bug that is in production but hasn't caused harm yet is a latent bug. Fastly's summary calls it "an undiscovered software bug" introduced by a deployment that began on 12 May, one "that could be triggered by a specific customer configuration under specific circumstances". It passed testing and ran on Fastly's servers for 27 days, because no customer's configuration had hit those circumstances yet.

Fastly hasn't published what the bug was, so we won't guess. But the shape is common enough to show with a made-up example. Here's a routing function with two optional settings. Every combination a person thought to test works, and one valid combination nobody tried crashes:

package routing

// Config is one customer's routing settings. Both fields are optional.
type Config struct {
	Backends []string // servers to send requests to; empty means "use the default"
	Sticky   bool     // send each user to the same backend every time
}

// PickBackend chooses where to send one user's request.
func PickBackend(c Config, userID uint32) string {
	if c.Sticky {
		return c.Backends[userID%uint32(len(c.Backends))] // ← divides by zero when Backends is empty
	}
	if len(c.Backends) == 0 {
		return "default"
	}
	return c.Backends[0]
}

"Sticky with no backends" is a legal configuration: the customer turned on stickiness and relies on the default server. But the sticky branch was written assuming there's always at least one backend. In Go, userID % 0 panics (crashes with a runtime error) with "integer divide by zero"; in JavaScript, 42 % 0 is NaN, backends[NaN] is undefined, and reading .host from it throws a TypeError.

Hand-picked tests miss it because the person writing the tests has the same blind spot as the person writing the code. A machine that tries inputs nobody thought of doesn't. That's fuzzing: a tool generates thousands of random, mutated inputs and reports any that crash your code. When the possible inputs are few, you can simply try them all:

package routing

import "testing"

func FuzzPickBackend(f *testing.F) {
	f.Add(true, uint8(2), uint32(42)) // a starting example; the fuzzer mutates it
	f.Fuzz(func(t *testing.T, sticky bool, n uint8, userID uint32) {
		c := Config{Sticky: sticky, Backends: make([]string, n%4)} // 0 to 3 backends
		for i := range c.Backends {
			c.Backends[i] = "backend"
		}
		PickBackend(c, userID) // a panic here fails the fuzz test and saves the input
	})
}

fuzz: elapsed: 0s, gathering baseline coverage: 1/1 completed, now fuzzing with 4 workers
--- FAIL: FuzzPickBackend (0.01s)
    --- FAIL: FuzzPickBackend (0.00s)
        testing.go:1693: panic: runtime error: integer divide by zero

    Failing input written to testdata/fuzz/FuzzPickBackend/fffd6d2e42404523

sticky=true, 0 backends: Cannot read properties of undefined (reading 'host')

The first lines are the fuzzer warming up (it runs on 4 CPU cores here); the FAIL line and the saved input are what matter. It found the crash in a fraction of a second. Fuzzing is no guarantee (a real configuration language has far too many combinations to try them all), but it explores the corners people forget. Unit Testing §12 covers Go's fuzzer in more depth.

Why it reached so much of the network

Fastly's summary doesn't explain how one customer's configuration could affect 85% of the network, and we won't speculate. What it does say is where the company is heading: "fully leveraging the isolation capabilities of WebAssembly and Compute@Edge", Fastly's platform for running customers' code on its servers, "to build greater resiliency from the ground up". The goal of isolation is the opposite of what happened: one customer's input should only ever be able to hurt that customer. §11 comes back to this idea as cells.

Why it was hard to fix

This is one of the shortest outages on the page, and noticing it wasn't the problem: Fastly's monitoring spotted it within one minute. The hard part was finding which of many customers' configurations had set it off. That took until 10:27, 39 minutes after the alert.

The way back

Once the configuration was found, Fastly disabled it, and services began to recover at 10:36; Fastly says 95% of the network was normal within 49 minutes (Fastly doesn't say what the 49 minutes are counted from; from the 09:47 start, that would be 10:36). Notice what fixed it: not a code fix (that only started deploying at 17:25) but switching off the input that triggered the bug. When a bug is triggered by data, removing the data is often the fastest way out.

What changed

From Fastly's summary, published the same day, these were its plans:

• Deploy the bug fix across the network "as quickly and safely as possible" (it began at 17:25 UTC).
• Conduct a complete post mortem of the processes and practices followed during the incident.
• Figure out why the bug wasn't detected during software quality assurance and testing.
• Evaluate ways to improve its remediation time (how long it takes to fix a problem).
• Keep investing in isolation (WebAssembly and Compute@Edge) to make the platform safer at its foundations.

Sources

• Fastly (Nick Rockwell): Summary of June 8 outage, 8 June 2021. Timeline, numbers, causes and fixes on this page come from here.
• The Go and JavaScript code is our own illustration of a latent bug, not Fastly's code (Fastly hasn't published the bug). Its output is real, from Go 1.24 and Node 22, as labelled.

The setup

To be reachable on the internet, a network announces its address ranges to its neighbors using BGP. Each range is an IP prefix. Which prefixes get announced is decided by a routing policy: a list of rules, called terms, that the router reads from top to bottom. A term that accepts or rejects a prefix settles it, and the router stops reading for that prefix, so the order of the terms matters: a term below a reject that catches everything is never reached. If a prefix stops being announced ("withdrawn"), the rest of the internet forgets how to reach it.

For 18 months, Cloudflare had been converting its busiest locations to a more resilient design it calls Multi-Colo PoP (MCP). By June 2022, 19 data centers used it, including Amsterdam, Frankfurt, London, Tokyo, Singapore and Sydney. The design adds an extra layer of routers, the spines, that lets Cloudflare switch parts of a data center off for maintenance without disturbing traffic. These 19 locations are only 4% of Cloudflare's network, but they carry a large share of its traffic.

On 21 June, engineers were rolling out a tidy-up: adding informational labels (BGP "communities") to the site-local prefixes, the addresses Cloudflare's servers use to talk to each other and to customers' origin servers (the customers' own servers behind Cloudflare). The change had a dry run (a rehearsal that shows what would change without changing it), a stepped rollout plan that sent it out in stages, and peer review by multiple engineers.

The trigger

At 06:27 UTC, the rollout reached the MCP locations and the change was deployed to their spines. In Cloudflare's words, this "swiftly took these 19 locations offline".

On the spines, the change didn't just add labels. Cloudflare's postmortem shows the diff: several terms were marked with !, which in that format means "moved". The two site-local terms had moved from the top of the policy to the bottom, below a term named REJECT-THE-REST whose whole body is then reject;. See what that does:

The spread

With the site-local prefixes withdrawn, three things broke at once. Cloudflare lost direct access to the affected locations. Its servers could no longer reach customers' origin servers. And its internal load balancer, Multimog, could no longer forward requests between servers inside each location. The groups of servers in a location aren't all the same size, and Multimog is what moves requests between them; without it, Cloudflare says, "our smaller compute clusters in an MCP received the same amount of traffic as our largest clusters", so the smaller ones overloaded. The outage hit 50% of total requests.

Why it was hard to fix

The change removed the very routes engineers would normally use to reach those locations and undo it. Cloudflare had backup procedures for exactly this and used them to take control of the affected locations, but it added "difficulty". It took until 06:58 to find and understand the root cause.

The way back

At 06:58 work began to revert the change, and the first data center came back. The last revert was done at 07:42. It took longer than necessary because "network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically". Several people fixing the same thing at the same time, without coordination, can undo each other's work.

Why the safety steps didn't catch it

The change was staged, which makes this the most instructive part of the story. The first location got it at 03:56 and the busiest locations at 06:17, with no problems, because none of them used the MCP design. Cloudflare: "the stagger policy did not include an MCP data center until the final step." The stepped rollout tested the change everywhere except the kind of place it would break, then sent it to all of those at once.

The review didn't catch it either. A re-ordering in a long diff is a single character per line, and "an initial glance at this diff might give the impression that this change is identical".

What changed

Cloudflare's plans, from the postmortem:

• Process: change procedures and automation get MCP-specific test and deploy steps.
• Architecture: the policy statement will be redesigned so terms can't be put in the wrong order by accident.
• Automation: enforce a better stagger policy for network configuration rollouts, and an automated "commit-confirm" rollback. Cloudflare says the first "would have significantly lessened the overall impact", and the second "would have greatly reduced the Time-to-Resolve".

Sources

• Cloudflare (Tom Strickx and Jeremy Hartman): Cloudflare outage on June 21, 2022. Timeline, numbers, causes and fixes on this page come from here.
• The policy widget simplifies the diff in the postmortem; its term names are Cloudflare's.

The setup

Google runs its data centers with cluster management software: programs that decide which jobs run on which machines, and move or stop them when needed. One of its jobs is handling maintenance events such as power or network work. Each job is labelled with what to do when maintenance comes: move to a machine that isn't affected, or stop and wait.

Google's network has a control plane too: software that works out how traffic should flow between locations and tells the network equipment. On Sunday 2 June 2019, three things lined up. Google calls the first two "normally-benign misconfigurations":

1. The network control jobs in the affected regions were labelled "stop" during maintenance, not "move".
2. The cluster management software running the network control plane was eligible for a particular, relatively rare type of maintenance event.
3. The software that starts maintenance events had a bug that let it deschedule (stop) several independent clusters at once, "crucially even if those clusters were in different physical locations".

The key is the difference between a building and a cluster. Google runs each location's machines as several logical clusters: groups of jobs managed as one unit. A logical cluster's jobs can sit in more than one physical location, so stopping a logical cluster stops its jobs everywhere.

The trigger

At 11:45 US/Pacific on Sunday 2 June 2019 (18:45 UTC), that rare type of maintenance event started in a single physical location. The automation listed the logical clusters with jobs there, and stopped each of them whole:

The spread

Fail static: a few minutes of grace

Google designed its network to fail static (the static stability idea from §01): if the control plane disappears, the network equipment keeps forwarding traffic using the last instructions it was given, as an extra layer of defense. It worked. The network "ran normally for a short period - several minutes - after the control plane had been descheduled".

But the control plane didn't come back in minutes. After those several minutes, routes between the affected locations were withdrawn: the network stopped offering paths between them. (Google uses BGP, the protocol networks use to tell each other which addresses they can reach, inside its own network too.) The report doesn't explain why the grace period lasted only minutes; a common reason in network design is that routes are treated as stale and dropped if nothing refreshes them, so they don't keep sending traffic down paths that may no longer exist. Capacity dropped sharply, the remaining links filled up, and some Google Cloud regions became unreachable. That's how a control-plane outage became a data-plane outage.

Then recovery hit a second wall. With every copy of the network control plane stopped in several locations, its configuration data had been lost (in Google's words; its fix, making the control plane "persist its configuration", suggests the data lived in the running jobs and wasn't saved anywhere they could reload it from) and had to be rebuilt and sent out again, during a huge network event, for several locations at once. The new configuration started rolling out at 14:03, an hour after the root cause was found. Try the two things Google said it would change:

How bad it was

Google's report breaks the impact down service by service. A few numbers from it:

• Compute Engine instances (Google's virtual machines) in us-east4, us-west2, northamerica-northeast1 and southamerica-east1 were unreachable for the whole incident. Other US regions saw packet loss, for example up to 33% in us-east1 at the start. us-west1 and all European and Asian regions had no regional congestion.
• Cloud Storage (file storage) average error rates during the incident ranged from 96.2% in us-west2 to 0.7% in us-central1.
• Pub/Sub (messaging) saw a 27% global drop in successful publish and subscribe requests.
• Depending on the region, the packet loss lasted between 3 hours 19 minutes and 4 hours 25 minutes.

Why it was hard to fix

Engineers were alerted two minutes after the failure began. Finding the cause took until 13:01, and Google gives the reason plainly: "Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network." Google had backup plans for exactly that, but they cost time too: engineers travelled to secure facilities built to withstand the worst failures, and less critical classes of network traffic were given lower priority to reduce congestion.

Meanwhile, teams across Google moved their own services' traffic away from the affected regions, so they could keep serving from elsewhere.

The way back

At 13:01 the cause was found and engineers halted the maintenance automation. Then came the slow part: rebuilding the lost control-plane configuration and sending it out to several locations during a huge network event. The new configuration began rolling out at 14:03. As the control plane restarted in each location, network capacity began to come back at 15:19, and full service resumed at 16:10.

What changed

From the report's "Prevention and follow-up" section:

• Halted the maintenance automation until it can't deschedule jobs in several physical locations at the same time, and hardening the cluster management software to reject such requests "regardless of origin".
• Reconfiguring the network control plane to reject maintenance requests of the type involved.
• Making the control plane in each location keep its configuration, so it doesn't need rebuilding and redistributing after all its jobs stop: Google expects this to cut recovery time "by an order of magnitude" (about tenfold).
• Letting the network fail static for longer, "to allow an adequate window for recovery with no user impact".
• Reviewing and testing emergency tools and procedures (including how Google talks to customers) against network failures like this one, and adding this and similar catastrophic failures to its continuous disaster recovery testing.

Sources

• Google Cloud: Google Cloud Networking Incident #19009 (incident report), June 2019. Timeline, numbers, causes and fixes on this page come from here.
• The two-location layout in the maintenance widget and every duration in the fail-static widget are our own illustrations; the mechanism is Google's.

The setup

Amazon Kinesis lets programs send in a continuous stream of records (clicks, logs, measurements) and read them back in order, in real time. Customers use it directly, and so do several other AWS services. This incident was in us-east-1, AWS's region in Northern Virginia.

AWS describes Kinesis in two layers. The back end is many clusters (groups of servers working together) that store and process the streams; AWS had already split it into cells. In front of them sits the front-end fleet, whose job is "small but important": check who's calling, apply limits, and route each request to the right back-end cluster. To route, each front-end server keeps a cache called the shard-map. Part of building that cache is a constant stream of messages from the other front-end servers, and for that, each server creates an operating-system thread for every other server in the fleet.

The trigger

At 2:44 AM PST on 25 November 2020 (10:44 UTC), a "relatively small addition of capacity" started going into the front-end fleet; it finished at 3:47 AM. AWS calls it "the trigger, though not root cause".

The spread

The first alarms fired at 5:15 AM, about an hour and a half after the last new server went in. Step through the day:

How adding servers broke every server

Work it through with small numbers. With 4 front-end servers, each one keeps 3 threads for its peers. With 5, each keeps 4:

With N servers, each server keeps N − 1 peer threads, and the fleet as a whole keeps N × (N − 1). Add 10% more servers and every existing server needs 10% more peer threads.

Threads aren't free: each one reserves memory for its own stack and some bookkeeping inside the operating system. So operating systems cap how many threads a process or user may create, to stop one program exhausting the machine: a thread limit, one kind of ulimit. Nobody notices it until a program asks for one thread too many and the request fails. Kinesis had never been big enough to reach its limit. The "relatively small addition of capacity" made it that big, and because every server has the same number of peers, every server crossed the line, not just a few. Their shard-maps never finished building, so they couldn't route requests.

On Linux you can see the limits and a process's current thread count like this:

cat /proc/sys/kernel/threads-max   # the most threads the whole machine allows
ulimit -u                          # the most processes + threads this user may run
grep Threads /proc/1234/status     # threads that process 1234 has right now

128504
64252
Threads:	13

Why other services were hit too

Kinesis sits underneath other AWS services, so when it failed, they were hit too, and the services built on top of those were hit in turn: a cascading failure. Try breaking Kinesis, then break CloudWatch on its own:

Two details in AWS's account are worth a second look:

• Cognito's dependency was supposed to be harmless. Its data stream to Kinesis was "designed to be best effort", buffered locally to ride out short outages. A long outage hit a latent bug in that buffering code, and Cognito's web servers blocked. A dependency you think is optional is only optional if you've tested life without it for hours, not seconds.
• The status page tool needed the broken system. AWS couldn't post to its Service Health Dashboard at first, because the posting tool used Cognito. This is Part 1's recovery dependency trap again, one layer removed: the tool didn't depend on Kinesis directly, it depended on something that did.

Why it was hard to fix

• Too many symptoms. Errors showed up "in all aspects of the various calls" between servers, which made it hard to "separate side-effects from the root cause". It took until 9:39 AM, 4 hours 24 minutes after the first alarms, to confirm the cause.
• Every candidate fix needed a full restart. All the suspects required changing every front-end server's configuration and restarting it. Guess wrong and you restart twice, so the team investigated further rather than act on the leading theory (memory pressure, which turned out to be wrong).
• The restart itself had to be slow. A starting server builds its shard-map with the same resources it uses to serve requests. Bring servers back too fast and they have almost nothing left for requests, look unhealthy, and get removed again.

The way back

By the time the cause was confirmed at 9:39 AM, the extra capacity had already been removed, so the thread count would fit under the limit again. AWS chose not to raise the limit without testing, and restarted the fleet instead. To speed things up, the team changed servers to load their data at startup straight from the authoritative metadata store (the master copy) instead of from their neighbors.

The first restarted servers took traffic at 10:07 AM. The fleet of "many thousands of servers" could only come back at a few hundred per hour; errors dropped steadily from noon, and Kinesis was fully normal at 10:23 PM. The services that depended on it recovered on their own schedules; CloudWatch's metrics and alarms fully recovered at 10:31 PM, with delayed data backfilled over the following hours.

What changed

From the summary (most of these were announced as plans, “in the very short term” or “in the medium term”):

• Fewer, bigger servers: larger CPU and memory, so fewer servers, so fewer threads on each. AWS notes that threads per server are "directly proportional to the number of servers in the fleet".
• Fine-grained alarms on thread consumption, and finishing tests of a higher OS thread limit.
• Much faster cold starts for the front end, including moving the front-end server cache to a dedicated fleet.
• Big users separated: a few large AWS services, such as CloudWatch, move to a separate, partitioned front-end fleet.
• Cells for the front end too, accelerated. AWS: cellularization keeps each part "operating within a previously tested and operated range" and "will provide better protection against any future unknown scaling limit".
• Dependents fixed too: Cognito's web servers now survive Kinesis errors without exhausting their buffers; CloudWatch keeps 3 hours of metrics locally (done in us-east-1, rolling out globally); support engineers train regularly on the backup status-page tool.

Sources

• AWS: Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region, about 25 November 2020. Timeline, numbers, causes and fixes on this page come from here.
• The thread-limit widget's numbers are made up for illustration. The Linux commands and output, and the Go thread limit, are general facts checked for this page, not from AWS's summary.

The setup

DynamoDB is AWS's managed database. Each table is split into partitions, spread over many storage servers and copied for safety. Which partitions a server holds is its membership, and an internal metadata service keeps track of it. Storage servers check in with the metadata service regularly to confirm their membership, and again after a network disruption or on startup. If an answer doesn't arrive within a time limit, the server takes itself out of service and retries. (AWS doesn't say why; a common reason in systems like this is that, without a current answer, a server can't be sure it is still responsible for the data it holds, so it stops serving rather than risk giving wrong answers.)

Over the months before, customers had rapidly adopted a new feature, Global Secondary Indexes (GSIs). Each GSI has partitions of its own, so, in AWS's words, a table with many partitions "could have its contribution of partition data to the membership lists quickly double or triple". Building some servers' membership lists began to take nearly as long as the servers would wait. AWS: "We did not have detailed enough monitoring for this dimension (membership size)", and the metadata service didn't have the capacity for these much heavier requests.

The trigger

At 2:19 AM PDT on Sunday 20 September 2015 (09:19 UTC), a brief network disruption hit a portion of DynamoDB's storage servers in us-east-1. That kind of blip is normally handled "seamlessly": the affected servers ask for their membership again and carry on.

The spread

This time, many servers asked at the same moment, and some of the answers were the newly enormous ones. Several large requests at once slowed processing further, and answers began missing the time limit. Play it out with made-up numbers:

The servers that missed their answers went out of service and retried. The retries kept the metadata service busy, so it answered other servers late too, servers that had nothing to do with the original blip and were just checking in on schedule. They dropped out as well. Even servers whose renewal had just succeeded could fail the next one. That is a retry storm built from timeouts:

Other AWS services that keep their own data in DynamoDB had problems too: EC2 Auto Scaling saw significantly increased API faults (and built up a large backlog of jobs), CloudWatch had delayed and missing metrics, SQS had slightly elevated errors at first and significantly elevated ones once DynamoDB traffic was paused at 5:45 AM, and some AWS Console logins were held up for tens of seconds because a call that didn't even need to succeed had "a very long timeout".

Why it was hard to fix

The obvious fix was to add capacity to the metadata service. But adding capacity takes administrative requests, and the metadata service was too overloaded to accept them. After several failed attempts, the team was stuck in a loop: the service needed help, and was too busy to receive it.

The way back

At 5:06 AM AWS did the counter-intuitive thing: it paused requests to the metadata service. That stopped the retries, the load dropped, and the service could finally accept the requests to add significant capacity. Then the team re-enabled requests, put storage servers back into service, and let normal load return. The pause had a cost: once DynamoDB finished disabling traffic at 5:45 AM, SQS's errors rose significantly. By 7:10 AM, DynamoDB's error rates were low enough for most customers and dependent services; Auto Scaling cleared its backlog at 10:52 AM.

One last detail is worth remembering. A leftover error rate of 0.15–0.25% looked too small to matter, but it was concentrated on a few customers, who saw much higher rates. AWS says it "should have posted" to the status dashboard sooner; the cause, one metadata partition not taking its full traffic, was fixed on Monday. An average can hide a few people having a terrible time.

What changed

From AWS's summary (the first already done, the rest announced):

• Significantly more capacity for the metadata service (done).
• Stricter monitoring of performance dimensions such as membership size, to plan capacity ahead.
• Fewer membership requests, more time allowed: storage servers ask less often, and queries get longer to finish.
• Segmenting DynamoDB into many instances of the metadata service, each serving only part of the storage fleet: cells, to "further contain the impact".
• Dependents hardened too: SQS caching so sending and receiving keep working without the metadata table, Auto Scaling processing backlogs in parallel, CloudWatch caching changes, and a short timeout on the Console login call.

Sources

• AWS: Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region, about 20 September 2015. Timeline, numbers, causes and fixes on this page come from here.
• The membership-timeout widget's numbers are made up for illustration; the mechanism is AWS's.

The setup

Amazon EBS gives EC2 servers network disks, called volumes. To keep data safe, each volume's data is copied to more than one storage node in an EBS cluster. If a node loses contact with the node holding its copy, it assumes that node failed and immediately makes a new copy somewhere else: re-mirroring. While that happens, the volume is blocked, and to the server using it, the disk looks "stuck". Normally, finding space for a new copy takes milliseconds.

Each cluster's nodes talk over two networks: a fast primary network for normal traffic, and a lower-capacity secondary network for backup connectivity, "not designed to handle all traffic from the primary network". Above the clusters sits the EBS control plane, which handles API requests like "create a volume", for all the Availability Zones in the region.

The trigger

At 12:47 AM PDT on 21 April 2011 (07:47 UTC), engineers made a routine change to upgrade the primary network in one Availability Zone of us-east-1. A standard step is to move traffic off one router onto another router on the primary network. The shift was "executed incorrectly", and the traffic went onto the secondary network instead, which couldn't carry it. Many nodes were cut off from each other completely.

The spread

Remember the rule from the setup: a node that loses contact with its partner assumes the partner failed and must make a new copy elsewhere. So when the change was rolled back and the network returned, a large number of nodes had already given up on their partners at the same moment, and instead of reconnecting to them, all of them started searching the cluster for free space. (AWS's fix, below, changes exactly this: reconnect to the previous copy first.) Here's why that matters:

The free space ran out almost immediately. Nodes that found none kept searching in a loop: a "re-mirroring storm". About 13% of the volumes in the zone got stuck. Then the storm reached the region:

Two details turned a zone problem into a region problem:

• A long timeout used up the threads. The stuck cluster couldn't serve "create volume" requests, and the control plane was set to wait a long time for each. The waiting requests filled its regional pool of threads (a fixed set of workers that take requests one at a time, like a worker pool), so it couldn't answer requests for the other zones either. AWS says better timeout logic "would have prevented the API impact from 12:50 AM PDT to 2:40 AM PDT".
• Retries flooded the control plane. Every re-mirror needs the control plane to agree which copy is the writable one. With nodes crashing and retrying, those negotiations overloaded it from 5:30 AM until the sick cluster was cut off at 8:20 AM.

RDS databases built on EBS were hit too: at the peak, 45% of single-zone RDS databases in the affected zone had stuck disks, because each database uses several volumes and one stuck volume can stop it.

Why it was hard to fix

• Searching harder made it worse. Nodes "did not back off aggressively enough". The flood of connection attempts also triggered a rare race condition in the node software that crashed nodes, which left more volumes needing new copies, which added more searches.
• Space can't be downloaded. The stuck volumes needed free storage, and failed nodes weren't reused until every copy on them was safe, deliberately, so data could still be recovered. So AWS had to physically move spare servers from across the region and install them.
• The fix fought the fix. The changes that stopped nodes from flooding each other (11:30 AM) also made it hard to add the new servers to the cluster without flooding the old ones again, so throttles had to be tuned carefully.

The way back

By 12:04 PM on 21 April the outage was contained to one zone and the cluster had stopped getting worse, with about 13% of its volumes still stuck. New capacity went in from about 2:00 AM on 22 April; by 12:30 PM all but about 2.2% of the zone's volumes were restored. Bringing the control plane back for that zone needed finer-grained throttles and a separate control plane instance just for it; API access returned at 6:15 PM on 23 April. Most of the last volumes were restored by hand from snapshots taken early on as a precaution; the final 1.04%, on failed machines with no snapshot, needed forensic recovery. In the end, 0.07% of the zone's volumes could not be restored in a consistent state.

What changed

From AWS's summary, mostly announced plans:

• A bigger capacity buffer for large recovery events, in capacity planning and alarms ("already increased... significantly").
• Retry logic that backs off more aggressively in a large interruption, and tries to reconnect to previous copies instead of "futilely searching for new nodes".
• A fix for the race condition, to be tested and deployed within weeks.
• Better timeouts so one slow zone can't exhaust the control plane's threads, zone-aware load shedding, and moving more of the control plane into per-cluster services in the same zone.
• Automating the recovery steps the team did by hand, and clearer, more frequent communication during events.

Sources

• AWS: Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region, 29 April 2011. Timeline, numbers, causes and fixes on this page come from here.
• The re-mirroring widget's numbers are made up for illustration; the mechanism and fixes are AWS's.

The setup

AWS's summary describes two networks. Customers' applications and most AWS services live on the main AWS network. Some of AWS's own foundational services, "monitoring, internal DNS, authorization services, and parts of the EC2 control plane", live on a separate internal network. Special networking devices connect the two, translating addresses between them (a kind of NAT).

(The AWS product names are explained in §05.)

The trigger

At 7:30 AM PST on 7 December 2021 (15:30 UTC), an automated activity to add capacity to one AWS service on the main network triggered "an unexpected behavior from a large number of clients" on the internal network. (AWS doesn't explain why scaling one service had that effect.) They opened a huge surge of connections, and the devices between the networks were overwhelmed: congestion.

The spread

The congestion hit AWS's own control planes, its support tools and, worst of all, its operators' view of the problem:

How a surge becomes a storm

A surge on its own passes. The trouble is what clients do when a request fails. AWS's summary describes the loop in one sentence: the delays "increased latency and errors for services communicating between these networks, resulting in even more connection attempts and retries. This led to persistent congestion and performance issues on the devices connecting the two networks."

That's a retry storm. Every failed request comes back as a retry, on top of the normal traffic, so the device that was overloaded gets more work, fails more requests, and receives more retries. AWS's clients were built to avoid this: they had "well tested request back-off behaviors". But "a latent issue prevented these clients from adequately backing off". The code path had been "in production for many years"; this was the first time the automated scaling triggered that behavior.

See the difference backoff and jitter make, with the same one-off burst:

The fixes in that widget are two small ideas:

• Exponential backoff: after each failure, wait longer before trying again (100 ms, 200 ms, 400 ms…), so a struggling service gets room to recover.
• Jitter: make each wait random, so clients that failed at the same moment don't all come back at the same moment. Without jitter, backoff just moves the spike later (the waves in the widget).

// retry calls fn until it succeeds, waiting a random time that grows after
// each failure: exponential backoff with "full jitter".
func retry(ctx context.Context, fn func(context.Context) error) error {
	const base, maxWait, tries = 100 * time.Millisecond, 10 * time.Second, 6
	for attempt := 0; ; attempt++ {
		err := fn(ctx)
		if err == nil {
			return nil
		}
		if attempt == tries-1 {
			return fmt.Errorf("giving up after %d tries: %w", tries, err)
		}
		ceiling := min(maxWait, base<<attempt) // 100ms, 200ms, 400ms, … at most 10s
		wait := rand.N(ceiling)                // ← the jitter: anywhere from 0 up to the ceiling
		select {
		case <-time.After(wait):
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

Retrying well has more to it (only retry safe operations, respect Retry-After, keep a retry budget); Idempotency §09 Retrying well covers it.

Why it was hard to fix: operating blind

The congestion "immediately impacted the availability of real-time monitoring data" for AWS's own operators, because the monitoring lived on the internal network. They worked from logs instead, spotted elevated errors from internal DNS (the name lookups AWS's own services use to find each other), and moved DNS traffic off the congested paths by 9:28 AM. That helped, but didn't end it, and monitoring was still not visible. AWS lists three reasons the rest of the fix "progressed slowly":

1. The impaired monitoring "limited our ability to understand the problem".
2. AWS's internal deployment systems run on the internal network too, so pushing changes was slow.
3. Many services and customer applications on the main network were still working, so the team moved deliberately, to avoid breaking them.

Even communication was caught: the congestion stopped the Service Health Dashboard tooling from failing over (switching) to its standby region (updates resumed by 8:22 AM), and customers' ability to create support cases was impacted from 7:33 AM until 2:25 PM, because the support contact center also relied on the internal network.

The way back

Congestion improved significantly by 1:34 PM, and all network devices had fully recovered by 2:22 PM. Some services needed hours longer to work through their backlogs, and the reason is worth a closer look.

Running things kept running

Look at what broke. Running EC2 instances were fine; the APIs to launch or describe them had increased error rates and latency. Existing load balancers stayed healthy; creating new ones was slow. DNS queries to Route 53 were answered normally; the Route 53 APIs for changing DNS records were impaired from 7:30 AM to 2:30 PM, preventing changes. S3 and DynamoDB worked, except through a particular private access path (VPC endpoints).

In the language of §01: the data plane mostly kept working, and the control plane (the APIs for creating and changing things) was impaired. That's the good outcome of static stability (the fail-static idea from §01): keep doing your job with the last settings you had, even when the manager is gone. §04 shows what happens when it runs out. It also explains the long tail: API Gateway servers that had got into a bad state needed replacing, which "was not possible until the EC2 APIs began recovering", and Fargate had a backlog of containers to start once launches worked again.

What changed

• Disabled the scaling activities that triggered the event until all fixes are in.
• Fixing the back-off bug: a fix in development, expected to deploy within two weeks of the summary.
• Network configuration that protects the affected devices "even in the face of a similar congestion event".
• API Gateway: working on mitigations so its servers stay healthy even when the internal network is unavailable, and on faster recycling.
• A new Service Health Dashboard "early next year", and a support system architecture that "actively runs across multiple AWS regions".

Sources

• AWS: Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, 10 December 2021. Timeline, numbers, causes and fixes on this page come from here.
• The retry simulation and its numbers are our own illustration, not AWS's data. The Go and JavaScript retry functions were run with Go 1.24 and Node 22 for this page.

The setup

This is a different part of Cloudflare from §03's story. Cloudflare's worldwide network of servers, the part that sits in front of customers' websites, is its data plane. Its control plane is primarily "the customer-facing interface for all of our services including our website and APIs": the dashboard and API customers use to change their settings. Separately, its analytics services handle logging and analytics reporting. Both went down.

That control plane ran mainly in three data centers near Hillsboro, Oregon, far enough apart that one disaster shouldn't hit all three, close enough to keep copies of the data in sync continuously. Most critical services ran on a high availability (HA) cluster spread across all three, so that losing any one facility wouldn't matter. Some didn't, on purpose or not yet: logging was deliberately left off it, and some newer products hadn't been moved onto it.

The largest of the three facilities, PDX-04, is run by a company called Flexential. It held Cloudflare's largest analytics cluster, more than a third of the HA cluster's machines, and was "the default location for services that have not yet been onboarded onto our high availability cluster".

The trigger

At about 11:40 UTC on 2 November 2023, a ground fault cut off PDX-04's remaining power, leaving only its batteries. How a building with three power sources went dark:

Cloudflare's account of the power failure is honest about its limits: from the point where Flexential chose to run generators alongside the remaining utility feed, Cloudflare says it didn't yet "have clarity from Flexential on the root cause", and says that some of what follows is "informed speculation". In outline:

• One utility power feed went down for unplanned maintenance at 08:50; generators started. Flexential didn't tell Cloudflare, and Cloudflare's tools couldn't see the change.
• At about 11:40 a ground fault on a transformer triggered a protective shutdown that also stopped all 10 generators.
• The UPS batteries, meant to last about 10 minutes, began failing after about 4 by Cloudflare's own equipment. Flexential employees told Cloudflare, unofficially, that three things slowed the restart: the generators had to be restarted by hand on site, the building's access control system (its electronic entry system) wasn't on battery backup, and the overnight shift had no experienced operations or electrical expert.

None of that is Cloudflare's own system, but its postmortem is clear about what it means: "we also must expect that entire data centers may fail". Losing PDX-04 was a failure the design was meant to survive.

The spread

Cloudflare's routers at PDX-04 went offline at 11:44. Step through the two days:

What was supposed to happen, and what did

Cloudflare had a plan for losing PDX-04. Compare it with what actually happened:

The difference is one arrow. Two critical services, Kafka (which moves log data) and ClickHouse (which stores and queries analytics), ran only in PDX-04 (Cloudflare had deliberately kept logging off the HA cluster). But some services on the HA cluster turned out to depend on them, and only some: Cloudflare says it "only needed to fail over a small percentage" of its control plane, while most services kept running on the two healthy facilities. Cloudflare: "Those dependencies shouldn't have been so tight, should have failed more gracefully, and we should have caught them."

Why weren't they caught? Cloudflare had tested its HA cluster by taking each of the other two facilities entirely offline (one at a time, and both together), and by taking the HA part of PDX-04 offline. It "had never tested fully taking the entire PDX-04 facility offline". The one test that would have exposed the hidden arrow was the one never run. Cloudflare also names a process gap: new products weren't formally required to move onto the HA cluster before being declared generally available, so "the redundancy protections we had in place worked inconsistently depending on the product".

Why it was hard to fix

• Waiting on the building. Generators came back at 12:48, but Cloudflare's circuit breakers turned out to be faulty, and Flexential didn't have enough replacements on hand, and couldn't give a restoration time.
• Products without a tested recovery plan. Some newer products, such as the Stream service for uploading videos, didn't come up properly at the disaster recovery sites, and some log processing services can't run there at all.

The way back

• Fail over to Europe. At 13:40 Cloudflare decided to move to its disaster recovery sites in Europe; the first services came up there at 13:43.
• A thundering herd at the recovery site. When services came up there, all the API calls that had been failing arrived at once and overwhelmed them, so Cloudflare added rate limits. By 17:57 the moved services were stable.
• Rebuilding from zero. Once PDX-04 had clean power (22:48), Cloudflare chose to let its exhausted team rest and start in the morning, a delay it accepted to avoid "additional mistakes". Because servers had probably lost power several times, the only safe path was a full rebuild: 3 hours to bring back the configuration management servers (which hold the setup every other server is built from), then 10 minutes to 2 hours per server, with some services depending on others. Everything was restored at 04:25 UTC on 4 November.

What changed

Cloudflare declared a "Code Orange": all non-critical engineering work shifted to the reliability of the control plane. Its list includes:

• Remove the control plane's dependencies on the core data centers where possible, and make it keep working even if all of them are offline.
• Require every generally available product to run on the HA cluster (if it relies on the core data centers at all), with no software dependencies on specific facilities, and to have a tested disaster recovery plan.
• Test the blast radius of system failures, and run more rigorous chaos testing (deliberately breaking things in a controlled way to check the system survives), including the full removal of each core data center.
• Audit the core data centers, and build a logging and analytics recovery plan that drops no logs even if all core facilities fail.

Sources

• Cloudflare (Matthew Prince): Post mortem on the Cloudflare Control Plane and Analytics Outage, 4 November 2023. Timeline, numbers, causes and fixes on this page come from here.
• The expected-versus-actual map is our simplification of the systems the postmortem names; it doesn't show every dependency Cloudflare had.

Side by side

How each one started:

Why it spread:

How recovery went and what changed:

The patterns

1. The trigger was routine, often automatic, sometimes just bad luck. Part 1's triggers were mostly human: a typo, a wrong terminal, a missed server. Here, a customer saved a setting, an engineer tidied a policy, a maintenance system did maintenance, a team added servers, a network blipped, a scaling system scaled, a power company did repairs. Two were mistakes inside routine changes: EBS's traffic shift was "executed incorrectly", and Cloudflare calls its 2022 re-ordering "our error". You can't stop these from happening; the defence has to be in the design.

2. The cause was waiting: for weeks, months or years. A bug deployed 27 days before (Fastly). A rollout plan that only reached the risky design at the end (Cloudflare BGP). A bug in maintenance software and two harmless-looking settings (Google). A thread limit that had always been there (Kinesis). Membership lists that had grown for months, without detailed enough monitoring (DynamoDB). Too little spare space and a rare race condition (EBS). A back-off bug "in production for many years" (AWS network). Dependencies on services in one building (Cloudflare power). Each one was a latent condition. Testing the path you expect is not enough; the rare path is where these live.

3. One change reached everything at once. Fastly's bug was already on its servers when one configuration set it off; Google's automation stopped clusters in several locations in one go. Cloudflare's rollout was staged, but every MCP location got the change in the same final step. The stage that matters is the first one that looks like the place that can break.

4. Retries turned a blip into an outage. DynamoDB's storage servers retried until healthy servers dropped out too. EBS nodes kept searching for space that wasn't there. AWS's 2021 clients failed to back off. DynamoDB's trigger was a brief blip and EBS's bad network change was rolled back, yet the retries kept the failure going. The fixes all make clients ask less or more gently: pausing requests (during DynamoDB's event), and better back-off (planned for EBS and AWS 2021).

5. Failures travelled along dependencies. Kinesis's outage impaired Cognito, CloudWatch, Lambda, EventBridge and the container services. DynamoDB's degraded SQS, Auto Scaling, CloudWatch and Console logins. One EBS zone's slow requests used up a thread pool that served every zone. AWS's congested internal network impaired the control planes of EC2, ELB, Route 53 and more. Cloudflare's dead facility took down parts of its control plane through Kafka and ClickHouse. Some dependencies were known but turned out tighter than expected (Cognito's "best effort" stream); others nobody had noticed at all (Cloudflare "discovered" its).

6. The operators' own tools were hit. Cloudflare's 2022 change cut engineers' own access to the locations they had to fix. Google's tools competed for the congested network. AWS's monitoring and deploy systems ran on the congested network in 2021; its status page tool needed Cognito during the Kinesis outage. DynamoDB's metadata service was too busy to accept the requests that would have added capacity. Part 1 called this the recovery dependency trap; in the cloud it is the rule rather than the exception, because the provider's internal tools run on the provider's own platform.

7. Recovery was a load problem of its own. Kinesis could only restart a few hundred servers an hour. EBS needed new hardware before stuck disks could move. Cloudflare's disaster recovery site was hit by a thundering herd of waiting API calls. Auto Scaling, API Gateway and Fargate had backlogs to work through. Coming back is a traffic spike, and the system has to survive it.

8. Running things kept running. The bright side: running EC2 instances, Route 53 DNS answers and Cloudflare's global network kept working while control planes were impaired, and Google's network kept forwarding for several minutes without its control plane. Where the data plane was built to survive without the control plane, customers were spared the worst. §11 makes this a rule.

Where the hours went

The best way to see patterns 6 and 7 is to split each outage into phases: until someone noticed, until the cause was known, until recovery started, and until it was over.

Where the sources give the times, detection was fast: one minute at Fastly, about two at Google, five minutes to an incident declaration at Cloudflare in 2022, and about four at Cloudflare in 2023 (from the ground fault to its routers going offline, though it never saw the earlier switch to generator power). The hours went into finding the cause with impaired tools, and above all bringing a big system back safely. EBS's bar shows the extreme: most of its 65 hours were spent finding space and draining backlogs. That's where the next section puts its effort: making failures smaller, so recovery is smaller too.

Rule 1: Split the system into cells

A cell is a complete, independent copy of a service that serves only part of the customers. A bad change, a poison input or a restart in one cell stays in that cell. AWS's Kinesis back end was already cellular; the front end, which failed, wasn't yet, and AWS's fix was to "greatly accelerate" that work.

Cells buy three things; the Kinesis summary names the first two:

• A smaller blast radius: one cell's customers, not everyone's. With 16 cells, a bad change in the first one reaches about 6% of customers.
• A tested size: each cell stays "within a previously tested and operated range", so a limit like Kinesis's thread count is reached in a load test, not in production. You grow by adding cells, not by growing cells.
• Faster recovery: a cell is small, so restarting it is quick. Restarting "many thousands of servers" at a few hundred an hour took Kinesis over 12 hours.

DynamoDB's 2015 fix points the same way: "many instances of the metadata service each serving only portions of the storage server fleet", so one overloaded metadata service can't take out every storage server. And EBS's 2011 outage shows what happens when one shared piece isn't split: a single slow zone used up a thread pool that served every zone.

Roll every change out one cell at a time, and the first cell is your canary. Cloudflare's 2022 outage adds the fine print: the early stages must include every kind of cell. Its rollout was staged, but "did not include an MCP data center until the final step", so the first place that could break got the change at the same moment as all the others like it. The hard part is the router in front of the cells, which must be extremely simple, because it's the one piece every customer shares.

Rule 2: Make the data plane statically stable

The data plane should keep serving on its last good settings when the control plane disappears: what Google calls failing static. Two things decide whether that works, and Google's fixes target both: how long the data plane can run alone (minutes weren't enough), and whether the control plane can restart without rebuilding its configuration. The same idea in code: refresh config when you can, keep the last good one when you can't, and keep a copy on disk so even a restart doesn't need the control plane.

// Router serves requests with the last configuration it managed to load.
type Router struct {
	cfg  atomic.Pointer[Config]
	path string // local copy on disk
}

// Refresh asks the control plane for new config. On failure it keeps the old one.
func (r *Router) Refresh(ctx context.Context, fetch func(context.Context) (*Config, error)) {
	cfg, err := fetch(ctx)
	if err != nil || cfg == nil {
		slog.Warn("control plane unreachable, keeping last good config", "err", err) // ← fail static
		return
	}
	r.cfg.Store(cfg)
	if b, err := json.Marshal(cfg); err == nil {
		tmp := r.path + ".tmp" // write a temp file, then rename: a crash mid-write can't corrupt the saved copy
		if os.WriteFile(tmp, b, 0o644) == nil {
			_ = os.Rename(tmp, r.path) // ← so a restart doesn't need the control plane either
		}
	}
}

// Start loads config from the control plane, or from disk if the control plane is down.
func (r *Router) Start(ctx context.Context, fetch func(context.Context) (*Config, error)) error {
	r.Refresh(ctx, fetch)
	if r.cfg.Load() != nil {
		return nil
	}
	b, err := os.ReadFile(r.path)
	if err != nil {
		return fmt.Errorf("no config from control plane or disk: %w", err)
	}
	var cfg Config
	if err := json.Unmarshal(b, &cfg); err != nil {
		return err
	}
	r.cfg.Store(&cfg)
	return nil
}

The test of static stability is blunt: switch off the control plane in a test environment and watch what the data plane does for an hour. If the answer is "stops after five minutes", you've found Google's 2019 problem before it finds you.

Rule 3: Give automation a blast radius too

Two of the eight triggers were automation doing its normal job (AWS's automated scaling and Google's maintenance system), and three more were routine operations work (Kinesis's capacity addition, Cloudflare's policy change, EBS's network upgrade). All of them need the same limits as people:

• Scope limits: one location, one cell, one percent at a time. Google said its fixes would make both the maintenance automation and the cluster manager refuse requests that span several physical locations, "regardless of origin".
• A stop button: AWS "immediately disabled the scaling activities that triggered this event"; Google "immediately halted the datacenter automation software". Every automated system that changes production needs a kill switch that people know about.
• Undo by default. Cloudflare's planned "commit-confirm" rollback undoes a network change automatically unless someone confirms it worked, which matters most when the change cuts off the very people who would undo it.
• Watch what it causes, not just what it does. In AWS's 2021 event, the harm came from how clients reacted to the scaling activity, "an unexpected behavior". Stage automated changes like deploys, and watch the systems around them, not just the one being changed.

Rule 4: Draw the dependency map, then test it

Two of these postmortems contain a dependency someone hadn't fully understood: Cognito's "best effort" stream and the status page tool's need for Cognito (both Kinesis), and some of Cloudflare's HA services leaning on Kafka and ClickHouse. A dependency map helps, but the map is only a theory. Cloudflare's lesson is that the map must be tested by removing things:

• Turn off whole units: a full data center, a full region, a full "optional" service, not just the parts you expect to lose. Reliability & SRE §12 covers how to do it safely.
• Test for hours, not seconds. Cognito's buffer survived short Kinesis blips by design; the latent bug only appeared in a long outage.
• Make "optional" really optional: a dependency marked best-effort must have a timeout, a bounded buffer and a tested path that drops data rather than blocking. A circuit breaker is one way to fail fast instead of piling up.

Rule 5: Retries must back off, and recovery must be gentle

DynamoDB, EBS and AWS's 2021 network event were all retry storms; Cloudflare's 2023 recovery hit a thundering herd. All are the same physics: many clients acting at the same moment.

• Every client gets exponential backoff with jitter and a cap on tries (the code in §08). Test it under real overload, because AWS's well-tested back-off had a latent bug.
• Servers protect themselves with rate limits, as Cloudflare did at its recovery site, and load shedding: it's better to answer some requests quickly than all of them never. Give operators a big lever too: DynamoDB only recovered once AWS paused requests to its metadata service.
• Prefer the known-good path when retrying: AWS's planned EBS fix has nodes reconnect to their previous copies instead of searching for new space.
• Keep a buffer for recovery, not just for normal load: EBS ran out of the spare space a mass re-mirror needed, and AWS's first fix was carrying that extra capacity.
• Plan recovery as a traffic event: bring servers back gradually (Kinesis's few hundred an hour was the safe pace for its design), warm caches before taking full traffic, and let backlogs drain at a controlled rate.

Rule 6: Keep the tools you need off the thing they watch

Monitoring, deploy systems, status pages, admin access and support tools were caught in five of the eight outages (Cloudflare 2022, Google, Kinesis, DynamoDB, AWS network), and Cloudflare's monitoring couldn't see the switch to generator power in 2023. For each tool you would reach for in an incident, ask: does it work if this region, this network, this login system or this data center is down?

Rule 7: Find your limits before production does

Kinesis's thread limit was always there; the fleet simply hadn't reached it. DynamoDB's time limit was always there; the membership lists simply hadn't grown that big, and their size wasn't monitored in enough detail. Fastly's bug was always there; no configuration had hit it.

• For every resource with a hard limit (threads, file descriptors, connections, memory, IP addresses), alert on consumption against the limit, as AWS said it was adding for threads. "80% of the thread limit" is a warning you can act on; "thread creation failed" is an outage.
• Load-test past your current size, by a lot, to find the next scaling limit while it's cheap. Load Testing §09 shows how.
• Feed your systems valid but unusual input on purpose: fuzzing, every combination of options, very large and very empty values (§02).

Work out your answer before you click. Every question is covered above, and each explanation names the section.

Words to know: data plane (does the work for each request) · control plane (manages the data plane) · fleet (all servers of one service) · cell (an independent slice of a service) · fail static, or static stability (keep running on the last good instructions) · latent bug (in production, not yet triggered) · retry storm (retries that keep an overload going) · backoff and jitter (wait longer, and a random amount, before each retry) · thundering herd (many clients arriving at the same moment) · IP prefix (a block of addresses a network announces).

Before you add capacity, run automation or ship a config change, ask: what's the smallest first step, and does it look like the place that can break? What limit could this push us past? If this breaks, what else breaks, and can I still see and fix it?