Famous Outages, Animated

Pilots study other pilots' crashes. They read the investigation reports, find the chain of small events, and practise the moment where the chain could have been broken. Software engineers can do the same, because many companies publish a Postmortem after a serious Incident: a written account of what happened, why, and what they changed.

This page replays six famous ones, step by step, using the companies' own words:

Each section follows the same path: a timeline you can play, an animation of the mechanism (what actually broke), what the company changed, the concept behind it with a link to the topic that teaches it, and the sources. After the six, §08 compares them side by side, §09 turns the lessons into design rules, and the page ends with a quiz and a cheat sheet.

Why bother with other people's outages?

• The mistakes repeat. A typo in a command, a backup nobody restored, a deploy that missed one server: you will see versions of these at your own job. It's cheaper to learn them from someone else's bad day.
• They show where safety nets go. Every outage here was stopped, or could have been stopped, by a specific check. Seeing the check that was missing teaches you where to put yours.
• They make abstract advice concrete. "Use staged rollouts" sounds like bureaucracy until you watch a firewall rule reach every server on Earth within seconds.

This page builds on Reliability & SRE. You don't need all of it, but §09 Blameless postmortems explains what a postmortem is for and why it never blames a person.

How to read a postmortem

Two habits make postmortems far more useful:

1. Separate the trigger from the causes. "An engineer typed the wrong number" is the trigger. The causes are why the tool accepted it, why it affected so much at once, and why recovery took hours. The trigger is almost always something that will happen again; the causes are what you can fix.
2. Look for the gaps in the timeline. Minutes between the trigger and the first alert, or between "we know what's wrong" and "it's fixed", point to the most valuable improvements. In several stories here, the fix was known quickly but the tools needed to apply it were broken too.

Cloudflare sits in front of its customers' websites: requests to those sites pass through Cloudflare's servers first, its edge. One of the things those servers do is run a WAF, a set of rules that inspect each request and block attacks. Cloudflare updates those rules constantly; in the 60 days before the postmortem it handled 476 change requests for them, about one every three hours.

On 2 July 2019 one of those updates contained a Regular expression that, on some inputs, made the matching engine try an enormous number of possibilities. As the rule spread, the CPU cores serving HTTP and HTTPS traffic on Cloudflare's network worldwide hit nearly 100%, and visitors to Cloudflare-protected sites saw 502 Bad Gateway errors (a HTTP status code meaning "the server in the middle couldn't get a response from the one behind it").

How one line of regex ate a CPU

The rule's regular expression was long, but Cloudflare's postmortem points to one critical part, .*(?:.*=.*), which behaves like the simpler .*.*=.*: "anything, then anything, then an = sign, then anything".

The engine Cloudflare used, PCRE, is a backtracking engine: it tries one way to match, and when that fails, it goes back and tries the next. For .*.*=.* on a line with no = in it, the answer is "no match", but the engine can only conclude that after trying every way to split the line between the two .* parts, starting from every position. Count them for the line xxx:

• Starting at the first x, three characters are left. The first .* can take 3, 2, 1 or 0 of them, and the second .* any number of what remains: 1 + 2 + 3 + 4 = 10 splits, and each one then fails to find =.
• Starting at the second x (two characters left): 3 + 2 + 1 = 6. At the third: 3. At the very end: 1.
• Total: 20 dead ends for three characters.

The pattern is a sum of sums. For a line of n characters it comes to exactly (n+1)(n+2)(n+3) ÷ 6 dead ends, roughly n³ ÷ 6 for long lines: 1,771 for 20 characters, 12,341 for 40. That's the cube of the length: double the line, about eight times the work.

Here's the same pattern run for real with Node's regex engine (V8), which also backtracks, next to Go's regexp, which doesn't:

package main

import (
	"fmt"
	"regexp"
	"strings"
	"time"
)

func main() {
	re := regexp.MustCompile(`.*.*=.*`) // Go's engine: linear time, no backtracking
	for _, n := range []int{500, 1000, 2000, 4000, 100000} {
		s := strings.Repeat("x", n) // a line with no "=" in it
		t := time.Now()
		ok := re.MatchString(s)
		fmt.Printf("%d chars: %v, match=%v\n", n, time.Since(t).Round(time.Microsecond), ok)
	}
}

500 chars: 47µs, match=false
1000 chars: 37µs, match=false
2000 chars: 73µs, match=false
4000 chars: 160µs, match=false
100000 chars: 4.128ms, match=false

500 chars: 19 ms
1000 chars: 139 ms
2000 chars: 1063 ms
4000 chars: 8300 ms

(Measured with Go 1.24 and Node 22, the versions installed where this page was written; the site targets Go 1.26 and Node 24. Neither engine's basic strategy changed between those versions, but your exact times will differ.)

Each doubling of the line makes V8 about 8× slower: 4,000 characters take over 8 seconds of one CPU core. Go handles 100,000 characters in about 4 ms. The full story of why (and the patterns to watch for) is in Regular Expressions §07 Catastrophic backtracking and §08 How Go matches.

Why it reached every server at once

Cloudflare's normal software releases go out in stages across its points of presence (PoPs): first a PoP used only by employees (dogfooding), then a small slice of free customers' traffic, then three "canary" PoPs, then the world, over hours or days. WAF rules skipped all of that, by design, because new attacks need new rules fast. Rules were a config push through Quicksilver, Cloudflare's distributed key-value store (a database that maps names to values, copied to every machine), which reached every machine worldwide in a couple of seconds.

The new rule was even deployed in "simulate" mode, which logs what it would block without blocking anything. But to know what it would block, the rule must run, so the expensive regex still ran on real traffic. A staged rollout, like the one Cloudflare uses for software, would have started with a single location instead of all of them at once; §07 lets you watch the difference.

What made it worse, and what slowed the fix

Cloudflare's postmortem lists the causes that "converged", including:

• A protection that would have limited a regex's CPU use had been removed by mistake during a refactoring a few weeks earlier.
• The regex engine had no guarantee on how long a match could take.
• The test suite checked what rules blocked and allowed, but not how much CPU they used.
• The procedure for rule changes allowed a non-emergency change to go global without a staged rollout.
• Once the cause was clear, Cloudflare's own Access login service was down too, so staff couldn't log in to the internal control panel to flip the switch, and had to use a rarely practised bypass. Some had lost access because their credentials had timed out for security reasons.

What Cloudflare changed

From the postmortem's list (with the status it gave on 12 July 2019):

• Re-introduced the CPU-usage protection that had been removed (done).
• Manually inspected all 3,868 WAF rules for other risky backtracking (inspection complete).
• Added performance profiling for every rule to the test suite (planned for 19 July).
• Switching to the RE2 or Rust regex engine, which both have run-time guarantees (planned for 31 July).
• Staged rollouts for rule changes, like other software, while keeping an emergency global path for active attacks (no date given).
• A way to take the Cloudflare dashboard and API off Cloudflare's own edge in an emergency, and automatic status page updates (no date given).

Sources

• Cloudflare, John Graham-Cumming: Details of the Cloudflare outage on July 2, 2019, 12 July 2019. Timeline, causes, fixes, the regex and its step counts.
• Timings of the code above: measured for this page, as labelled.

Amazon S3 is Object storage: a service where programs store and fetch files ("objects") over HTTP. Customers' applications keep their data in it, and so do other AWS services. AWS runs S3 separately in each region; this outage hit us-east-1 in Northern Virginia.

On the morning of 28 February 2017, the S3 team was debugging a billing system that was running slower than expected. An authorized engineer, using an established playbook, ran a command to take a small number of servers out of a subsystem used by that billing process. One of the command's inputs was typed incorrectly, and a much larger set of servers was removed, including servers belonging to two subsystems that S3 can't work without.

What the removed servers did

AWS's summary describes two subsystems that lost capacity:

• The index subsystem knows the metadata and location of every object in the region. Every GET (read), LIST, PUT (write) and DELETE needs it.
• The placement subsystem decides where new objects are stored. Uploads (PUT) need it, and it needs the index subsystem to work.

Losing a significant part of their capacity meant both had to be fully restarted, and while they restarted S3 couldn't serve requests. S3's subsystems are designed to lose significant capacity with "little or no customer impact", but AWS wrote that it hadn't completely restarted these subsystems in its larger regions "for many years". S3 had grown enormously since then, and restarting them, including safety checks on the integrity of the metadata, took longer than expected.

Why a typo could do this much

The command itself was routine: removing capacity is something AWS does all the time. What was missing was a check on how much it removed. Play with the same situation below. The numbers are made up (AWS didn't publish server counts), but the logic is the fix AWS described:

In AWS's words, "the tool used allowed too much capacity to be removed too quickly". The fix wasn't "be more careful when typing". It was a guardrail in the tool:

#!/usr/bin/env bash
# remove-capacity.sh <pool> <count>: take <count> servers out of <pool>, but never below its minimum.
set -euo pipefail

refuse() { echo "refusing: $*" >&2; exit 2; }

pool="$1"
count="$2"
[[ "$count" =~ ^[1-9][0-9]{0,4}$ ]] || refuse "count must be a whole number from 1 to 99999, got '$count'"

mapfile -t servers < "pools/$pool.servers"   # read the list ONCE; count and drain use the same array
for s in "${servers[@]}"; do
  [[ "$s" =~ ^[a-z0-9]([a-z0-9.-]*[a-z0-9])?$ ]] || refuse "bad line in pools/$pool.servers: $(printf '%q' "$s")"
done
dups=$(printf '%s\n' "${servers[@]}" | sort | uniq -d)
[[ -z "$dups" ]] || refuse "listed twice in pools/$pool.servers: $dups"
in_service=${#servers[@]}
(( in_service > 0 )) || refuse "pool $pool has no servers listed"

minimum=$(cat "pools/$pool.min")             # the pool's minimum required capacity
[[ "$minimum" =~ ^(0|[1-9][0-9]{0,4})$ ]] || refuse "minimum in pools/$pool.min is not a number: '$minimum'"

if (( count > in_service - minimum )); then  # ← the guardrail
  echo "refusing: removing $count of $in_service would leave $((in_service - count)), minimum is $minimum" >&2
  exit 1
fi

for server in "${servers[@]:0:count}"; do    # the first <count> servers in the list
  echo "draining $server"
  sleep 30                                   # ← slowly: one at a time, time to notice
done

$ ./remove-capacity.sh index 2
draining index-01
draining index-02
$ ./remove-capacity.sh index 12
refusing: removing 12 of 20 would leave 8, minimum is 12
$ ./remove-capacity.sh index 1O
refusing: count must be a whole number from 1 to 99999, got '1O'
$ ./remove-capacity.sh index 9223372036854775816
refusing: count must be a whole number from 1 to 99999, got '9223372036854775816'

Most of this script is input checking, and every check is there because an earlier draft of it could be fooled. Without them, -5 slipped past the arithmetic and head -n -5 drained 15 servers; 1O (a letter O) made the check itself fail, which bash ignores inside an if, so the script carried on; an empty minimum file silently counted as 0; a minimum written as 012 was read as octal (10); a 20-digit count wrapped around bash's 64-bit integers into a small number; a blank line in the server list was counted as a server but never drained, so each blank line let the script go one more server below the minimum; and a server listed twice (even as index-20 and index-20.) was counted twice, with the same effect. Reading the list once into an array, refusing any line that isn't a single server name, and refusing duplicates closed those holes. A guardrail has to validate everything it reads before it trusts the arithmetic, which is exactly the lesson of this outage.

The status page that depended on S3

While S3 was down, AWS couldn't update the individual services on its own Service Health Dashboard: the dashboard's administration console depended on S3. Until 11:37 AM PST, AWS posted updates on Twitter and in a banner instead. We'll come back to this trap, a recovery tool that depends on the broken system, in §09.

What AWS changed

• The tool now removes capacity more slowly, and has safeguards that stop it taking any subsystem below its minimum required capacity. AWS said it was also auditing its other operational tools for similar checks.
• Faster recovery: AWS splits services into small partitions it calls cells, so that a restart or failure affects only part of the system. It had planned to split the index subsystem further later that year, and moved that work up to start immediately.
• The Service Health Dashboard's admin console was changed to run across multiple AWS regions.

Sources

• Amazon Web Services: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region, about the event of 28 February 2017. Timeline, subsystems, causes and fixes. Times in PST.
• The shell script is our own illustration of the guardrail AWS describes, run for this page as shown.

GitLab.com, GitLab's hosted service for Git repositories, issues and merge requests, ran on one main PostgreSQL Database. GitLab's setup had a single primary that handled all the load and a single secondary, a Replica in hot standby that only existed to take over if the primary failed. (Replicas, and how they stay in sync, are covered in Replication §03 Leader and followers.)

On the evening of 31 January 2017, a database load problem turned into a replication problem, and close to midnight UTC, while trying to fix replication, an engineer ran a delete on the wrong machine.

How the wrong directory got deleted

The secondary had fallen too far behind. PostgreSQL replicas follow the primary's write-ahead log, and the primary had already thrown away log segments the secondary still needed (GitLab wasn't archiving them). The only way back was to rebuild the secondary from scratch: delete its data directory, then run pg_basebackup to copy everything over from the primary.

That copy kept hanging with no useful output. After several attempts and config changes, one engineer suspected an earlier attempt had left files behind and decided to wipe the directory again. But the command ran on the primary. When they noticed, they stopped it within a second or two, but about 300 GB of data was already gone.

Then the team went to the backups

On paper GitLab had several independent copies of its data. Check each one the way the team did:

Two failures stand out because they were silent:

• The nightly pg_dump backup was a cron job on an application server. The packaging decided which PostgreSQL tools to use by looking for a local database; there wasn't one on that server, so it picked version 9.2 tools, which refused to dump the 9.6 database and exited with an error, every time it ran.
• Failures did send an email. But the emails weren't signed for DMARC, so the receiving mail server rejected them. Nobody ever saw one.

The LVM snapshot that saved the day had been taken by hand at about 17:20, for unrelated testing. Restoring from it meant copying the data off the disks of the staging environment, disks that were cheap, network-attached and throttled to around 60 Mbps: about 18 hours of copying.

How bad was it?

GitLab.com was down for about 18 hours. Database changes made between 17:20 UTC and the deletion were lost for good: GitLab estimated at least 5,000 projects, 5,000 comments and about 700 user accounts. (The postmortem's introduction gives the end of the window as 00:00 UTC; its "Data loss impact" section says 23:30.) Git repositories and wikis are stored separately and were not lost. Customers running their own GitLab installations were not affected.

GitLab handled the communication unusually openly: a public document with live notes, and a YouTube livestream of the recovery that peaked at around 5,000 viewers.

A backup only counts once you've restored it

GitLab's own "five whys" ends on the deepest cause: the backup procedure wasn't tested regularly "because there was no ownership, as a result nobody was responsible for testing this procedure." The fix is a job that restores the newest backup every day, checks the result, and fails loudly through the same alerting you use for production:

#!/usr/bin/env bash
# restore-test.sh: prove the newest backup can actually be restored. Run it every day.
set -euo pipefail

backup_dir="${BACKUP_DIR:-/backups}"
backup=$(ls -t "$backup_dir"/*.dump 2>/dev/null | head -n 1 || true)  # newest backup file
if [[ -z "$backup" ]]; then
  echo "FAIL: no backup found in $backup_dir" >&2; exit 1
fi
age_hours=$(( ($(date +%s) - $(stat -c %Y "$backup")) / 3600 ))  # stat -c is GNU/Linux
if (( age_hours > 26 )); then
  echo "FAIL: newest backup is ${age_hours}h old" >&2; exit 1
fi

dropdb --if-exists restore_test
createdb restore_test
pg_restore --no-owner --exit-on-error -d restore_test "$backup"  # ← the real test

users=$(psql -d restore_test -tAc "select count(*) from users")
if (( users == 0 )); then
  echo "FAIL: restored database has no users" >&2; exit 1
fi
echo "OK: restored $(basename "$backup"), $users users"

$ ./restore-test.sh
OK: restored app-2026-11-05.dump, 1250 users
$ ./restore-test.sh
pg_restore: error: input file does not appear to be a valid archive (too short?)
$ ./restore-test.sh
FAIL: newest backup is 72h old
$ BACKUP_DIR=empty ./restore-test.sh
FAIL: no backup found in empty

What GitLab changed

From the postmortem's list of follow-up issues:

• Prometheus monitoring for backups, and automated testing of restoring PostgreSQL backups.
• An owner for data durability.
• Hourly LVM snapshots instead of daily, an investigation of point-in-time recovery (continuous WAL archiving, see Replication §11 Running replicas in production), and a look at Azure disk snapshots for the database servers.
• A clearer shell prompt (PS1) on every host, so it's obvious which machine and environment you're typing into.
• A sane max_connections value (it was set to 8,000, which stopped PostgreSQL restarting during the night's repair attempts until it was lowered), and better replication runbooks.

GitLab was explicit about the philosophy: rather than preventing engineers from running certain commands, it focused on "improve disaster recovery, and making it more obvious as to what host you're using". As it put it, an ideal environment is "one in which you can make mistakes but easily and quickly recover from them."

Sources

• GitLab: Postmortem of database outage of January 31, 10 February 2017. Timeline, backup failures, data loss estimate, five whys and follow-up issues. Times in UTC.
• The restore-test script is our own illustration, run for this page as shown.

Knight Capital was a large U.S. broker-dealer: a firm that trades stocks for its customers and for itself. According to the SEC, its trading was roughly 10% of all trading in listed U.S. stocks in 2011–2012. Orders went to the stock exchanges through an order router called SMARS.

This story is told by the U.S. Securities and Exchange Commission, in the order that settled its case against Knight in October 2013. Knight agreed to the order without admitting or denying its findings.

The setup: a new feature, an old flag

The New York Stock Exchange was launching a new Retail Liquidity Program (RLP), a way of handling orders from individual (retail) investors, on 1 August 2012, and Knight wrote new SMARS code for it. SMARS's job is to take a customer's parent order and send smaller child orders to exchanges until the parent is filled.

• The new RLP code replaced old, unused code called Power Peg, which Knight had stopped using in 2003 but never removed. It was still present and could still be called: Dead code that wasn't really dead.
• The new code also reused the flag that used to switch Power Peg on. Here the flag was a marker carried on each incoming order, not a setting on the server: with Power Peg deleted, an order marked with that flag would now get the RLP treatment instead.
• In 2005 Knight had moved the part of Power Peg that counted filled shares (the part that tells it to stop) to an earlier point in the code, and never retested Power Peg afterwards.

Starting on 27 July 2012, Knight's technicians copied the new code onto SMARS's eight servers, a few at a time on successive days. One technician didn't copy it to one of the eight. Knight didn't require a second technician to review deployments and had no written deployment procedure for SMARS, so nobody noticed.

The morning of 1 August

Two details from the SEC's order deserve a second look:

• There was a warning. Starting around 8:01 a.m., before the market opened, Knight's systems sent 97 automated emails mentioning SMARS and "Power Peg disabled". They weren't designed as alerts, people generally didn't read them, and nobody acted on them before or after the open.
• The "rollback" made it worse. Knight had no procedures for responding to incidents, so engineers debugged in live trading. Removing the new code from the seven good servers put Power Peg back on all eight, and orders kept arriving with the flag. The SEC says this "worsened the problem". Our takeaway (not the SEC's words): a rollback is only safe if the old code still understands everything the world is now sending it.

The SEC also found that Knight had no automated way to stop SMARS when its own output looked wrong: no control comparing the orders leaving SMARS with those entering it, and its main risk monitor relied entirely on people watching a screen, with no automated alerts.

How much damage

From 212 customer orders, SMARS sent millions of child orders, getting about 4 million executions in 154 stocks for more than 397 million shares in about 45 minutes. Knight ended up holding about $3.5 billion of stocks it didn't want to own (long positions in 80 stocks) and owing about $3.15 billion of stocks it had sold without owning (short positions in 74 stocks). Closing those positions lost it more than $460 million.

It hurt other investors too. For 75 of the stocks, Knight's trades were over 20% of the volume and helped move prices by more than 5%; for 37 of them, prices moved more than 10% while Knight was over half the volume. Some traders got worse prices because of it, others better. The SEC fined Knight $12 million for not having the risk controls the rules require.

Make "every server got it" a gate

The cheapest safety net here is a check that every server is running the version you just shipped, before the feature goes live:

#!/usr/bin/env bash
# verify-deploy.sh <expected-version>: every server must report the version we just shipped.
set -euo pipefail

expected="$1"
checked=0
bad=0
while IFS= read -r host || [[ -n "$host" ]]; do         # servers.txt: one host:port per line
  [[ -z "$host" ]] && continue                          # skip blank lines
  checked=$((checked + 1))
  got=$(curl -fsS --max-time 5 "http://$host/version" || echo "unreachable")
  if [[ "$got" != "$expected" ]]; then
    echo "MISMATCH $host is on: $got" >&2
    bad=$((bad + 1))
  fi
done < servers.txt

if (( checked == 0 )); then
  echo "no servers listed in servers.txt: do NOT go live" >&2
  exit 1
fi
if (( bad > 0 )); then
  echo "$bad of $checked server(s) not on $expected: do NOT go live" >&2   # ← the gate
  exit 1
fi
echo "all $checked servers on $expected"

$ ./verify-deploy.sh rlp-2.0
MISMATCH 127.0.0.1:4732 is on: rlp-1.9
curl: (7) Failed to connect to 127.0.0.1 port 4733 after 0 ms: Couldn't connect to server
MISMATCH 127.0.0.1:4733 is on: unreachable
2 of 3 server(s) not on rlp-2.0: do NOT go live
$ ./verify-deploy.sh rlp-2.0
MISMATCH 127.0.0.1:4732 is on: rlp-1.9
1 of 2 server(s) not on rlp-2.0: do NOT go live

In a modern setup the deploy tool does this for you: an automated, scripted deploy (see CI/CD §05 Build once, deploy many) puts the same artifact on every machine and checks it, and a rollout controller refuses to continue if any instance isn't healthy on the new version.

What the SEC said was missing

The order lists the failures behind the event. In plain terms:

• Deployment: no written procedure for deploying SMARS code, no second-person review, and no rule for testing old code that was being reused.
• Dead code: code no longer meant to be used was left on servers that could reach the market.
• Stopping: no automated control to halt SMARS on abnormal activity, and capital limits that weren't linked to anything that could block orders automatically.
• Incident response: no procedures to guide staff when a serious technical problem began.

Sources

• U.S. Securities and Exchange Commission: Order in the Matter of Knight Capital Americas LLC, Release No. 70694, File No. 3-15570, 16 October 2013. All facts, numbers and times above (Eastern Time). Knight consented to the order without admitting or denying the findings.
• The verify-deploy script is our own illustration, run for this page against local test servers.

On 4 October 2021, Facebook, Instagram and WhatsApp stopped working worldwide. To the outside world, the first sign was DNS: names like facebook.com simply stopped resolving to an address. Inside Facebook, the problem had started one layer deeper, in the network that connects its own buildings.

Two pieces of networking help here. If they're new to you, skim DNS §02 What happens when you type example.com and TCP/IP §05 Routing first, or just read on:

• DNS turns a name into an IP address. Your phone asks a resolver, and the resolver asks the domain's own name servers (its authoritative servers). Facebook ran its own.
• BGP is how networks tell each other which IP addresses they can reach. A network makes a route advertisement, "send traffic for these addresses to me"; withdrawing it tells everyone to stop.

Inside: what Meta says happened

Meta (Facebook's parent company) published two posts. The second explains the chain. Step through it:

Notice that nothing in step 3 is a bug. Withdrawing routes when you can't reach your data centers is a good rule for one site with a broken connection: traffic goes to a healthy site instead. It only became a disaster because every site lost its connection at the same moment, so there was no healthy site left. Health checks that can all fail together can take everything down together.

Outside: what the rest of the internet saw

Meta's posts give no times. Cloudflare, which runs the public DNS resolver 1.1.1.1 and watches routing data from its global network, published what it observed that day:

From any machine, a lookup looked like this. This is an excerpt of Cloudflare's own output (it also queried whatsapp.com), run against its resolver (1.1.1.1) and Google's (8.8.8.8). Look at status: SERVFAIL:

➜  ~ dig @1.1.1.1 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com.			IN	A
➜  ~ dig @8.8.8.8 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com.			IN	A

SERVFAIL means "I tried, but couldn't get an answer". The resolvers were fine; Facebook's name servers were still running too, but no route led to them. (DNS §08 "It's always DNS" shows how to read dig output and the other ways DNS fails.)

Why recovery took hours

The fix was to bring the backbone back. Getting to it was the problem:

• Meta's normal access to its data centers and its out-of-band access were down, because both depended on the network that was down.
• Losing DNS broke many of the internal tools engineers would normally use to investigate.
• So engineers went to the data centers in person. Those buildings are designed to be hard to get into, and the routers hard to modify even with physical access. Meta called the slower recovery a trade-off it believes is worth it: "greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this."

Bringing everything back at once had its own risk. Meta wrote that individual data centers had seen power use drop by tens of megawatts, and suddenly reversing that could endanger "everything from electrical systems to caches". It had practised for this with "storm" drills that take a service, data center or whole region offline, though never one simulating the whole backbone going down. Services came back gradually, without further system-wide failures.

What Meta changed

Meta's posts are lighter on specifics than the others here. They say:

• The audit tool that should have stopped the command had a bug, and the incident was triggered by a faulty configuration change on Meta's side. No malicious activity was involved, and Meta had no evidence that user data was compromised.
• Meta would look for ways to simulate a backbone-wide failure in its storm drills, and would strengthen its "testing, drills, and overall resilience".

Sources

• Meta Engineering, Santosh Janardhan: Update about the October 4th outage, 4 October 2021 (updated 5 October). Cause in brief; no malicious activity; no evidence of user data compromise.
• Meta Engineering, Santosh Janardhan: More details about the October 4 outage, 5 October 2021. The command, the audit-tool bug, the DNS/BGP rule, on-site recovery, storm drills.
• Cloudflare, Celso Martinho and Tom Strickx: Understanding how Facebook disappeared from the Internet, 4 October 2021. All times (UTC), the route withdrawals, the dig output and the 30× query increase, as observed from outside.

CrowdStrike sells security software for company computers. Its Falcon sensor runs on each machine, watches what programs do, and blocks attacks. To see everything, part of it runs as a kernel driver, loaded early when Windows boots. (Why the kernel is special is covered in Threads & Processes under user mode vs kernel mode.)

The sensor gets new detection rules in two ways. Sensor Content ships with each sensor release, which customers can schedule and which goes out in stages. Rapid Response Content is configuration data that CrowdStrike pushes from its cloud to respond to new threats quickly, several times a day. This incident came through the second path.

Templates, instances and field 21

CrowdStrike's root cause analysis uses a few names worth learning:

• A Template Type is built into the sensor's code. It defines a kind of detection and the input fields it can look at.
• A Template Instance is a rule written by a threat-detection engineer for one template type: "match when field 3 looks like this and field 7 looks like that". Instances are bundled into Channel Files and pushed as Rapid Response Content.
• A Content Validator checks new instances in the cloud before they ship; the Content Interpreter in the sensor runs them.

Sensor 7.11, released in February 2024, added an IPC Template Type (IPC = interprocess communication) for spotting attacks that abuse named pipes (a way Windows programs talk to each other). Its definition said it had 21 input fields. The sensor code that called the interpreter supplied 20. Try what happens:

Every test case and every instance before 19 July put a wildcard in field 21, so the interpreter never had to read it. CrowdStrike's root cause analysis says so directly ("due to the use of wildcard matching criteria for the 21st input during testing and in the initial IPC Template Instances"), and its fix to the validator, allowing only wildcards in field 21, "prevents the out-of-bounds access".

Why a bad read crashes the whole machine

Reading index 20 of a 20-item array is an Out-of-bounds read. What happens next depends on the language and where the code runs. In Go and JavaScript you can see the difference a bounds check makes:

package main

import "fmt"

// getInput returns field i (0-based) of the inputs, or an error if the rule
// asks for a field that doesn't exist.
func getInput(inputs []string, i int) (string, error) {
	if i < 0 || i >= len(inputs) { // ← the bounds check
		return "", fmt.Errorf("field %d requested, only %d supplied", i+1, len(inputs))
	}
	return inputs[i], nil
}

func main() {
	inputs := make([]string, 20) // the sensor supplies 20 inputs

	if _, err := getInput(inputs, 20); err != nil { // the rule asks for the 21st
		fmt.Println("skipping rule:", err)
	}

	fmt.Println(inputs[20]) // no check: Go stops the program here
}

skipping rule: field 21 requested, only 20 supplied
panic: runtime error: index out of range [20] with length 20

goroutine 1 [running]:
main.main()
	…/main.go:21 +0xb2
exit status 2

skipping rule: field 21 requested, only 20 supplied
undefined

The sensor's interpreter is written in C++, which doesn't check array bounds by itself. Its inputs were an array of 20 pointers: each entry holds the memory address of one input string. CrowdStrike's crash analysis shows what happened when the code asked for entry 21. Reading that slot worked, but the slot sat just past the end of the array and held some other value (ffffd6030000006a in the dump), not the address of a string. The code treated that value as an address and followed it, into memory that wasn't mapped. Doing that inside the kernel is an error Windows can't contain, so it stops the machine with a blue screen (the crash dump in the RCA names PAGE_FAULT_IN_NONPAGED_AREA in csagent.sys).

Why it reached so many machines

Sensor releases went through a staged rollout: internal dogfooding, then early adopters, then everyone, and customers could choose to run the latest version or one or two versions older. Rapid Response Content had no such stages: one of CrowdStrike's fixes is to "implement a staggered deployment strategy for Rapid Response Content … starting with a canary deployment". The two new instances went to Windows machines running sensor 7.11 or later that were online between 04:09 and 05:27 UTC. CrowdStrike's report says it deployed them based on the earlier stress test, its trust in the validator, and the earlier instances that had worked.

Compare the two ways to ship one bad update:

What CrowdStrike changed

From the PIR and the RCA, each fix maps to one of the causes:

• Count the fields at build time: the content compiler now validates the number of inputs a template type provides (in production 27 July 2024).
• Check bounds at run time: the interpreter now checks the index before reading an input, and checks that the input array has as many entries as the content expects (added 25 July; the RCA of 6 August says the fix was being backported to sensors 7.11 and above in a hotfix due by 9 August).
• Fix the validator: only wildcards allowed in field 21 for this template (done), and new checks that a rule doesn't match on more fields than are supplied (due in production by 19 August, per the RCA).
• Test what production uses: automated tests with non-wildcard values for every field of every template type.
• Stage the content: new deployment rings, starting with a small "canary" group of machines, with "bake-in" time and telemetry before each next ring, and customer control over when and where Rapid Response Content is deployed.
• Two independent third-party security vendors reviewing the sensor code, plus an independent review of the end-to-end quality process.

Sources

• CrowdStrike: Falcon Content Update Preliminary Post Incident Report, 24 July 2024 (updated 25 July). Timeline of the IPC template, the release process, the planned fixes.
• CrowdStrike: Technical Details: Falcon Content Update for Windows Hosts, 20 July 2024. Times of the update and the revert (UTC), Channel File 291, affected systems.
• CrowdStrike: External Technical Root Cause Analysis — Channel File 291, 6 August 2024 (PDF). The 21-versus-20 mismatch, wildcard testing, crash dump, findings and mitigations, the 99%-online figure.
• The Go and JavaScript programs are our own illustration, run for this page as shown.

Side by side

How each one started and why it spread:

How recovery went and what changed:

The patterns

1. A tiny trigger, several missing nets. None of these started with anything dramatic: a typo, a regex, a missed copy, a maintenance command. Each postmortem lists several causes that had to line up. That's good news: you don't need to prevent every trigger, you need enough nets that one trigger can't get through all of them.

2. The fast path skipped the safety steps. Cloudflare's software went through staged releases, but its WAF rules didn't. CrowdStrike's sensor releases were staged, but its Rapid Response Content wasn't. Knight's orders carried the reused flag to every server, whatever code each one ran. Config, rules, flags and content change behavior just like code, often faster, and they were the path with the least protection.

3. The tool trusted its input. AWS's tool removed whatever it was told to. GitLab's engineer had nothing on screen saying "this is the primary". Facebook had a tool to catch dangerous commands, but it had a bug. A good tool has a guardrail for the invariants that matter, and the guardrail itself gets tested.

4. Failures nobody could see. GitLab's nightly pg_dump failed every time it ran, and the alert emails were rejected. Knight got 97 emails mentioning "Power Peg disabled" before the market opened, but they weren't designed as alerts. Cloudflare's tests passed because they didn't measure CPU. A check that can fail silently is not a check.

5. The rare path had never been run. AWS hadn't fully restarted those subsystems in years. GitLab had never test-restored its pg_dump backups. Knight never retested Power Peg after changing it in 2005. Facebook had drilled many failures, but never the whole backbone. Whatever you only do in an emergency, you'll do slowly and badly, unless you rehearse it.

6. The fix needed the thing that was broken. Cloudflare's login, AWS's status dashboard and Facebook's internal tools all depended on the system that was down. §09 is about this trap.

How big did each one get?

Blast radius is how much a single failure can affect. Here's each incident placed on two questions: did it hit one system or region, or the whole world? And did it break other companies' products for their users (because those companies build on it), or only the company's own product?

The bottom-right corner is where infrastructure providers live: Cloudflare sits in front of its customers' websites and CrowdStrike runs inside its customers' computers, so one bad change reaches many companies at once. If you run a platform other teams build on, your blast radius is theirs too, and the bar for staged rollouts is higher.

Rule 1: Every change has a small first step

The worst outages here reached everything at once. The fixes all add a small first step:

• Staged rollouts for every kind of change, not just code. Cloudflare committed to staged rollouts for WAF rules, and CrowdStrike to canary rings for Rapid Response Content. Feature flags, DNS records, load balancer rules and permissions deserve the same. Keep an emergency fast path, but make it a deliberate choice with a name, not the default.
• Partition the system so one failure stays local. AWS's cells limit how much one restart or failure affects; regions and availability zones do the same at a larger scale.
• Bound every loop that acts on the world. Knight's router had no limit that stopped it automatically. A cap on orders, emails, deletions or retries per minute turns a runaway into a small, noisy event. (Rate Limiting covers the mechanics.)

Rule 2: Put the safety check in the tool

Training and checklists help, but people make typos, especially under pressure. The durable fixes in these postmortems are checks that refuse: AWS's minimum-capacity guardrail, CrowdStrike's compile-time field count and runtime bounds check, a deploy gate that verifies every server's version. Two refinements:

• Test the guardrail. Facebook had an audit tool; it had a bug. A safety check that's never been seen to block anything may not work.
• Make the dangerous context obvious. GitLab chose to make it "more obvious as to what host you're using" (a distinct shell prompt per environment) rather than ban commands. Color, prompts and confirmations that name the target ("type the hostname to continue") catch the wrong-terminal mistake.

Rule 3: Recovery must not depend on what's broken

Three of six incidents hit this recovery dependency trap. Ways out, each taken from a fix one of these companies made or recommended:

A useful exercise: list the five tools you'd reach for in an incident and, for each, ask "does this work if our DNS, our login, our main region or our network is down?"

Rule 4: Rehearse the rare path

Every incident here involved something done rarely: a full restart (AWS), a restore (GitLab), old code paths (Knight), a backbone-wide failure (Facebook), a field nobody had used (CrowdStrike). Rehearsal turns these into routine:

• Restore tests on a schedule (the script in §04).
• Game days and drills, like Facebook's storm exercises, that take down a service, a data center or a region on purpose. See Reliability & SRE §12 Chaos engineering.
• Test with realistic data, not just the easy case: CrowdStrike's tests used wildcards where production would one day use real values; Cloudflare's tests checked results, not CPU.

Rule 5: Make every alarm a real alarm

GitLab's backup failures went to email that bounced; Knight's 97 warnings weren't designed as alerts. If a signal means "something is wrong", route it to the same place as your other pages, and alert on absence too: "no successful backup in 26 hours" catches a job that silently stopped running. Observability §10 covers alert design.

Rule 6: Have an off switch, and make it reachable

Cloudflare's "global terminate" for the WAF is the success story here: once found, one switch restored traffic within minutes. Knight had no such brake. For every component that can hurt everyone (a rule engine, a router, a background job), ask: what's the Kill switch, who can flip it, and does flipping it depend on the broken system?

Work out your answer before you click. Every question is covered above, and each explanation names the section.

Reading a postmortem: summary → timeline (note the time zone and the gaps) → impact (who measured it?) → contributing factors (not one villain) → fixes (does each map to a cause?).

Before you push a change, ask: what's the smallest first step? How will I know it's bad? How do I turn it off, and does that depend on the thing I'm changing?