CAP & PACELC

The 30-second version

The first half of that sentence is the CAP theoremCAP theoremA proven result (Gilbert & Lynch, 2002, from a 2000 talk by Eric Brewer): when the network between replicas splits, a system can't both answer every request and keep every answer up to date. It must give up one of the two until the network heals.. The second half is PACELCPACELCAn extension of CAP (Daniel Abadi, 2010): if there's a Partition, choose Availability or Consistency; Else, in normal operation, choose lower Latency or Consistency., and it's the half you'll feel every day.

Try it. The simulator below runs exactly this story. CP mode refuses when unsure; AP mode always answers.

What you just saw

• In CP mode, Branch B refused everything while the line was cut, even a harmless balance check. Branch A carried on. When the line came back there was nothing to fix: only one branch had changed the account.
• In AP mode, both branches stayed open. Two $60 withdrawals were each fine on their own copy, but together they broke the rule “balance never goes below $0”. Repairing the line can't put the cash back.
• Deposits merged fine in AP mode. +$50 here and +$50 there add up to the same total in any order. Some kinds of data survive “always answer” perfectly well; §09 is about telling them apart.

The part people forget

Go back to CP mode with the line up and press a few buttons at Branch B. Every transaction there needs a phone call to A before B can answer: watch the ☎ B→A→B +70 ms badge and the “customers kept waiting” counter grow. Nothing is broken, and B is still slower than it would be on its own. In AP mode the same buttons answer at once (+0 ms) and phone the other branch afterwards. Between real data centers that “phone call” is a network round trip of anywhere from 1 ms to over 200 ms, paid on every request that needs an up-to-date answer.

That is the everyday trade-off: latency versus consistency. Partitions are rare; round trips happen on every request.

Where this topic goes

• §02–§03: the minimum distributed-systems background, then what CAP actually says (and the popular “pick 2 of 3” version gets wrong).
• §04–§05: what “consistent” means precisely, and the ladder of weaker promises below it.
• §06–§08: PACELC, real systems, and choosing consistency per request in Go and JavaScript code.
• §09–§11: designing with it, what partitions look like in production, and pitfalls.

Distributed systems in five minutes

The words

• A nodeNodeOne machine (or one process) taking part in a distributed system — a database replica, a broker, a server in a cluster. is one machine (or one running process) in the system.
• A distributed systemDistributed systemA program that runs on several machines that cooperate over a network. Any machine or link can fail or slow down independently, which is what makes them hard. is several nodes working together over a network to look like one service.
• A replicaReplicaA full copy of a database kept on another machine and updated as the original changes, for reading load and for taking over if the original dies. is one copy of the same data, kept on a different node. You keep replicas so that losing one machine doesn't lose the data, and so that users far away can read from a copy near them.

In a group chat, every phone has a copy of the conversation: each phone is a replica of the chat history. When you send a message, it goes out over the network and the other phones update their copies when it arrives. Until then, their copies are behind yours.

Nodes only talk through messages

A node can't look inside another node's memory. To find out anything (“what's the balance?”, “are you alive?”), it sends a message and waits for a reply. Three things can happen to that message:

1. It arrives quickly. (The normal case, almost always.)
2. It arrives late: the network is congested, or the other node is busy.
3. It never arrives: a cable, switch or firewall dropped it, or the other node crashed.

You can't tell slow from dead

Here is the fact that everything in this topic grows from. Below are three very different failures. Look at them from the client's side only:

All the client knows is “I asked and nobody answered”. So it uses a timeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever.: “if I hear nothing in 2 seconds, I'll treat the other side as unreachable.” But a timeout is a guess. The other node might be fine and about to reply. It might even have done the work already.

A network partition

A network partitionNetwork partitionWhen the network splits so that some nodes can't reach others, even though all of them are still running. Each side may think the other side is dead. is when the network splits the nodes into groups that can't reach each other, even though every node is still running. Each group can still talk among itself and to some users. Each group may think the other group has died.

Partitions don't need a dramatic cable cut. A misconfigured firewall rule, a failing network switch, or a node so overloaded that it answers nothing in time all look the same from the outside. (§10 shows the real-life causes.)

CAP, stated properly

Eric Brewer proposed this in a talk in 2000; Seth Gilbert and Nancy Lynch proved it in 2002. The name comes from three properties. Each one means something narrower in the theorem than the same word means in everyday engineering talk, so read these carefully.

The three letters

C — Consistency. The system behaves as if there were one single copy of the data. Once a write is confirmed, every read anywhere returns it (or something newer). The precise name is linearizabilityLinearizabilityThe strongest consistency guarantee: the system behaves as if there were one copy and every operation happened at a single instant. Once any read sees a new value, no later read sees the old one., and §04 is all about it. Note: this isn't the C in ACIDACIDThe four guarantees of a database transaction: Atomic (all or nothing), Consistent (rules hold), Isolated (concurrent transactions don't see each other's half-done work), Durable (once committed, it survives a crash)., which is about a database's rules being kept inside a transaction. Same letter, different idea.

A — Availability. Every request that reaches a node that hasn't crashed gets a real answer, not an error. “Sorry, try later” doesn't count. This is much stricter than the everyday meaning of availabilityAvailabilityThe fraction of time (or requests) a system answers successfully — 99.9% means about 43 minutes of failure a month. (“up 99.9% of the time”): it says every working node must answer every request.

P — Partition tolerance. The system keeps following its rules even when the network loses messages between nodes. In other words: partitionsNetwork partitionWhen the network splits so that some nodes can't reach others, even though all of them are still running. Each side may think the other side is dead. can happen, and the system has a plan for them.

The proof, in four lines

Take the simplest possible system: two replicas, A and B, each holding x = 0.

1. The network between A and B splits. No messages get through.
2. A client writes x = 1 to A. If A wants to be available, it must accept and confirm the write. It can't tell B.
3. Another client reads x from B. If B wants to be available, it must answer. All it has is x = 0.
4. The read returned 0 after a confirmed write of 1. That's not consistent. The only way to stay consistent was for A or B to refuse, which isn't available.

That's the whole proof. Nothing clever is required, which is a good sign that the result is fundamental: it comes straight from §02's facts.

Why “pick 2 of 3” is misleading

CAP is often drawn as a triangle with the slogan “pick any two”. That suggests three equal options: CA, CP and AP. It's wrong in three ways.

• P isn't optional. Between machines connected by a network, partitions will happen. “Not P” would mean “we've decided the network never fails”, which isn't a decision you get to make. So the real choice is only between C and A.
• The choice only exists during a partition. On a normal day, when messages get through, a system can be perfectly consistent and answer every request. CAP says nothing at all about that case. (§06 shows that the normal case has a different trade-off.)
• It's not one switch for the whole system. A system can make different choices for different data or different operations: refuse withdrawals but keep answering balance checks with a “might be a minute old” label. Real systems also sit on a scale between the two extremes rather than at one end.

A better statement, close to how Brewer himself restated it in 2012: when a partition happens, each operation on each piece of data must either wait or risk being wrong.

What CAP doesn't say

• It doesn't say anything about speed. A system that takes 30 seconds to answer still counts as “available” in the theorem. Users wouldn't agree. (§10 comes back to this.)
• It doesn't say a CP system is often down. It's only unavailable on the side of a split that can't confirm, and only during the split.
• It doesn't say an AP system is always wrong. Most of the time its copies agree; it just can't promise that.
• It doesn't promise you'll get even one of the two. Plenty of real systems are neither: a database with a slow async replica that also loses data on failover gives up C and can still be unavailable. CAP is an upper limit, not a guarantee.

What “consistent” means: linearizability

“Every read returns the latest write” sounds simple until you ask: latest according to whom? On different machines, operations overlap in time, and a request isn't instant: it's sent, travels, gets processed, and the answer travels back. Linearizability is the precise answer to “latest”.

Operations take time

To keep things small, consistency is usually explained on a registerRegisterIn consistency theory: one named value you can read and overwrite, like a single variable or a single database row. Consistency models are easiest to explain on one register.: one named value, x, that you can write and read. Think of one row in a database or one key in a cache.

Draw each operation as a bar from the moment the client sent it to the moment it got the answer. Somewhere inside that bar, the system actually did the work, but the client can't see exactly when. That instant is the operation's linearization pointLinearization pointThe single instant, somewhere between when an operation was sent and when its answer arrived, at which it appears to take effect. A history is linearizable if such instants can be chosen for every operation..

The rule for linearizability:

1. You must be able to put one dot inside every bar.
2. If you then read the dots left to right, every read must return the value of the latest write before its dot (or the starting value, if there's none).

If at least one way of placing the dots works, the history is linearizable. If no placement works, it isn't.

Try it

The checker below searches every possible ordering for you. Start with Overlapping write: Bob saw the new value, and Carol, who started after Bob had finished, saw the old one. Click the reads to change their values and watch the dots appear when a valid placement exists.

Three lessons from those histories:

• Overlapping operations may go either way. While Alice's write is in progress, a read may return 0 or 1. Both are fine.
• But the system can't change its mind. Once Bob has seen 1, the write has “happened”. Carol's later read must also see 1. This is the property that makes linearizable systems feel like one copy.
• Finished means finished. In Stale after the write finished, the write was confirmed before the read began. In real time it came first, so a linearizable read must return 1. Distance, replicas and caches are no excuse.
• Concurrent writes may land in either order, but once chosen it's fixed. In Two writers, then reads, Alice's and Bob's writes overlap, so the final value may be 1 or 2. But Carol reads after both have finished, so her two reads must agree: 1 then 1, or 2 then 2. Reading 1 and then 2, with no write in between, is not allowed.

Why it's expensive

To promise “no read will ever see the old value after anyone saw the new one”, a replica can't just answer from its own copy. It has to make sure it isn't behind. In practice that means one of:

• Sending every read and write through a single leaderLeader / primaryThe one copy of a database that accepts writes. It records every change in a log and sends that log to the followers. Also called primary or master., and having that leader check with a majority (a quorumQuorumA majority (or another agreed minimum) of replicas that must answer before a read or write counts — e.g. 2 of 3. Any two majorities overlap, so a read quorum always includes at least one replica that saw the latest write quorum.) that it's still the leader before answering, because a leader cut off by a partition might have been replaced without knowing it.
• Running a consensusConsensusGetting several nodes to agree on one value (or one ordered log of values) even if some of them crash or messages are lost. Raft and Paxos are consensus algorithms. protocol such as RaftRaftA popular consensus algorithm designed to be understandable: nodes elect a leader, the leader appends commands to a log, and an entry counts once a majority has stored it. Used by etcd, Consul, CockroachDB. so that a majority agrees on the order of every write. The Consensus & Raft topic covers how.

Either way, each operation waits for at least one round trip to other nodes. During a partition, the side without a majority can't do that, so it must refuse. Linearizability is the C in CAP.

The consistency ladder

A consistency modelConsistency modelThe promise a data system makes about what readers can see after writes — from “always the latest value” (linearizable) down to “eventually the same everywhere”. is the contract between a data store and your code: “here's what reads can return after writes”. The stronger the contract, the less your code has to worry about. The weaker it is, the faster and more available the store can be. Here are the rungs you'll meet, strongest first:

Each rung, by the surprise it allows

The checker below has one history per rung boundary. For each, it tells you which models allow it. Go through them in order; each story is something real users have reported as a bug.

Stale after the write finished (linearizable ✗, sequential ✓). Sequential consistencySequential consistencyEvery client sees all operations in one shared order that respects each client's own order. Unlike linearizability, it ignores real time between different clients, so a read can be “behind” a write that already finished elsewhere. promises that everyone sees the same order of operations and that each client's own operations stay in the order it did them. It does not promise that the order matches the clock on the wall between different clients. So Bob may be “behind” Alice: the store can pretend his read happened before her write. ZooKeeper's ordinary reads behave roughly like this: every server applies the writes in the same order, but the server you read from may be a little behind.

Two writes, two orders (sequential ✗, causal ✓). Alice and Bob write at the same moment, independently. Carol sees 1 then 2; Dan sees 2 then 1. There's no single order everyone agrees on, so this isn't sequential. But neither write depended on the other, so causal consistencyCausal consistencyA guarantee that if one write caused another (you read A, then wrote B because of it), everyone sees A before B. Unrelated writes may appear in any order. allows it. Think of two unrelated comments on a post showing up in different orders for different readers. Usually harmless.

Reply before the post (causal ✗, session ✓). Bob read Alice's post and then replied: his reply was caused by the post. This cause-and-effect relationship is called causalityCausality (happened-before)Event A could have influenced event B — e.g. B replies to A's message. Distributed systems try to preserve this order even when clocks disagree.. Causal consistency promises that anyone who sees an effect also sees its cause. Carol sees the reply but not the post, an answer to a question she can't see. Read-your-writes and monotonic reads (below) don't forbid this: they only constrain what Carol sees of her own writes and reads, and Carol never saw the post herself. (A third session guarantee, writes-follow-readsWrites-follow-readsA session guarantee: if you read X and then write Y, every replica applies Y only after X. It's what stops a reply from showing up before the post it answers., would forbid it; all four session guarantees together add up to causal.)

My write vanished (session ✗, eventual ✓). Alice saves, reloads, and her change is gone. Session guaranteesSession guaranteesPromises made to one client (one “session”) rather than to everyone: read-your-writes, monotonic reads, monotonic writes and writes-follow-reads. Cheap to provide on top of replicas. All four together add up to causal consistency. are promises to each client about its own view: read-your-writesRead-your-writesA guarantee that after you change something, your own later reads show the change. Other users may still see the old value for a moment. (you always see your own changes) and monotonic readsMonotonic readsA guarantee that once you've seen some data, later reads never show you an older version of it: time never appears to go backwards for one user. (your view never goes backwards). This history breaks the first. “Session guarantees” is the family name; the ladder and the checker show these two because they're the ones you'll use most. Replication §06 shows how replica lag causes exactly this, and §07 how to fix it in code; this topic won't repeat that.

Eventual allows all of the above. Eventual consistencyEventual consistencyA promise that if writes stop, every copy or derived view of the data (replicas, caches, read models, other services) will end up agreeing — but for a while, different readers may see different (stale) versions. only promises that if writes stop, all copies end up the same. About any individual read, it promises nothing.

What each rung costs

The big break is between sequential and causal. Causal is roughly the strongest model that can stay available on both sides of a partition; everything above it needs coordination that a partition blocks. (The precise result, by Mahajan, Alvisi and Dahlin in 2011, is about a slightly stronger variant called real-time causal consistency.) That's why research and newer databases put so much effort into causal consistency: it's as much as you can have without giving up availability.

PACELC: the trade-off you pay every day

CAP only talks about the rare moment when the network is split. Daniel Abadi pointed out in 2010 that this misses the trade-off engineers actually argue about. Even with a perfect network, a replica can only promise an up-to-date answer by checking with other replicas first, and checking takes a round trip. That's PACELCPACELCAn extension of CAP (Daniel Abadi, 2010): if there's a Partition, choose Availability or Consistency; Else, in normal operation, choose lower Latency or Consistency., read as “if P then A or C, else L or C”.

Worked example: one write, three choices

Your database has three replicas in three cloud regionsCloud regionA geographic area where a cloud provider runs its own set of data centers, such as US East (Virginia) or Europe (Ireland). Round trips inside a region take about 1 ms; between continents, 70–250 ms.: the leader in US East (Virginia), followers in Europe (Ireland) and Asia (Singapore). Your app server sits next to the leader. Rough round-trip timesRound-trip time (RTT)How long it takes for a message to go from one machine to another and for the reply to come back. Roughly 1 ms inside a data center, 70–150 ms across an ocean. between regions:

The leader writes to its own disk in about 1 ms. Then it can reply at three different moments:

Now a request that saves five things one after another: 5 ms, 355 ms, or over a second. Nothing is broken. The network is perfect. You're paying for consistency with time.

Reads have the same choice

A user in Europe can read from the replica in Ireland (~1 ms, but maybe a little behind), or from the leader in Virginia (~70 ms, always current). It's the same trade-off again, per read.

Try the combinations yourself. Things to try:

1. User in Europe, writes leader only, reads nearest: reads take ~1 ms and may be stale; writes still pay the round trip to the leader in Virginia (~71 ms).
2. Same user, reads from the leader: reads are fresh, and pay the trip to Virginia.
3. Writes leader + 1 follower, reads nearest 2. Here W = 2 (copies that confirm each write) and R = 2 (copies each read asks). R + W = 4 is more than the 3 copies, so every read asks at least one copy that has the latest write (Replication §10 explains this quorum rule). Reads see the latest acknowledged write (with caveats, §11) without having to go specifically to the leader. For a Europe user the nearest two include Virginia anyway; switch the user to Asia and the nearest two are Singapore and Ireland, so a fresh read waits ~165 ms instead of the 215 ms trip to Virginia.
4. Move the leader to Asia and watch every write from Europe get slower.

How systems get labeled

A PACELC label has two halves: what the system does during a partition, and what it does otherwise.

• PA/EL: answers during partitions, and doesn't wait for other replicas normally. Fast and always up, eventually consistent. (Dynamo-style stores at their defaults.)
• PC/EC: refuses during partitions, and waits for other replicas normally. Slower, but behaves like one copy. (Consensus-based systems.)
• PC/EL and PA/EC exist too, and are usually a sign of configuration choices; §07 has examples.

Most real systems let you choose per request, which is why §08 is about code.

Real systems, honestly classified

Here's where common systems land with typical settings. Read the caveats column; it matters more than the label.

Abadi's 2010 blog post and 2012 paper put Dynamo, Cassandra and Riak in PA/EL; BigTable, HBase and fully ACID distributed databases in PC/EC; MongoDB (with the defaults of that time) in PA/EC; and Yahoo's PNUTS in PC/EL. Several of those defaults have changed since, which is exactly the point: the label belongs to a configuration, not a product.

LOCAL_QUORUM across two data centers

LOCAL_QUORUM means “a majority of the replicas in my data center”. That's great for latency, but a write in one data center and a read in another can miss each other completely:

So LOCAL_QUORUM everywhere is PC/EC within a data center and PA/EL between them: during a split between data centers both sides keep serving (PA), and on a normal day nobody waits for the other ocean (EL). If a read in DC 2 must see a write made in DC 1, use QUORUM (or EACH_QUORUM for writes) and pay the cross-ocean round trip. One more trap: some drivers can be configured with a downgrading retryDowngrading retryA client setting that, when a strong request (like QUORUM) fails, silently retries it at a weaker consistency level (like ONE). It keeps answering, but with a weaker guarantee than the code asked for. that silently turns a failed QUORUM into ONE (one of the Cassandra cards in the classifier below).

Now you try. For each setup, decide what it does during a partition (A or C) and on a normal day (L or C):

Three questions to ask about any system

1. Which operations? Writes, reads, and transactions often have different behavior. “Reads are PA, writes are PC” is a common honest answer.
2. Which settings? Consistency levels, write/read concerns, synchronous_commit, driver retry policies. Look at what your code sends, not the default in a blog post.
3. Has anyone tested it? Claims like “strongly consistent” have often failed independent testing with injected partitions (§10). If there's a public JepsenJepsenA testing tool and series of reports by Kyle Kingsbury that run real databases, inject partitions, clock jumps and crashes, then check the recorded history against the consistency the database claims. Many popular databases have failed it. report for the version you run, read it.

Choosing consistency per request, in code

Most data stores expose the PACELC dial directly. The general name is tunable consistencyTunable consistencyChoosing per query how many replicas must answer (ONE, QUORUM, ALL) — trading speed and availability against reading the latest data.. Here's the same idea in four systems, each with the cost of turning it up.

Postgres: how many copies before “committed”?

With a synchronous standby configured (Replication §05 sets one up), Postgres's synchronous_commit setting decides what COMMIT waits for. The useful part: you can change it for one transaction.

// write saves one row. level decides how many copies must have it before COMMIT returns.
func write(ctx context.Context, db *pgxpool.Pool, level, note string) error {
	return pgx.BeginFunc(ctx, db, func(tx pgx.Tx) error {
		// true = only for this transaction (same as SET LOCAL)
		_, err := tx.Exec(ctx, "SELECT set_config('synchronous_commit', $1, true)", level)
		if err != nil {
			return err
		}
		_, err = tx.Exec(ctx, "INSERT INTO events(note) VALUES ($1)", note)
		return err
	})
}

// ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // ← give up after 2 s
// defer cancel()
// write(ctx, db, "local", "page_view")  ← own disk only: fast
// write(ctx, db, "on", "payment 42")    ← waits for the standby too

Measured on the test machine, with primary and standby on the same computer (so the “network” costs almost nothing), 200 commits each:

Even with zero distance, waiting for a second copy more than doubled commit time. Across regions, add a full round trip (~70 ms between US East and Europe) to every on commit.

Then we stopped the standby. local commits carried on as if nothing had happened. on commits hung until the Go context deadline of 2 s, and failed with timeout: context deadline exceeded. That's CAP in one setting: local chose availability, on chose consistency. (And careful: the timed-out row was still committed on the primary. Replication §05 explains why that's dangerous.)

Redis: WAIT for replicas

RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages). replicates asynchronously. The WAIT numreplicas timeout command blocks until that many replicas have received this connection's earlier writes, or the timeout passes, and returns how many did.

conn := rdb.Conn() // one dedicated connection: WAIT only counts writes made on it
defer conn.Close()

if err := conn.Set(ctx, "balance", 30, 0).Err(); err != nil {
	return err
}
acked, err := conn.Wait(ctx, 1, 100*time.Millisecond).Result() // ← 1 replica, at most 100 ms
if err != nil {
	return err
}
if acked < 1 {
	log.Println("only on the primary so far") // the SET is NOT undone
}

Run against a local primary and replica: with the replica up, WAIT returned 1 in well under a millisecond. With the replica stopped, it waited the full 100 ms and returned 0, and the value stayed written on the primary. WAIT tells you how far a write got; it doesn't make Redis strongly consistent, and the Redis docs say so. A failover can still lose a write that WAIT reported as copied, if the newly promoted replica wasn't one of those that got it.

Cassandra: a consistency level on every query

In CassandraCassandraA distributed database built for huge write volumes spread across many machines. Data is grouped by a partition key and sorted within each partition, which suits “latest messages in this chat” queries. Internally it stores data in LSM trees. each query carries its own consistency level, deciding how many of the N replicas must answer. These are the N, W, R numbers from Replication §10.

// feed: any one replica will do. Fast, maybe a little stale.
func feed(ctx context.Context, session *gocql.Session, uid string) ([]string, error) {
	iter := session.Query(`SELECT post_id FROM feed WHERE user_id = ?`, uid).
		Consistency(gocql.One).WithContext(ctx).Iter()
	var ids []string
	var id string
	for iter.Scan(&id) { // ← one row per loop; false when rows run out or on error
		ids = append(ids, id)
	}
	if err := iter.Close(); err != nil { // ← always Close: it frees the iterator and reports errors
		return nil, err
	}
	return ids, nil
}

// balance: wait for 2 of the 3 replicas in this data center.
func balance(ctx context.Context, session *gocql.Session, acct string) (int64, error) {
	var cents int64
	err := session.Query(`SELECT cents FROM balances WHERE account = ?`, acct).
		Consistency(gocql.LocalQuorum).WithContext(ctx).Scan(&cents)
	if err != nil {
		return 0, err // ← CP: an error instead of a guess
	}
	return cents, nil
}

The LOCAL_QUORUMLOCAL_QUORUMA Cassandra consistency level: a majority of the replicas in the coordinator's own data center (2 of 3). It says nothing about other data centers, so a write and a read in two different data centers may not overlap. read only guarantees fresh data if writes to balances also use LOCAL_QUORUM (2 + 2 > 3) in the same data center. Across data centers the two quorums don't overlap (the diagram in §07); use QUORUM/EACH_QUORUM there. A read at QUORUM after a write at ONE can miss it. For “only if nobody else did first” logic, such as claiming a username, plain consistency levels aren't enough: use a lightweight transactionLightweight transaction (LWT)Cassandra's compare-and-set: INSERT … IF NOT EXISTS or UPDATE … IF col = x. Correct even under concurrency, but several times slower, and limited to one partition. (INSERT … IF NOT EXISTS), which runs PaxosPaxosThe classic consensus algorithm (Leslie Lamport, 1989). Proven correct but famously hard to understand and implement; Raft was designed as an easier alternative. and is linearizable.

DynamoDB: one flag per read

DynamoDBDynamoDBAmazon's managed key-value and document database. Every item has a partition key (and optionally a sort key); you query by those keys, and AWS spreads the data across machines for you. reads are eventually consistent by default. Set ConsistentRead and the read reflects every write that succeeded before it.

out, err := db.GetItem(ctx, &dynamodb.GetItemInput{
	TableName:      aws.String("accounts"),
	Key:            map[string]types.AttributeValue{"id": &types.AttributeValueMemberS{Value: id}},
	ConsistentRead: aws.Bool(true), // ← strong read: costs 2× an eventual one
})

The cost is spelled out in the AWS docs, and it's PACELC on a price list: a strongly consistent read uses twice the read capacity of an eventually consistent one, can have higher latency, isn't supported on global secondary indexes, and “might not be available if there is a network delay or outage”, in which case DynamoDB may return a server error (HTTP 500). Strong reads give up availability; that's the C-or-A choice, in the docs.

(The DynamoDB and Cassandra snippets were compiled against the real Go modules and loaded in Node 22, but not run against a live cluster. The Postgres and Redis ones were run as shown.)

The dial, side by side

Designing around it: which data needs what

Back in §01, two kinds of operation behaved very differently in AP mode. Deposits merged cleanly. Withdrawals broke the rule “balance never below $0”. That difference is the key to designing with CAP.

Rules that need one decision-maker

An invariantInvariantA rule that must always hold true about your data — “an account balance is never negative”, “area equals width × height”. Code (and transactions) must keep it true before and after every change. is a rule that must always be true, like “balance ≥ 0” or “each username belongs to one person”. Some invariants can only be checked by looking at all recent writes at once:

• Balances and stock that can't go below zero. Two replicas that each see “1 seat left” will each sell it.
• Uniqueness. Usernames, email addresses, order numbers. Two replicas that each see “alice is free” will give it to two people.
• Locks and leadership. “Only one worker processes this job”, “only one node is the leader”. Two sides of a partition that each grant the lock have no lock at all. (Use a coordination serviceCoordination serviceA small, highly available key-value store (etcd, Consul, ZooKeeper) that several machines use to agree on facts like “who is the leader right now”. It uses consensus internally, so its answers can't contradict each other., with a fencing tokenFencing tokenA number that increases every time a new leader is chosen or a lock/lease is granted. Storage remembers the highest number it has seen and rejects writes carrying an older one, so a paused old holder can't overwrite the new holder's work. on everything the lock-holder touches; see Consensus & Raft.)

These operations need linearizability: a leader, a consensus system, or a lightweight transaction. During a partition, they should refuse on the side that can't confirm. An error message is cheaper than an oversold flight.

Data that's fine eventually

• Counters people glance at: likes, view counts, follower counts. Off by a few for a few seconds, nobody notices.
• Feeds, timelines, recommendations, search results. They were never exact anyway.
• Presence and activity: “online”, “last seen 5 minutes ago”, typing indicators.
• Your own analytics and logs, as long as they converge.

These can be AP: served from the nearest replica, written locally, merged later.

Make operations merge on their own

The deposit trick generalizes. Operations whose order doesn't matter are called commutativeCommutative operationsOperations whose result doesn't depend on order: +50 then +20 gives the same total as +20 then +50. Commutative updates can be applied on different replicas and merged later without conflict.: +50 then +20 equals +20 then +50. If every update is commutative, replicas can accept them independently and merge later without conflicts.

• Store “+1 like from replica 2” instead of “likes = 18”. Each replica counts its own increments; the total is the sum. This is the simplest CRDTCRDTConflict-free replicated data type: a data structure (counter, set, text…) designed so concurrent changes can always be merged automatically, in any order, to the same result., and Replication §09 introduces the others (sets, text).
• Store events (“added item”, “removed item”) instead of overwriting state, then compute the state from the events.

Compare with “set balance = 40”: two different overwrites can't both win, so one gets thrown away, usually by last write winsLast write wins (LWW)A conflict rule: attach a timestamp to every write and keep the one with the biggest timestamp. Simple, but it silently discards the other write, and clocks on different machines disagree..

Patterns for the hard cases

Home region. Give each record one owner. Alice's account lives in Europe; all writes to it go there, even from Asia. Writes are linearizable per record, and most users are near their home. This home regionHome regionIn a system running in several geographic regions, the one region that owns a given piece of data (e.g. all of customer 42's keys). Requests for it are routed there so two regions never process the same key at once. pattern is the most common way to run in several regions without conflicts.

Escrow (split the budget). A shop has 100 units of stock in two regions. Give each region 50 to sell on its own, with no coordination. Only when a region runs low does it ask the other for more. Each region can approve sales locally, even during a partition, and the total can never go negative. The idea is old (database researchers described “escrow transactions” in 1986) and shows up today in inventory, quota and budget systems.

Accept, then apologize. Sometimes availability is worth occasionally breaking the rule. Amazon's shopping cart famously accepted every “add to cart” and merged conflicting carts later; an airline may overbook and compensate. If the business can repair a rare mistake (refund, voucher, apology email), you can choose AP and handle the conflict as a business process. Sagas are the tool for undoing work across services with compensatingCompensating actionA step that semantically undoes an earlier step that already committed — refund a payment, release a reserved seat — used when you can't roll back across services. steps.

Mix within one request. A checkout page can read product descriptions and reviews from the nearest replica, and run the payment and stock decrement against the leader. Consistency is chosen per operation, not per app.

Staff-level: partitions in real life

What partitions actually look like

Network partitionsNetwork partitionWhen the network splits so that some nodes can't reach others, even though all of them are still running. Each side may think the other side is dead. do happen inside a single data center, more often than most people assume: a 2014 survey of public incident reports by Peter Bailis and Kyle Kingsbury (“The Network Is Reliable”) collected dozens, from switch failures to misconfigured firewalls. But in day-to-day operation, the partition you'll actually meet is usually one of these:

• A paused process. A stop-the-world pausePause (GC or VM)A moment when a program's runtime freezes all its threads, for example for garbage collection. From outside, a paused process looks exactly like a dead or unreachable one. for garbage collectionGarbage collector (GC)Part of the language runtime that automatically frees memory nobody can reach anymore. Anything still referenced — like a stuck goroutine — is never freed., a virtual machine being moved to another host, or a container being starved of CPU. The node is frozen for seconds; then it wakes up and carries on as if no time had passed.
• An overloaded node. It's alive, but its queues are so long that heartbeatsHeartbeat (application)A small “are you still there?” message your own protocol sends every few seconds. If a few heartbeats go unanswered, the program treats the other side as gone. and requests time out. To everyone else, it's unreachable.
• An asymmetric partition. A can send to B, but B's replies to A are lost. A thinks B is dead; B thinks everything is fine.
• A partial partition. A can reach B and C, but B and C can't reach each other. Each of them gets a different picture of who is alive.
• Gray failureGray failureA component that is neither fully up nor down: slow, dropping some requests, or reachable from some machines but not others. Health checks often say “fine” while users see errors.. The node passes its health checks but fails 30% of real requests. It's “up” on the dashboard and “down” for users.

The paused leader

The paused process is the most dangerous, because the paused node doesn't know it was gone:

A checked “am I still leader?” just before the pause, and the answer was yes. Then it froze. By the time it resumed, B had been elected, and now two nodes act as leader: split brainSplit brainTwo nodes both believe they are the leader and both accept writes, so the data forks into two histories. Usually caused by a failover while the old leader was still alive.. No amount of checking in A's own code can close this gap, because the pause can land between the check and the write. The fix lives on the receiving side: every write carries a fencing tokenFencing tokenA number that increases every time a new leader is chosen or a lock/lease is granted. Storage remembers the highest number it has seen and rejects writes carrying an older one, so a paused old holder can't overwrite the new holder's work. (a number that goes up with each new leader), and storage rejects tokens older than the newest it has seen. Replication §08 shows fencing; Consensus covers leases.

Test it: inject faults, check the history

Consistency bugs only appear under faults, so you have to cause the faults. The approach popularized by JepsenJepsenA testing tool and series of reports by Kyle Kingsbury that run real databases, inject partitions, clock jumps and crashes, then check the recorded history against the consistency the database claims. Many popular databases have failed it.:

1. Run several clients doing reads and writes, and record each operation's start time, end time and result (exactly the bars in §04).
2. Meanwhile, inject faults: cut the network between nodes, pause processes, skew clocks, kill and restart nodes.
3. Afterwards, check the recorded history against the model the system claims: “is there a valid linearization?”

Tools for each step: iptables rules or tc netem on Linux to drop or delay packets, Toxiproxy as a fault-injecting proxy between your app and its dependencies, and checkers such as Porcupine (a Go library) or Knossos and Elle (Jepsen's own). Even without a full Jepsen setup, a test that freezes your leader process with kill -STOP (exactly like a long GC pause) for 30 seconds and then resumes it with kill -CONT will teach you a lot about your failover.

Multi-region architectures through PACELC

A useful trick for the consensus option: place replicas so that a majority is close together. With replicas in Virginia, Ohio and Ireland, a majority (Virginia + Ohio) can commit in ~10–15 ms, while Ireland still holds a copy if a whole US region fails.

Harvest and yield: “available” isn't all or nothing

CAP's A is binary: answer or don't. Real services can degrade. Armando Fox and Eric Brewer named the two dials harvest and yieldHarvest and yieldTwo ways to measure a degraded answer (Fox & Brewer, 1999). Yield: the share of requests that get an answer. Harvest: how complete each answer is. Answering from 9 of 10 shards keeps yield, lowers harvest.:

• Yield: the share of requests that get an answer.
• Harvest: how complete each answer is.

A search engine with 10 shards and one unreachable can fail every query (yield drops) or answer from 9 shards (harvest drops to 90%, yield stays 100%). For search, feeds and dashboards, a 90% answer beats an error. For a bank balance, it doesn't.

Latency is the real availability

CAP counts an answer that arrives after 30 seconds as “available”. Users don't: after a couple of seconds they've given up, and your load balancer has probably retried elsewhere. In practice:

• Your timeout is your partition detector. Every timeout in your code is the line where “slow” becomes “treat it as gone”. Set it too short and every tail-latencyTail latencyThe latency of the slowest few requests — usually reported as p99 or p99.9. It's what your unluckiest users feel, and it grows much faster than the average under load. spike looks like a partition and triggers failovers; too long and users wait for a node that isn't coming back.
• Most incidents are slowdowns, not clean splits. A dependency that gets 10× slower causes queues, timeouts and retries everywhere. That's a latency problem, and PACELC's “E” side, not the P side, is where you'll spend your design effort.
• Retries after a timeout can double-apply writes. The first attempt may have succeeded (§02's “reply lost” case). Make writes idempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly.; the Idempotency topic shows how.

Pitfalls

Check yourself

Predict the answer before you click. Each explanation points back to the section that covers it.

Cheat sheet