Consensus & Raft

The 30-second version

Think of five friends in a group chat deciding where to eat. Phones die, messages arrive late, and someone is in a tunnel. The group still needs one answer that everybody ends up agreeing on, not two halves of the group showing up at two different restaurants.

Computers have exactly this problem. A Distributed systemDistributed systemA program that runs on several machines that cooperate over a network. Any machine or link can fail or slow down independently, which is what makes them hard. runs on several machines (each one is called a NodeNodeOne machine (or one process) taking part in a distributed system — a database replica, a broker, a server in a cluster., or here a server) that talk over a network, and any of them can fail at any moment. ConsensusConsensusGetting several nodes to agree on one value (or one ordered log of values) even if some of them crash or messages are lost. Raft and Paxos are consensus algorithms. is the name for getting them to agree anyway.

The rule that makes it work: a majority

The trick is simple to state: a decision counts once a majority has accepted it. With 5 servers, that's 3. Two different majorities of the same 5 servers always share at least one member, so the group can never make two conflicting decisions. Try crashing servers:

Notice what happens when too many are down: the survivors stop instead of guessing. That's the deal consensus offers: it would rather be unavailable for a while than give two different answers.

Where you meet it

You'll rarely write a consensus algorithm yourself. You'll run systems that use one inside, and depend on them:

• Kubernetes stores all of its cluster state in etcdetcdAn open-source, Raft-based key-value store used for configuration, leader election and locks. Kubernetes keeps all of its cluster state in etcd., a small database that uses the RaftRaftA popular consensus algorithm designed to be understandable: nodes elect a leader, the leader appends commands to a log, and an entry counts once a majority has stored it. Used by etcd, Consul, CockroachDB. algorithm.
• Kafka, CockroachDB, Consul and many others use Raft (or the older PaxosPaxosThe classic consensus algorithm (Leslie Lamport, 1989). Proven correct but famously hard to understand and implement; Raft was designed as an easier alternative.) to choose leaders and keep copies identical.
• Tools like PatroniPatroniA popular open-source tool that runs Postgres clusters with automatic failover. It stores “who is the leader” in a consensus store (etcd, Consul or ZooKeeper)., from the replication topic, ask etcd “who is the leader?” instead of trusting their own timeouts.

What this topic covers

1. Why agreeing is hard at all, and what's provably impossible (§02).
2. The two core ideas: apply the same commands in the same order (§03), and use majorities and numbered terms (§04).
3. Raft, the most popular algorithm, with a live five-server simulator you can crash and split (§05), then its elections (§06) and log copying (§07) up close.
4. What goes wrong with network splits and reads (§08), a short tour of Paxos (§09), real code (§10), and running it in production (§11).

Why agreeing is hard

The group chat, again

Five friends (Ana, Ben, Cleo, Dev, Eli) are deciding where to eat tonight. Here's how it goes wrong in real life, with the computer-science name for each problem:

The last row is the one that hurts. Booking two restaurants is annoying; two database servers both accepting writes means the data forks into two histories (the split-brain story in the replication topic shows exactly this happening with Postgres).

Three simple ideas that don't work

1. “One fixed boss decides.” Fine until the boss's phone dies. Then nobody can decide anything. Picking a new boss means first agreeing the old one is gone, which is the same problem again.
2. “Wait until everyone agrees.” One dead phone and you wait forever.
3. “Whoever answers first wins.” Late messages mean Cleo and Dev saw different “first” answers. Now they disagree about who won.

What we need is a method with two properties, often called safety and livenessSafety vs livenessSafety: nothing bad ever happens (two servers never decide differently). Liveness: something good eventually happens (a decision is reached). Raft never gives up safety; it can lose liveness, for example when no majority is reachable.:

• Safety: never decide two different things. Not even for a moment, not even if messages are delayed for an hour. This one is non-negotiable.
• Liveness: eventually decide something. Ideally quickly, but at least eventually.

The bad news: FLP

In 1985, Fischer, Lynch and Paterson proved a famous result, now called FLP impossibilityFLP impossibilityA 1985 proof (Fischer, Lynch, Paterson): if messages can be delayed for any length of time and even one server may crash, no algorithm can guarantee that the servers always finish agreeing. Real systems use timeouts and randomness to finish in practice.. In plain words:

If messages can be delayed for any amount of time, and even one server might crash, then no algorithm can guarantee that the servers always finish agreeing.— FLP, paraphrased

Why? In an asynchronous networkAsynchronous network (model)A theoretical model where messages always arrive eventually, but there is no upper limit on how long they take, so a slow server looks exactly like a crashed one., where there's no limit on message delay, the algorithm can never tell “Eli crashed” from “Eli's reply is still on its way”. If it waits for Eli, Eli may be dead and it waits forever. If it moves on without Eli, it might move on at exactly the wrong moment, and a cleverly timed sequence of delays can keep it undecided indefinitely.

How real systems cope anyway

FLP doesn't say consensus is useless. It says you can't guarantee finishing in every possible timing. Real systems keep safety always and get liveness in practice:

• Timeouts. “No word from the leader for 300 ms means it's probably dead; start an election.” The guess can be wrong (the leader was just slow), but a wrong guess only costs time: an unnecessary election. It never causes two decisions, because safety comes from majorities (§04), not from the timeout.
• Randomness. If two servers keep starting elections at the same moment, they can block each other forever. Random waits make that astronomically unlikely (§06).
• Assume the network is mostly OK. The model is called partial synchronyPartial synchronyThe realistic model most consensus systems assume: the network can misbehave for a while (arbitrary delays), but eventually it behaves well for long enough to finish an election. Safety holds always; progress only during the good periods.: the network may misbehave for a while, but eventually it's well-behaved long enough to finish. During a bad stretch the system stalls; it never lies.

What it's used for

Consensus is slow-ish (every decision needs network round trips to a majority) and handles small amounts of data. So it's used for the small, critical facts that everything else depends on:

The systems people use for the first four are called coordination servicesCoordination serviceA small, highly available key-value store (etcd, Consul, ZooKeeper) that several machines use to agree on facts like “who is the leader right now”. It uses consensus internally, so its answers can't contradict each other.: etcd, ZooKeeperZooKeeperAn older coordination service from the Hadoop world (Apache), with its own Raft-like protocol called Zab. Long used by Kafka (before KRaft), HBase and many others for leader election and configuration. and Consul. You'll use them through a small API, and §10 shows how.

Same commands, same order, same state

Here's the drift happening. Three copies of a bank balance get the same three commands, but the network delivered them in a different order to each copy:

Switch to “agreed log order” and they all match. That's the replicated state machineReplicated state machineSeveral copies of the same program that start in the same state and apply the same commands in the same order, so they always end in the same state. Consensus is used to agree on that order. idea. “State machine” just means a program whose state changes only by applying commands: a key-value store, a bank ledger, a counter.

The log

Every server keeps a logLog (consensus)An ordered, numbered list of commands (entry 1, 2, 3…). Every server keeps one; consensus makes sure all servers end up with the same entries in the same positions.: a numbered list of commands.

The whole job of consensus is to make sure that every server ends up with exactly the same command at each index. Once all servers agree that entry #3 is x = x + 1, and agree on #1 and #2 before it, each can apply it to its own copy and get the same result.

Two words you'll see constantly from now on:

• An entry is committedCommitted entryA log entry that the leader has confirmed is stored on a majority of servers (following Raft's commit rule). It will be in every future leader's log, so it is safe to apply and to tell the client “OK”. when the group has agreed on it for good: it can never be removed or changed. (When exactly that happens is the clever part; §04 and §07.)
• A server applies an entry to its state machine only once it's committed, and always in index order.

Why a leader?

Who decides the order? Letting every server propose commands at once makes agreement slow and complicated. So most practical systems, Raft included, do what replication does: elect one leader, and let it put commands in order. Followers copy the leader's log.

That sounds exactly like leader/follower replication. The difference is in two places where plain replication guesses and consensus doesn't:

1. Choosing the leader. Consensus guarantees at most one leader per numbered era (a term, §04), even if the old leader is still running somewhere.
2. Deciding when a write is safe. A write is committed only once a majority stores it, so a committed write survives any failover. With asynchronous replication, a failover can lose confirmed writes; with consensus it can't.

Majorities and terms

Idea 1: majorities overlap

A QuorumQuorumA majority (or another agreed minimum) of replicas that must answer before a read or write counts — e.g. 2 of 3. Any two majorities overlap, so a read quorum always includes at least one replica that saw the latest write quorum. here simply means a majority: more than half of the servers. With 5 servers, any 3.

Worked example with 5 servers, S1 to S5:

1. A write is stored on S1, S2, S3: a majority, so it's committed.
2. Later the leader crashes, and S3, S4, S5 vote for a new leader: also a majority.
3. Both groups contain S3. There are only 5 servers, and 3 + 3 = 6, so the two groups must share at least one.

S3 has the committed write, and (as §06 shows) it refuses to vote for any candidate whose log is missing it. So whoever wins the election already has the write. Information flows from one majority to the next through the overlap. Try to make two majorities that don't touch:

How many servers?

That's why consensus clusters almost always have 3 or 5 servers. An even number adds a machine without surviving any more crashes (§11 has the latency side of this).

The same arithmetic handles network splits. If 5 servers split into a group of 3 and a group of 2, only the group of 3 can form a majority. The group of 2 can't commit or elect anyone; it waits. At most one side of any split can make progress, and that's what makes split brain impossible.

Idea 2: terms

Raft divides time into numbered termsTerm (Raft)A numbered period of time in Raft, starting with an election. At most one leader per term. Every message carries the sender's term; a server that sees a higher term adopts it, and a leader that sees one steps down.: 1, 2, 3… Each term starts with an election, and each term has at most one leader. Why at most one? Each server votes at most once per term, and winning needs a majority of votes. Two winners would need two majorities of voters, which would share a server, and that server only had one vote.

Every message carries the sender's term, and three rules follow:

• See a higher term? Adopt it. A server that receives a message with a bigger term number immediately moves to that term. A leader or candidate that sees one steps back down to follower.
• See a lower term? Reject the message. It comes from someone who missed an election, like an old leader that was cut off.
• Terms never go backwards. Each server saves its current term to disk before acting on it.

Majorities and terms are the whole foundation. Raft, in the next section, is mostly the details of using them: how to run elections, and how to copy the log.

Raft, live: the whole protocol in one simulator

RaftRaftA popular consensus algorithm designed to be understandable: nodes elect a leader, the leader appends commands to a log, and an entry counts once a majority has stored it. Used by etcd, Consul, CockroachDB. was published in 2014 by Diego Ongaro and John Ousterhout with one stated goal: be understandable. It splits consensus into three smaller problems: electing a leader, copying the log, and keeping it safe. This section shows all of it running; §06 and §07 then zoom in.

Three roles

Every server is in exactly one role at a time:

• Follower. The normal state. It answers the leader's messages and votes in elections. It never starts anything unless it stops hearing from a leader.
• CandidateCandidateThe middle Raft role: a follower whose election timer ran out. It increases its term, votes for itself and asks the others for votes. It becomes leader with a majority, or goes back to follower if it hears from a leader.. A follower whose election timeoutElection timeoutHow long a Raft follower waits without hearing from a leader before it starts an election. Chosen at random from a range (like 150–300 ms) each time, so servers rarely time out together. ran out without any word from a leader. It starts a new term and asks for votes.
• Leader. Won a majority of votes. It takes all client writes and sends regular heartbeatsHeartbeat (application)A small “are you still there?” message your own protocol sends every few seconds. If a few heartbeats go unanswered, the program treats the other side as gone. so followers know it's alive.

Two messages

Raft needs only two kinds of request (each with a reply):

• RequestVoteRequestVoteThe message a Raft candidate sends to ask for votes. It carries the candidate's term and the index and term of its last log entry, so voters can refuse a candidate whose log is behind theirs.: a candidate asking “vote for me in term 5?”.
• AppendEntriesAppendEntriesThe message a Raft leader sends to followers: new log entries to store, the index and term of the entry just before them (for the consistency check), and how far entries are committed. With no entries it is just a heartbeat.: the leader sending new log entries. With no entries, the same message is the heartbeat.

The life of one write

1. A client sends x=1 to the leader. (A follower that receives it refuses and points the client to the leader.)
2. The leader appends it to its own log at the next index, say #4. It is not committed yet.
3. The leader sends it to every follower in AppendEntries. Each follower writes it to its log and replies “OK”.
4. As soon as the leader has the entry on a majority (itself plus 2 followers, out of 5), the entry is committed. The leader advances its commit indexCommit indexThe highest log position a server knows is committed: stored on a majority under the commit rule, so it can never be lost. Entries up to the commit index are safe to apply to the state machine. to 4.
5. The leader applies x=1 to its state machine and answers the client “OK”.
6. Its next AppendEntries (or heartbeat) carries the new commit index, and the followers apply x=1 too.

Try it

Every server's timeouts and message delays come from a seeded random generator, so the same seed and the same clicks always tell the same story. The narration below the logs is generated from the simulation itself.

Things to try, in order:

1. Press ▶ Play. Watch the rings (the election timers) fill up. The first server whose timer runs out becomes a candidate, collects votes, and becomes leader. The new leader immediately adds an empty no-opNo-op entryAn empty log entry that a new Raft leader appends right after winning. Once it commits, every entry before it is committed too, without waiting for a client write. entry (§07 explains why).
2. Send a few client writes. Watch the violet AppendEntries go out, the cyan replies come back, and each cell turn from dashed (stored) to solid (committed). The state machine column fills in. The row above the logs shows the leader counting copies of each entry (3/5 is a majority), and the box below the logs shows what it remembers about each follower: next, the next entry to send it, and match, the last entry it knows that follower has.
3. Crash the leader. The followers' rings keep filling because no heartbeats arrive; one times out, wins the next term and carries on. No committed write is lost: it was on a majority, and the new leader got votes from a majority.
4. Restart the crashed server with its ↻ restart button (or click the greyed-out server, then ↻). It comes back as a follower, hears a newer term, and gets the entries it missed.
5. Cut the leader off with ✂, then send a write to the old leader (click it to select it first). It stays dashed forever: the old leader can only reach 2 of 5 servers. The majority side elects a new leader. Heal the partition and watch the old leader step down and its uncommitted entry get deleted. §08 covers this case.
6. Crash three servers. Nothing can be committed until one comes back.

What must survive a crash

A restarted server in the simulator remembers some things and forgets others. That split is part of the protocol:

If a server forgot its votedFor in a crash, it could vote twice in the same term, and two candidates could each get a majority: two leaders in one term. That's why every one of these writes is followed by an fsyncfsyncA system call that forces data the OS has buffered in memory to actually be written to the disk. Slow (often milliseconds), but without it a power cut can lose “written” data. before the server replies, and why disk speed matters so much in production (§11).

Elections up close

How an election runs

When a follower's election timerElection timeoutHow long a Raft follower waits without hearing from a leader before it starts an election. Chosen at random from a range (like 150–300 ms) each time, so servers rarely time out together. runs out, it:

1. increases its termTerm (Raft)A numbered period of time in Raft, starting with an election. At most one leader per term. Every message carries the sender's term; a server that sees a higher term adopts it, and a leader that sees one steps down. by one (say from 2 to 3),
2. becomes a candidateCandidateThe middle Raft role: a follower whose election timer ran out. It increases its term, votes for itself and asks the others for votes. It becomes leader with a majority, or goes back to follower if it hears from a leader. and votes for itself,
3. sends RequestVoteRequestVoteThe message a Raft candidate sends to ask for votes. It carries the candidate's term and the index and term of its last log entry, so voters can refuse a candidate whose log is behind theirs. to every other server, and
4. restarts its election timer with a new random value.

Then one of three things happens:

• It wins: votes from a majority (itself included). It becomes leader and sends heartbeats right away so nobody else starts an election.
• Someone else wins: it receives AppendEntriesAppendEntriesThe message a Raft leader sends to followers: new log entries to store, the index and term of the entry just before them (for the consistency check), and how far entries are committed. With no entries it is just a heartbeat. from a leader whose term is at least its own, and goes back to follower.
• Nobody wins: a split voteSplit voteAn election where the votes are spread over several candidates so none gets a majority. Nobody wins that term; the candidates time out and try again in a new term.. Its timer runs out again and it tries again in the next term.

Split votes, and why the timeout is random

The stepper below shows a split vote happening, why it could repeat forever with a fixed timeout, and how a random one ends it. The code panel is the voter's side of RequestVote:

1func (n *Node) onRequestVote(r VoteRequest) bool {
2    if r.Term > n.term { // newer term: adopt it, forget my old vote
3        n.term, n.votedFor, n.role = r.Term, "", Follower
4    }
5    if r.Term < n.term {
6        return false // a candidate from the past
7    }
8    if n.votedFor != "" && n.votedFor != r.Candidate {
9        return false // one vote per term
10    }
11    if !n.logUpToDate(r.LastLogTerm, r.LastLogIndex) {
12        return false // candidate is missing entries I have (§06)
13    }
14    n.votedFor = r.Candidate // saved to disk before replying
15    n.resetElectionTimer()
16    return true
17}

A split vote that repeats forever, with servers constantly busy but no leader ever winning, is called a LivelockLivelockTasks keep running and reacting to each other but never make progress — like two people in a corridor who keep stepping aside in the same direction.. It's exactly the kind of endless indecision FLP (§02) warns about. Random timeouts don't make it impossible, just very unlikely. The Raft paper's rule of thumb:

message round trip ≪ election timeout ≪ time between server failures

With round trips of about 1 ms in one data center, the paper suggests timeouts of 150–300 ms: wide enough that one server usually times out first by more than a round trip and wins before the others wake up.

The voting rules

Look at the code panel again. A server grants its vote only if all of these hold:

1. The candidate's term isn't old (lines 5–7). If it's newer, the voter first adopts it (lines 2–4), which also clears the voter's previous vote, since that was for an older term.
2. One vote per term (lines 8–10). The voter saves votedFor to disk before replying, so even a crash and restart can't make it vote twice.
3. The candidate's log is at least as up to date as the voter's (lines 11–13).

The third rule is what makes Raft safe. “Up to date” compares only the last entry of each log:

• If the last entries have different terms, the one with the later term is more up to date.
• If they have the same term, the longer log is more up to date.

Why this guarantees no committed entry is lost

Put §04's overlap argument together with rule 3:

1. A committed entry is stored on a majority, say S1, S2, S3.
2. To win, a candidate needs votes from a majority. Any majority includes at least one of S1, S2, S3.
3. That server will only vote for a candidate whose log is at least as up to date as its own, so a candidate missing the entry is refused.
4. So every future leader already has every committed entry. Raft calls this the Leader CompletenessLeader CompletenessRaft's guarantee that every leader's log already contains every committed entry. It follows from the vote rule: voters refuse candidates whose logs are behind theirs. property, and it means a leader never needs to fetch missing committed entries from anyone: it only ever sends entries, never receives them.

There's one subtle hole in “stored on a majority means committed”, which the next section closes.

Log replication up close

The consistency check

When the leader sends entries to a follower in AppendEntriesAppendEntriesThe message a Raft leader sends to followers: new log entries to store, the index and term of the entry just before them (for the consistency check), and how far entries are committed. With no entries it is just a heartbeat., it also sends the index and term of the entry just before them: prevLogIndex and prevLogTerm. The follower accepts only if its own log has an entry at prevLogIndex with term prevLogTerm.

That one check gives Raft its Log Matching propertyLog Matching propertyRaft's guarantee that if two servers' logs have an entry with the same index and the same term, the two logs are identical in every entry up to that point.: if two logs have an entry with the same index and term, they're identical up to that point. Why? Think of it as a chain:

1. A leader creates at most one entry per index in its term, and never changes it. So same index + same term means same command.
2. A follower only accepted that entry after checking that the entry before it matched the leader's.
3. That earlier entry was accepted the same way, after its own check, and so on back to entry #1.

How logs end up different

If the leader is healthy, followers' logs are just shorter copies of its log. Differences come from crashes. A leader in termTerm (Raft)A numbered period of time in Raft, starting with an election. At most one leader per term. Every message carries the sender's term; a server that sees a higher term adopts it, and a leader that sees one steps down. 3 appends some entries, copies them to nobody (or to a minority), and crashes. Those entries were never committed. A new leader in a later term never had them and writes different entries at the same positions.

The leader handles both “missing” and “different” the same way. It keeps a nextIndexnextIndex / matchIndexTwo numbers the leader keeps per follower. nextIndex: the next entry to send it (guessed, then lowered on rejection). matchIndex: the highest entry the follower has confirmed it stores. for each follower, starts optimistic, and steps back one entry each time the follower says no:

1func (n *Node) onAppendEntries(a AppendRequest) bool {
2    if a.Term < n.term {
3        return false // a leader from an old term
4    }
5    n.becomeFollower(a.Term, a.Leader) // also restarts my election timer
6    // Consistency check: do I have the entry just before the new ones?
7    if a.PrevLogIndex > len(n.log) || n.termAt(a.PrevLogIndex) != a.PrevLogTerm {
8        return false // no: leader will retry one entry earlier
9    }
10    for i, e := range a.Entries {
11        idx := a.PrevLogIndex + 1 + i
12        if idx <= len(n.log) && n.termAt(idx) != e.Term {
13            n.log = n.log[:idx-1] // conflict: drop this entry and all after it
14        }
15        if idx > len(n.log) {
16            n.log = append(n.log, e)
17        }
18    }
19    if a.LeaderCommit > n.commitIndex {
20        n.commitIndex = min(a.LeaderCommit, a.PrevLogIndex+len(a.Entries))
21    }
22    return true
23}

When is an entry committed?

Recall that a committedCommitted entryA log entry that the leader has confirmed is stored on a majority of servers (following Raft's commit rule). It will be in every future leader's log, so it is safe to apply and to tell the client “OK”. entry is one that can never be lost or changed. The obvious rule is “once it's stored on a majority”. That's almost right. The full rule is:

A leader marks an entry committed when it's stored on a majority and it's from the leader's current term. Committing it also commits every entry before it.

Entries from earlier terms get committed only indirectly, when a current-term entry after them commits. The stepper shows why “on a majority” alone isn't enough. It's the scenario from Figure 8 of the Raft paper, simplified:

1// Leader: runs after a follower confirms entries up to some index.
2for idx := n.lastIndex(); idx > n.commitIndex; idx-- {
3    if n.termAt(idx) != n.term {
4        continue // never commit an older term's entry by counting copies
5    }
6    if n.countServersWith(idx) >= majority {
7        n.commitIndex = idx // this commits idx AND every entry before it
8        break
9    }
10}

The danger was a server whose log ends with a later term but lacks the entry: it can still win an election and overwrite it. Once the leader has an entry from its own term on a majority, no server lacking it can win anymore, because the voters with the newer last term refuse it.

That's also why a newly elected leader immediately appends a no-opNo-op entryAn empty log entry that a new Raft leader appends right after winning. Once it commits, every entry before it is committed too, without waiting for a client write. entry (you saw it in the simulator). Without it, entries left over from earlier terms would stay uncommitted until the next client write happened to arrive, and a quiet cluster could keep them in limbo for a long time.

Followers learn about commits from the leader

Followers don't count anything. Every AppendEntries carries leaderCommit, the leader's commit indexCommit indexThe highest log position a server knows is committed: stored on a majority under the commit rule, so it can never be lost. Entries up to the commit index are safe to apply to the state machine., and the follower sets its own commit index to min(leaderCommit, index of the last new entry) (line 20 of the handler above), then applies everything up to it. In the simulator, that's the moment a follower's dashed cells turn solid, one heartbeat after the leader's.

Network splits, old leaders, and reads

The old leader in the minority

A Network partitionNetwork partitionWhen the network splits so that some nodes can't reach others, even though all of them are still running. Each side may think the other side is dead. splits the servers into groups that can't reach each other. Run the simulator in §05: cut the leader off and send it a write. Here's what happens, and why it's safe:

1. The old leader (termTerm (Raft)A numbered period of time in Raft, starting with an election. At most one leader per term. Every message carries the sender's term; a server that sees a higher term adopts it, and a leader that sees one steps down. 1) and one follower are on one side; three servers on the other.
2. The old leader keeps accepting client writes and appending them, but it can reach only 2 of 5 servers. Nothing it appends can commit, so none of those clients ever gets “OK”. They time out.
3. The three-server side stops getting heartbeats, elects a new leader in term 2, and commits new writes normally.
4. When the split heals, the old leader receives a message with term 2. It steps down at once, and the new leader's AppendEntries delete its uncommitted entries (§07).

Compare that with the split brain in the replication topic, where both Postgres servers accepted and confirmed writes. In Raft, two servers may both believe they're leader for a while, but only one of them, the one with a majority, can commit. The term numbers tell everyone which is which.

Reads are the hard part

A read doesn't change anything, so it's tempting to answer it straight from the leader's memory. But the server answering might be an old leader that hasn't noticed it was replaced:

1func (l *Leader) Read(key string) (string, error) {
2    l.waitForOwnTermCommit()       // 0. a new leader first commits its no-op (§07)
3    readIndex := l.commitIndex     // 1. remember how far commits have got
4    if !l.confirmLeadership() {    // 2. one heartbeat round: does a majority
5        return "", ErrNotLeader    //    still accept me as leader?
6    }
7    l.waitUntilApplied(readIndex)  // 3. let the state machine catch up
8    return l.kv[key], nil          // 4. now the answer is up to date
9}

A read that returns an older value after a newer one was already confirmed breaks linearizabilityLinearizabilityThe strongest consistency guarantee: the system behaves as if there were one copy and every operation happened at a single instant. Once any read sees a new value, no later read sees the old one. (“behaves like one copy”). There are three standard fixes:

ReadIndex has one easy-to-miss first step (line 2 in the stepper's code). A freshly elected leader's commit index can be behind: it may hold entries from earlier terms that are already committed but that it can't yet count as committed, the Figure 8 case from §07. Its commit index only becomes trustworthy once an entry from its own term commits. That's another job for the no-opNo-op entryAn empty log entry that a new Raft leader appends right after winning. Once it commits, every entry before it is committed too, without waiting for a client write. every new leader appends: until it commits, the leader holds reads back.

What clocks have to do with it

Raft's safety never depends on clocks. Terms and log indexes are counters, not timestamps. A server whose clock is an hour wrong, or that is paused for ten seconds, can cause extra elections but never two committed values. Clock skewClock skewThe difference between two machines' clocks. Even with time sync (NTP) it is often milliseconds, and can be much more after a fault, so timestamps from different machines can't reliably order events. only affects how quickly things happen.

Lease reads are different. A lease is a promise like “no one can be elected for the next 900 ms”, and it's only true if 900 ms on the leader is about 900 ms on the followers. If the leader's process freezes for two seconds (a long garbage-collectionGarbage collector (GC)Part of the language runtime that automatically frees memory nobody can reach anymore. Anything still referenced — like a stuck goroutine — is never freed. pause, or the cloud provider briefly pausing its virtual machine), it can wake up believing its lease is still valid after a new leader was already elected. Implementations measure leases with a Monotonic clockMonotonic clockA clock that only ever moves forward at a steady rate, used for measuring elapsed time and timeouts. It has no meaning across machines. Go's time.Now() carries a monotonic reading; JS has performance.now(). and subtract a safety margin, but the assumption remains. The Clocks topic covers why machines disagree about time.

Leases outside Raft: still use fencing tokens

The same pause problem hits your code when it uses a consensus store for locks. Your process holds a leaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”. on the lock “billing-job”, freezes for 15 seconds, the lease expires, another worker takes the lock, and then your process wakes up and keeps writing. Raft can't stop that: the stale write goes to your database, not through Raft.

The fix is the fencing token from the replication topic: each lock holder gets a number that grows with every new holder, sends it with every write, and the storage rejects numbers lower than the highest it has seen. etcd hands you that number for free (the key's creation revisionRevision (etcd)A counter etcd increases on every change to any key. Each key remembers the revision at which it was created, so a newer lock or leader key always has a bigger number: a ready-made fencing token.); §10 shows it in code.

Paxos in one section

PaxosPaxosThe classic consensus algorithm (Leslie Lamport, 1989). Proven correct but famously hard to understand and implement; Raft was designed as an easier alternative., described by Leslie Lamport (written in 1989, published in 1998), was the first practical, proven-correct consensus algorithm. It powers Google's Chubby lock service and Spanner, and variants run inside many databases. It's also famously hard to follow, which is why Raft was invented. You don't need Paxos to use etcd, but you'll meet the name everywhere, and seeing it makes Raft's choices clearer.

Roles and the two phases

Basic Paxos decides one value. There are two main roles (a server can play both): proposersProposer / acceptor (Paxos)The two main roles in Paxos. A proposer suggests a value under a unique, increasing number; acceptors promise to ignore lower numbers and accept proposals. A value is chosen once a majority of acceptors accepted it under one number. suggest values, and acceptors vote. Every proposal has a unique number, and numbers only go up.

• Phase 1, prepare/promise. A proposer picks a number n and asks the acceptors Prepare(n). An acceptor that hasn't promised anything higher promises to ignore all proposals below n, and replies with the highest-numbered proposal it has already accepted, if any.
• Phase 2, accept. With promises from a majority, the proposer sends Accept(n, v). The value v must be the value of the highest-numbered proposal reported in the promises; only if none was reported may it use its own. Acceptors accept unless they've promised a higher number since.

A value is chosen once a majority of acceptors have accepted it under the same number.

1type Acceptor struct {
2    promised int    // highest proposal number I promised to respect
3    accN     int    // number of the proposal I last accepted (0 = none)
4    accV     string // and its value
5}
6 
7// Phase 1: "will you listen to proposal n?"
8func (a *Acceptor) Prepare(n int) (ok bool, accN int, accV string) {
9    if n <= a.promised {
10        return false, 0, ""
11    }
12    a.promised = n               // promise: ignore anything numbered below n
13    return true, a.accN, a.accV  // and report what I accepted before
14}
15 
16// Phase 2: "please accept value v under number n"
17func (a *Acceptor) Accept(n int, v string) bool {
18    if n < a.promised {
19        return false // I promised someone with a higher number
20    }
21    a.promised, a.accN, a.accV = n, n, v
22    return true
23}

Why the “adopt the earlier value” rule

Without it, P2 would have proposed sushi under number 2, and A1 and A3 would have accepted. Meanwhile A1 and A2 had already accepted pizza under number 1. Two values, each accepted by a majority: exactly what consensus forbids.

With the rule, any proposer that gets past phase 1 has heard from a majority, and that majority overlaps the majority that accepted any chosen value (§04 again). So it learns the chosen value and proposes it again. Once a value is chosen, every later proposal carries the same value.

Dueling proposers, and Multi-Paxos

Paxos has its own liveness problem. P1 prepares 1; P2 prepares 2 (so P1's accepts get rejected); P1 retries with 3 (so P2's get rejected); and so on forever. The standard fix sounds familiar: pick one leader (a “distinguished proposer”) and use random back-off when two compete.

A replicated log needs a decision for every slot: entry 1, entry 2, and so on. Running full two-phase Paxos per slot costs two round trips each. Multi-PaxosMulti-PaxosRunning Paxos for every slot of a log, with a stable leader that does the first (prepare) phase once and then only needs one round trip per entry. Very close to Raft in practice. lets a stable leader run phase 1 once for all future slots, and then only phase 2, one round trip, per entry. At that point it looks a lot like Raft.

Raft vs Paxos

The two are closer than they look. Raft's term is Paxos's proposal number; Raft's election plays the role of phase 1; AppendEntries is phase 2. Raft's main simplification is the up-to-date voting rule: because only a server that already has every committed entry can win, a new Raft leader never has to collect old values from anyone.

Code: a tiny election in Go, and etcd in practice

Part 1: a toy election in Go

This is Raft's election and nothing else: no log, no commits. Five nodes run as five goroutinesGoroutineA very lightweight task in Go, started with the go keyword. The Go runtime runs thousands of them on just a few OS threads. inside one program, and a fake network of channelsChannelA typed pipe between goroutines. Send with ch <- v, receive with v := <-ch. By default the sender waits until a receiver takes the value. carries their messages. It's about 170 lines in total, shown in three pieces. First, the messages and the network:

// Package miniraft is a tiny Raft-style leader election: no log, just terms and votes.
package miniraft

import (
	"math/rand/v2"
	"sync"
	"time"
)

type Role string

const (
	Follower  Role = "follower"
	Candidate Role = "candidate"
	Leader    Role = "leader"
)

type kind int

const (
	voteReq kind = iota
	voteResp
	heartbeat
)

type message struct {
	kind    kind
	from    int
	term    int
	granted bool // only used in voteResp
}

// Network delivers messages between nodes and can cut a node off.
type Network struct {
	mu    sync.Mutex
	inbox []chan message
	cut   []bool
}

func (n *Network) send(to int, m message) {
	n.mu.Lock()
	dropped := n.cut[to] || n.cut[m.from]
	n.mu.Unlock()
	if dropped {
		return // a cut-off node neither sends nor receives
	}
	select {
	case n.inbox[to] <- m:
	default: // inbox full: drop it, like a lost packet
	}
}

func (n *Network) Disconnect(id int) {
	n.mu.Lock()
	n.cut[id] = true
	n.mu.Unlock()
}

Next, one node. The important design choice: each node's state is owned by one goroutine. Only run reads or writes term, votedFor, role and votes, so they need no lock. (run keeps them in its own local variables. The term and role fields on the struct are just a copy for the test: after handling each event, run copies its two locals into them under the mutex, so the test can peek at them safely.) Messages and timers all arrive through one select:

const heartbeatEvery = 20 * time.Millisecond

// electionTimeout is random, so two followers rarely time out together.
func electionTimeout() time.Duration {
	return 150*time.Millisecond + rand.N(150*time.Millisecond)
}

type Node struct {
	id   int
	size int
	net  *Network
	stop chan struct{}

	mu   sync.Mutex // guards the two fields below, which tests read
	term int
	role Role
}

func (n *Node) State() (int, Role) {
	n.mu.Lock()
	defer n.mu.Unlock()
	return n.term, n.role
}

func (n *Node) broadcast(m message) {
	for to := 0; to < n.size; to++ {
		if to != n.id {
			n.net.send(to, m)
		}
	}
}

// run is the node's only goroutine; it alone owns term, votedFor, role and votes.
func (n *Node) run(inbox <-chan message) {
	term, votedFor, role, votes := 0, -1, Follower, 0
	timer := time.NewTimer(electionTimeout())
	beat := time.NewTicker(heartbeatEvery)
	defer timer.Stop()
	defer beat.Stop()

	for {
		select {
		case <-n.stop:
			return

		case <-timer.C: // heard nothing for a whole timeout: start an election
			if role != Leader {
				term++
				role, votedFor, votes = Candidate, n.id, 1 // vote for myself
				n.broadcast(message{kind: voteReq, from: n.id, term: term})
			}
			timer.Reset(electionTimeout())

		case <-beat.C:
			if role == Leader {
				n.broadcast(message{kind: heartbeat, from: n.id, term: term})
			}

		case m := <-inbox:
			if m.term > term { // someone is in a newer term: catch up and follow
				term, votedFor, role = m.term, -1, Follower
			}
			switch m.kind {
			case voteReq:
				grant := m.term == term && (votedFor == -1 || votedFor == m.from)
				if grant {
					votedFor = m.from
					timer.Reset(electionTimeout())
				}
				n.net.send(m.from, message{kind: voteResp, from: n.id, term: term, granted: grant})
			case voteResp:
				if role == Candidate && m.term == term && m.granted {
					votes++
					if votes > n.size/2 { // a majority: 3 of 5
						role = Leader
						n.broadcast(message{kind: heartbeat, from: n.id, term: term})
					}
				}
			case heartbeat:
				if m.term == term {
					role = Follower // a candidate that hears a leader of its term gives up
					timer.Reset(electionTimeout())
				}
			}
		}

		n.mu.Lock()
		n.term, n.role = term, role
		n.mu.Unlock()
	}
}

Follow one election through the code: the <-timer.C case is “my election timeout ran out”; the voteReq case is the voter's side of §06 (newer term adopted, one vote per term; there's no log, so no up-to-date check); the voteResp case counts votes until votes > n.size/2, a majority. A heartbeat from a leader of the current term turns a candidate back into a follower and restarts the timer. The first if m.term > term line is the “higher term wins” rule from §04, applied to every message.

Finally, a cluster to hold the nodes:

type Cluster struct {
	Net   *Network
	Nodes []*Node
	wg    sync.WaitGroup // counts running node goroutines
}

func NewCluster(size int) *Cluster {
	c := &Cluster{Net: &Network{inbox: make([]chan message, size), cut: make([]bool, size)}}
	for id := range size {
		c.Net.inbox[id] = make(chan message, 64)
	}
	for id := range size {
		n := &Node{id: id, size: size, net: c.Net, stop: make(chan struct{}), role: Follower}
		c.Nodes = append(c.Nodes, n)
		c.wg.Add(1)
		go func() {
			defer c.wg.Done()
			n.run(c.Net.inbox[id])
		}()
	}
	return c
}

// Stop tells every node to quit and waits until all of them have.
func (c *Cluster) Stop() {
	for _, n := range c.Nodes {
		close(n.stop)
	}
	c.wg.Wait()
}

• for id := range size (Go 1.22+) loops with id = 0, 1, …, size−1. Since Go 1.22 each loop iteration gets its own id and n, so each goroutine sees its own node.
• A sync.WaitGroup is a counter of running goroutines: Add(1) before starting one, Done() when it finishes (here via defer), and Wait() blocks until the count is back to zero. go func() { … }() starts an anonymous function as a goroutine. Without Wait, Stop would return while nodes were still running, and they could outlive the test.

Testing it: cut off the leader

The test waits for a leader, cuts it off from the network (from the others' point of view, it crashed), and checks that a new leader appears in a higher term:

package miniraft

import (
	"testing"
	"time"
)

// leader waits until exactly one connected node is leader, and returns it.
func leader(t *testing.T, c *Cluster, skip map[int]bool) (id, term int) {
	t.Helper()
	deadline := time.Now().Add(3 * time.Second)
	for time.Now().Before(deadline) {
		time.Sleep(50 * time.Millisecond)
		id, term = -1, 0
		leaders := 0
		for i, n := range c.Nodes {
			tm, role := n.State()
			if !skip[i] && role == Leader {
				leaders++
				id, term = i, tm
			}
		}
		if leaders == 1 {
			return id, term
		}
	}
	t.Fatal("no single leader within 3 s")
	return
}

func TestNewLeaderAfterCrash(t *testing.T) {
	c := NewCluster(5)
	defer c.Stop()

	old, oldTerm := leader(t, c, nil)
	t.Logf("leader is node %d in term %d", old, oldTerm)

	c.Net.Disconnect(old) // "crash" it: its messages go nowhere
	cut := map[int]bool{old: true}
	next, nextTerm := leader(t, c, cut)
	t.Logf("node %d cut off; new leader is node %d in term %d", old, next, nextTerm)

	if nextTerm <= oldTerm {
		t.Fatalf("new leader's term %d should be > %d", nextTerm, oldTerm)
	}
	if _, role := c.Nodes[old].State(); role == Leader {
		t.Logf("node %d still thinks it is leader of term %d: nobody listens", old, oldTerm)
	}
}

t.Helper() makes failure messages point at the caller's line. skip[i] on a nil map (the first call passes nil) is fine in Go: reading a missing key returns false.

Run it with the Race detectorRace detectorGo's built-in checker (go test -race). It watches every memory access while the program runs and reports two goroutines touching the same memory with nothing ordering them. on, which watches for two goroutines touching the same memory without a lock:

$ go test -race -v ./...
=== RUN   TestNewLeaderAfterCrash
    raft_test.go:36: leader is node 3 in term 1
    raft_test.go:41: node 3 cut off; new leader is node 2 in term 2
    raft_test.go:47: node 3 still thinks it is leader of term 1: nobody listens
--- PASS: TestNewLeaderAfterCrash (0.46s)
PASS
ok  	example.com/miniraft	1.467s

Which node wins changes from run to run (the timeouts are random). It also passed 30 runs in a row with -race -count=30. Notice the last log line: the cut-off node still believes it is the leader of term 1. That's §08 in miniature: being leader in your own head is harmless as long as nobody accepts your old term.

Part 2: leader election with etcd

Most of the time you don't need Raft inside your service: you need one instance of your service to be in charge, say, only one of the 6 running copies (podsPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy.) of your service should run the nightly billing job. etcd already runs Raft; your code just asks it for a leader. Both clients below put a key in etcd attached to a leaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”., a permission that expires unless the client keeps renewing it. If your process dies or loses contact with etcd, the lease runs out (here after 10 seconds), etcd deletes the key, and the next candidate takes over.

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"localhost:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// A session = a lease that this client keeps alive in the background.
	// If this process dies or is cut off, etcd deletes its keys after ~10 s.
	sess, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
	if err != nil {
		log.Fatal(err)
	}
	defer sess.Close()

	host, _ := os.Hostname()
	election := concurrency.NewElection(sess, "/jobs/billing/leader")

	// Blocks until we are the leader (or ctx is cancelled).
	if err := election.Campaign(ctx, host); err != nil {
		log.Fatal(err)
	}
	token := election.Rev() // ← grows with every new leader: a fencing token
	log.Printf("I am the leader, fencing token %d", token)

	select {
	case <-sess.Done(): // lease lost: someone else may be leader now
		log.Print("lost leadership: stop all leader-only work")
		return
	case <-ctx.Done(): // Ctrl+C: hand over politely
		resignCtx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		_ = election.Resign(resignCtx)
	}
}

The fencing token

Leadership through a lease has the pause problem from §08: your process can freeze, lose the lease, and wake up still acting as leader. election.Rev() in Go returns the etcd revisionRevision (etcd)A counter etcd increases on every change to any key. Each key remembers the revision at which it was created, so a newer lock or leader key always has a bigger number: a ready-made fencing token. at which this candidate's key was created. Every later leader's key is created later, so it gets a larger number: a ready-made Fencing tokenFencing tokenA number that increases every time a new leader is chosen or a lock/lease is granted. Storage remembers the highest number it has seen and rejects writes carrying an older one, so a paused old holder can't overwrite the new holder's work.. Send it with every write, and have the storage refuse older ones:

-- $1 = the fencing token from election.Rev()
UPDATE billing_state
SET    last_run = now(), fence = $1
WHERE  id = 1 AND fence <= $1;   -- 0 rows updated → a newer leader exists: stop

Without a check like this at the storage, the lease alone can't protect you. The etcd3 package's own documentation for its lock() helper warns the same thing: in rare situations the caller may lose the lock while its operations are still running.

Running consensus in production

3 or 5 servers, and where to put them

• 3 servers survive 1 failure. Fine for most clusters, but during planned maintenance (one server down for an upgrade) you're one crash away from an outage.
• 5 servers survive 2, so maintenance plus an unexpected crash is still fine. Commits get only barely slower in one data center (the leader waits for the 2nd-fastest of 4 followers instead of the faster of 2), but every server does the work of every write, and elections involve more servers.
• 7 or more rarely pays off. More voters means more messages per commit for little gain. Add read capacity with non-voting members instead (below).
• Place servers so no single failure takes out a majority. 3 servers in 2 data centers means one data center has 2 of them, and losing it loses the majority. Use 3 availability zonesAvailability zone (AZ)One of a few separate data centers inside a cloud region (e.g. us-east-1a, 1b, 1c), with its own power and network, so one can fail without taking down the others. (or 3 regions) for 3 or 5 servers.

Across regions, the arithmetic gets expensive: the leader must hear from a majority, and with servers spread over 3 regions, that majority always includes another region. Every write pays tens of milliseconds, and an election timeout tuned for one data center will fire constantly. That's why most systems run consensus within one region across zones, and only go cross-region when surviving a whole region's loss is worth it.

Keep consensus off the hot path

A consensus store is a small, slow, very reliable notebook for the facts everything else depends on. It is not your main database or a message queue:

• etcd's default storage quota is about 2 GB, and its docs advise not to go far beyond 8 GB. Requests are limited to about 1.5 MB by default.
• Every write is a disk flush on a majority, so throughput is thousands to low tens of thousands of small writes per second, not millions.
• Kubernetes itself is one of etcd's heaviest users. Storing your application's per-request data in the same etcd competes with Kubernetes' Control planeControl planeThe part of a system that manages and configures it — which servers exist, routing rules, who is leader, desired state — as opposed to the data plane, which carries the actual requests. Kubernetes' API server and etcd are a control plane., the part that runs your whole cluster.

The usual pattern: put who is leader, which shard lives where, feature flags and locks in etcd/ZooKeeper/Consul; put the actual data in a database. Systems that need consensus for all their data (CockroachDB, TiKV, YugabyteDB) split it into thousands of small ranges, each with its own Raft group and leader, so the work is spread over many servers (Multi-RaftMulti-RaftRunning thousands of small, independent Raft groups in one cluster, one per range of data, each with its own leader, so the write work spreads across all servers instead of one leader.).

Disks: fsync is the commit

Before a Raft server answers “OK, stored”, it must fsyncfsyncA system call that forces data the OS has buffered in memory to actually be written to the disk. Slow (often milliseconds), but without it a power cut can lose “written” data. the entry to disk (§05). So the slowest part of a commit inside one data center is usually the disk, not the network. etcd's guidance is that the p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them. of fsync time for its write-ahead logWrite-ahead log (WAL)An append-only file where a database records each change before applying it. After a crash it replays the log to recover; replicas can follow it to copy changes. (the file where it appends entries) should stay below about 10 ms.

Slow or shared disks (network storage shared with other busy machines, a disk that other programs also hammer) cause heartbeats to be late, followers to time out, and leaders to change for no visible reason. Leader flapping in etcd is very often a disk problem. Give consensus servers dedicated SSDs.

Changing who's in the cluster

Adding or removing a server changes what “majority” means. Switch every server from the old member list to the new one at different moments, and for a short time the old majority and the new majority might not overlap: two leaders in the same term become possible. Raft offers two safe methods:

• Joint consensusJoint consensusRaft's original method for changing which servers are in the cluster: for a while, decisions need a majority of the old set AND a majority of the new set, so the two can never decide independently.: a transition phase in which every decision needs a majority of the old set and a majority of the new set.
• One server at a time: adding or removing a single server always keeps old and new majorities overlapping, so changes are made one by one, each committed before the next.

New servers usually join as a learnerLearner (non-voting member)A server that receives the log but doesn't vote or count toward majorities. New servers start as learners until they have caught up, so a slow newcomer can't stall commits. (non-voting member) first: they copy the log without counting toward majorities until they've caught up, so a brand-new, empty server can't stall commits. etcd has supported learners since v3.4.

Snapshots and log compaction

The log grows with every write forever, unless something trims it. Each server periodically writes a snapshotSnapshot (Raft)A saved copy of the whole state machine as of some log index. Everything up to that index can be deleted from the log, and a far-behind follower can be sent the snapshot instead of the old entries. of its state machine (“the whole key-value store as of entry #1,000,000”), and then deletes the log up to that point: log compactionLog compactionA Kafka cleanup mode that keeps at least the newest message for each key and deletes older ones, so the topic holds the latest state per key instead of full history.. A follower that's so far behind that the leader has already deleted the entries it needs receives the snapshot instead (Raft's InstallSnapshotInstallSnapshotThe Raft message a leader uses to send a follower its whole snapshot, when the follower is so far behind that the log entries it needs have already been deleted. message), then continues from there.

(etcd also has a separate “compaction” of its key history, the old versions of keys it keeps for watchesWatch (etcd)A long-lived request that streams every change to a key or a key prefix, so a client can react to changes instead of asking again and again. etcd keeps old versions of keys so a watch can resume from an earlier revision.. That's a different setting from Raft log compaction.)

Elections that shouldn't happen

• A server that rejoins after being cut off has been timing out and raising its term the whole time. When it reconnects, its high term makes the healthy leader step down: a pointless election. Pre-votePre-voteAn optional extra Raft round: before starting a real election (and raising its term), a server asks “would you vote for me?”. A server that was merely cut off then can't disrupt a healthy leader when it reconnects. fixes it: a server first asks “would you vote for me?” and only raises its term if a majority says yes. etcd has it as an option (--pre-vote, on by default in recent versions).
• A leader that can't reach a majority keeps believing it's leader (§08). CheckQuorumCheckQuorumA Raft option: a leader steps down by itself if it hasn't heard from a majority within an election timeout, so a leader cut off in a minority stops acting as leader soon. makes it step down by itself if it hasn't heard from a majority within an election timeout.
• Planned maintenance: transfer leadership to another server before restarting the leader, instead of letting the cluster discover it's gone after a timeout.

What to monitor

Pitfalls

Check yourself

Predict the answer before you click. Each explanation points back to the section that covers it.

Cheat sheet