Time, Clocks & Ordering

The 30-second version

Here's a bug you can meet in your first month on a backend team. A customer pays for order #42 and cancels a moment later. The payments server charges the card and sends a message to the refunds server, which refunds the money. Both servers write log lines, and a log tool merges them, sorted by timestamp. Drag the slider:

The refund was caused by the charge, so it must have come after it. Yet the log shows it first. Which server's clock was right? Neither, exactly. Both clocks were a little off, and when two events are only milliseconds apart on different machines, “a little off” is enough to flip their order. The difference between two machines' clocks is called clock skewClock skewThe difference between two machines' clocks. Even with time sync (NTP) it is often milliseconds, and can be much more after a fault, so timestamps from different machines can't reliably order events..

Three ways to deal with time

This page is about how a distributed systemDistributed systemA program that runs on several machines that cooperate over a network. Any machine or link can fail or slow down independently, which is what makes them hard. (a program spread across several machines, each called a nodeNodeOne machine (or one process) taking part in a distributed system — a database replica, a broker, a server in a cluster.) copes with time. There are three families of answers:

Why computer clocks disagree

The time a computer “knows”

When your code asks for the current time (Date.now() in JavaScript, time.Now() in Go), it reads the wall clockWall-clock timeThe time of day your computer believes it is (like 2026-10-30 12:00:01). It can jump backwards or forwards when it's corrected, so it's unreliable for measuring durations or ordering events across machines.: the computer's belief about the time of day. Servers keep it in UTCUTCCoordinated Universal Time: the world's reference time, with no time zones and no daylight saving. Servers and databases should store and log times in UTC and only convert to local time for display. (no time zones, no daylight saving), and code usually handles it as Unix timeUnix timeA time written as the number of seconds (or milliseconds) since 1970-01-01 00:00:00 UTC, “the epoch”. Date.now() in JS and time.Now().UnixMilli() in Go both return it in milliseconds.: the number of milliseconds since 1 January 1970, UTC. For example, Date.now() might return 1793361600000.

Drift: every clock runs at its own speed

The wall clock is driven by a quartz oscillatorQuartz oscillatorThe tiny vibrating crystal that a computer counts to keep time. Cheap and steady, but no two vibrate at exactly the same speed, so every machine's clock drifts differently., a crystal that vibrates millions of times per second. The computer counts vibrations: “this many = one second”. But each crystal is a tiny bit faster or slower than its label says, and the speed changes with temperature. How far a clock wanders is its driftClock driftHow fast a clock gains or loses time compared with real time — for example “20 microseconds per second”. Every quartz clock drifts a little, and the rate changes with temperature., measured in ppmppm (parts per million)A way to write tiny rates. 1 ppm = 1 microsecond gained or lost per second, which adds up to about 86 milliseconds per day. (parts per million).

Worked example. 1 ppm means gaining or losing 1 microsecond (µs, a millionth of a second) every second. A day has 86,400 seconds. So a server whose clock runs 20 ppm fast gains 20 × 86,400 µs ≈ 1.7 seconds per day. Typical server crystals are somewhere around 10–100 ppm off (a rough range, it depends on the hardware and the temperature), which means seconds per day. Two servers drifting in opposite directions pull apart even faster: their rates add up (+40 ppm and −25 ppm are 65 ppm apart).

Time sync pulls clocks back

Left alone, clocks would be minutes apart within weeks. So every server runs a time-sync program that regularly asks a more accurate time server “what time is it?” and corrects the local clock. The standard protocol is NTPNTPNetwork Time Protocol: how computers keep their wall clocks roughly in sync with time servers — usually within milliseconds, but never perfectly.; §04 shows how it works. The interesting part here is how it corrects a wrong clock. There are two ways:

• StepStepping a clockCorrecting a wrong clock by jumping it to the right time in one go. Fast, but the clock may jump backwards, so code reading it can see time going in reverse.: jump the clock straight to the right time. Fast, but if the clock was ahead, it jumps backwards, and any code reading the clock sees time run in reverse.
• SlewSlewing a clockCorrecting a wrong clock gradually, by making it run a little faster or slower until the error is gone. Time never jumps or goes backwards, but a large error takes a long time to fix.: make the clock tick slightly slower (or faster) until the error is gone. Time never jumps, but a big error takes a long time to fix. A classic limit is 500 ppm, i.e. 0.5 ms of correction per second, so fixing a 1-second error takes about 2,000 seconds (half an hour).

Try both. Server A drifts +40 ppm (fast), server B −25 ppm (slow):

Three things to notice:

1. Without sync, the skew grows without limit: drag Look at time of day to 24:00 and the two servers are about 5.6 seconds apart.
2. With sync, the error stays small but never reaches zero. Between corrections, both clocks drift again. Real NTP corrects every few minutes, not every two hours, so real errors are milliseconds, not hundreds of milliseconds. But they're never zero.
3. Stepping a fast clock makes time go backwards. Any code that computes “now minus earlier” across that moment gets a negative number. §03 shows how to protect yourself.

Leap seconds: when the official time itself jumps

The Earth's rotation is slightly irregular, so the official world time, UTC, occasionally adds a leap secondLeap secondAn extra second occasionally added to UTC (written 23:59:60) to keep clocks in step with the Earth's slightly irregular rotation. 27 have been added since 1972; the last one was at the end of 2016.: the last minute of a day gets 61 seconds, ending in 23:59:60. It has happened 27 times since 1972, most recently at the end of 2016. Unix time has no way to write 23:59:60, so systems traditionally repeated a second, which means the clock stepped back by one second.

Software handled this badly. The 2012 leap second set off a Linux kernel bug that sent many servers to 100% CPU, and the 2016 one crashed part of Cloudflare's DNS service (§03 tells that story). Today, big providers such as Google and AWS use a leap smearLeap smearInstead of inserting a leap second all at once, the time servers run their clocks very slightly slow for many hours (Google: 24 hours) so the extra second is absorbed gradually and no clock ever shows 23:59:60 or jumps.: their time servers run very slightly slow for many hours (Google smears over 24 hours), absorbing the extra second so no clock ever jumps. In 2022, the world's timekeeping authorities voted to stop adding leap seconds by 2035.

Wall clock vs monotonic clock

A monotonic clockMonotonic clockA clock that only ever moves forward at a steady rate, used for measuring elapsed time and timeouts. It has no meaning across machines. Go's time.Now() carries a monotonic reading; JS has performance.now(). is the computer's stopwatch: a counter that only moves forward at a steady rate, usually counting from when the machine booted. Time sync (§02) never steps it. Its value means nothing on its own (“5,000.2 seconds since boot” isn't a time of day), and it means nothing on another machine. It's only good for one thing: end − start, inside one running program (one OS processProcessOne running program (e.g. one Node server or one Go binary). It has its own memory and one or more threads.).

Here's why that matters. Drag the slider to change the size of the NTP step:

Measuring a duration, correctly

package main

import (
	"fmt"
	"time"
)

func main() {
	start := time.Now()              // carries BOTH a wall reading and a monotonic reading
	fmt.Println(start)               // 2026-10-30 12:34:47.996327924 +0000 UTC m=+0.000042757

	time.Sleep(50 * time.Millisecond)

	elapsed := time.Since(start)     // ← uses the monotonic readings: immune to clock steps
	fmt.Println(elapsed)             // 51.141069ms
}

When the monotonic reading disappears

In Go, the monotonic reading lives only inside the running process. Several common operations drop it silently, and then Sub falls back to wall-clock math:

b, _ := json.Marshal(start) // "2026-10-30T12:34:47.996327924Z" ← wall time only (error ignored for brevity)
s := start.Round(0)         // s: same time, monotonic reading stripped (that's the idiom for it)
u := start.UTC()            // u: converting the time zone also strips it

start itself is unchanged: a time.Time is a plain value, and these return new values. So a time.Time you read back from JSONJSONJavaScript Object Notation: a text format for data — {"id": 7, "tags": ["a"]}. Readable by humans and every language; the default body format for web APIs., a database or another service is wall time only. Subtracting it from time.Now() is a wall-clock measurement, with all the jumps that implies. That's fine for “roughly how old is this record?”, and wrong for timeouts.

The bug that crashed a DNS service

At midnight UTC on 1 January 2017, a leap second was added (§02) and some clocks repeated a second. Cloudflare's DNS software, written in Go, measured how long upstream servers took to answer by subtracting two wall-clock readings. Across the leap second, one of those durations came out negative, and the code passed it to rand.Int63n, which panics on a negative number. Part of their DNS service went down for a while. At the time, Go's time.Now() had no monotonic reading; it gained one in Go 1.9 (August 2017), partly because of incidents like this.

Go's timers, time.Sleep and context.WithTimeout already use the monotonic clock, and so do setTimeout and setInterval in Node and modern browsers, so those are safe. The bugs come from code that stores Date.now() or a deserialized time and subtracts later. (One caveat: on some systems the monotonic clock pauses while the machine is asleep, so a laptop that slept for an hour may not count that hour.)

NTP in practice

One exchange, four timestamps

The client sends a request and notes the time on its own clock (t1). The server notes when it arrived (t2) and when it sent the reply (t3), both on the server's clock. The client notes when the reply arrived (t4), on its own clock again. From those four numbers:

• Delay δ = (t4 − t1) − (t3 − t2): the whole round trip minus the time the server spent thinking. That's the network part, the round-trip time (RTT)Round-trip time (RTT)How long it takes for a message to go from one machine to another and for the reply to come back. Roughly 1 ms inside a data center, 70–150 ms across an ocean..
• Offset θ = ((t2 − t1) + (t3 − t4)) / 2: how much the client should add to its clock.

Worked example. The client's clock is 50 ms behind. Each direction takes 10 ms, and the server needs 1 ms. The client sends at what its clock calls 950 (real time 1000). It arrives at real time 1010, so t2 = 1010; the server replies at t3 = 1011. The reply arrives at real time 1021, which the client's clock calls t4 = 971.

• δ = (971 − 950) − (1011 − 1010) = 21 − 1 = 20 ms of network.
• θ = ((1010 − 950) + (1011 − 971)) / 2 = (60 + 40) / 2 = +50 ms. Exactly right: the client is 50 ms behind.

Now make the two directions different:

When the request takes 30 ms and the reply 10 ms, the estimate is off by 10 ms: half the difference between the two directions. NTP can't see that difference; from the client's point of view both cases look the same. The worst possible error is half the round trip (δ / 2). That's why NTP over the internet, with long and lopsided routes, is much less precise than NTP inside a data center.

A real time-sync program

One exchange is noisy, so a real NTP client like chronychronyThe NTP program used by most modern Linux distributions (chronyd, plus the chronyc command to inspect it). It keeps the system clock synced to time servers. (the default on most Linux distributions) talks to several servers every few minutes, throws out outliers, and prefers exchanges with a small round trip. Two more ideas make it good:

• It learns your crystal's drift. After a while chrony knows “this clock runs 12 ppm slow” and continuously speeds it up by that much, so the error between syncs stays small. That's the Frequency line below.
• It mostly slews, and only steps when it must (§02). A common default config line is makestep 1.0 3: step only if the error is over 1 second, and only during the first 3 updates after boot. After that, everything is slewed, so time never jumps backwards.

$ chronyc tracking            # illustrative output, abridged
Reference ID    : A9FEA97B (169.254.169.123)
Stratum         : 4
System time     : 0.000012345 seconds fast of NTP time
Frequency       : 12.345 ppm slow
Root delay      : 0.000250000 seconds
Leap status     : Normal

Stratum is the server's stratumStratum (NTP)How many hops a time server is from a real reference clock. Stratum 0 is the atomic clock or GPS receiver itself, stratum 1 is a server wired to it, stratum 2 syncs from stratum 1, and so on.: how many hops it is from a reference clock such as an atomic clockAtomic clockA clock that counts the vibrations of atoms (caesium or rubidium). Extremely stable — off by far less than a microsecond per day. GPS satellites carry them, which is why a GPS receiver is also a very good time source. or a GPS receiver. System time is the current estimated error (12 µs here). Root delay is the total round trip all the way up to the reference clock, which bounds how wrong the estimate can be. On a machine without chrony, timedatectl at least tells you whether the clock is synchronized.

How close can clocks get?

These are rough, typical figures, not guarantees:

In the cloud, use the provider's time service instead of public servers: AWS offers the Amazon Time Sync Service at 169.254.169.123 (and, on some instance types, a PTP hardware clock with microsecond-level accuracy), and Google Cloud VMs sync from metadata.google.internal. Both apply a leap smear (§02).

Even the best setup leaves some error. The next section asks a different question: if clocks can't be trusted to order events across machines, what can?

Order without clocks: happened-before

In 1978, Leslie Lamport wrote a short paper, Time, Clocks, and the Ordering of Events in a Distributed System, that changed how engineers think about time. His observation: between machines, we can't know which event happened at an earlier instant. But we usually don't need to. What we need is: did one event know about the other? A refund that was triggered by a charge must be ordered after the charge. Two unrelated events on different servers can go in either order and nobody can tell the difference.

The three rules

Lamport called each participant a processProcess (in distributed-systems papers)One participant in a distributed system — usually one machine or one service instance — that does things one at a time. Lamport's papers say “process”; this page mostly says node. Not the same as an operating-system process.: a server, a phone, anything that does things one at a time. In this section and the next two we keep his word; it means the same as node from §01, and it is not the operating-system process of §03 (one machine could even run several). An event is anything a process does: a local step, sending a message, or receiving one. Lamport defined happened-beforeCausality (happened-before)Event A could have influenced event B — e.g. B replies to A's message. Distributed systems try to preserve this order even when clocks disagree., written a → b, with three rules:

1. Same process: if a and b happen on the same process and a comes first, then a → b.
2. Messages: if a is “send message m” and b is “receive m”, then a → b.
3. Chains: if a → b and b → c, then a → c.

That's all. If neither a → b nor b → a, the events are concurrentConcurrent events (distributed)Two events where neither happened-before the other: no chain of messages links them, so neither could have influenced the other. It says nothing about whether they happened at the same moment., written a ∥ b.

See it on a diagram

Distributed events are drawn as a space-time diagramSpace-time diagramA picture with one horizontal line per machine (time flows left to right), dots for events and arrows for messages. Lamport used it in 1978; it's the standard way to draw distributed events.: one horizontal line per process (time flows left to right), a dot per event, an arrow per message. The horizontal position only shows each process's own order; dots on different lines that line up vertically aren't “at the same time”.

Click any event. Its past (every event that happened before it) lights up violet, its future cyan, and the concurrent ones go hollow. To find the past by hand, start at the event and walk backwards: left along its own line, and back along any arrow that ends on that line.

Try these with the example loaded:

• Click B1. Nothing lights up on A at all: B1 happened before B received m1, so it knows nothing about A. A1 and A2 are concurrent with B1.
• Click C3. Its past includes A1 and A2, even though C never talked to A directly: A2 → B2 (message m1), B2 → B3 (same process), B3 → C3 (message m2). That's rule 3.
• Click A3 and B3 together: concurrent. Each has news the other lacks.
• Add your own events and messages; leave a message in flight, then deliver it and watch the receiver's later events gain a bigger past.

Why this is the right kind of order

Happened-before is exactly the order you can't afford to get wrong: a reply after its question, a refund after its charge, a delete after the create it deletes. For concurrent events, no observer could have told which came first, so any order is defensible, as long as every machine picks the same one. It's a partial order: some pairs are ordered, some aren't.

To use it in code, a machine needs to attach something to each event that captures happened-before. Clocks can't (§01). The next two sections build counters that can.

Lamport clocks

The rules

A Lamport clockLamport clockA single counter per machine: add 1 for every event, send it with every message, and on receive jump to max(mine, received) + 1. If A happened-before B, A's number is smaller — but a smaller number doesn't prove A came first. is simple: each process keeps a single integer, starting at 0:

// local event or send:
time = time + 1            // and a send attaches `time` to the message
// receive a message that carries `remote`:
time = Math.max(time, remote) + 1

The max is the whole trick. Receiving a message means “I now know about everything the sender knew”, so the receiver's counter jumps to at least the sender's number, then counts the receive itself.

Worked example (the diagram below, which is the same scenario as §05). A has a local event (A1 = 1) and then sends m1 (A2 = 2, the message carries 2). Meanwhile B has a local event (B1 = 1). When m1 arrives, B computes max(1, 2) + 1 = 3. The receive gets a bigger number than the send, as it must.

Every message arrow goes from a smaller number to a bigger one, and numbers grow along each line. Follow those two facts through the three rules of §05 and you get the clock condition:

If a → b, then L(a) < L(b).

A total order for free

Sometimes you need every machine to agree on one order of all events, even concurrent ones. For example, to decide which of two conflicting requests to apply first, without asking anyone. Sort by the Lamport number, and break ties by process name: (1, A) < (1, B) < (1, C) < (2, A) … The line under the diagram shows this order. It's a total orderTotal orderAn ordering where every pair of items can be compared, so there is always a definite first and second — like numbering every message in a chat 1, 2, 3. A partial order leaves some pairs with no defined order., and it's consistent with happened-before: whenever a → b, a comes first in the list.

Lamport's paper used exactly this to build a lock that several machines share without a central server. Today you meet it in places like collaborative-editing libraries (Automerge, for example, gives each change an ID made of a counter and the author's ID, and sorts them the same way) and as a tie-breaker in conflict resolution.

What a Lamport clock can't tell you

The clock condition only works in one direction. L(a) < L(b) does not mean a → b. In the example, click B1 (1) and A2 (2): 1 < 2, yet they're concurrent. Click C2 (2) and B2 (3): same story. From two numbers alone, you can't tell “a happened before b” apart from “a and b are concurrent”. What you can conclude:

• L(a) < L(b) → b did not happen before a. (Either a → b, or they're concurrent.)
• L(a) = L(b) on different processes → they're concurrent.

Two more limits:

• The numbers have nothing to do with real time. A busy server's counter runs ahead of a quiet one's. You can't ask “what happened around 12:00?”
• The total order is arbitrary for concurrent events. It's consistent and everyone agrees on it, but “A wins ties because its name sorts first” isn't the same as “A was really first”.

If you need to detect concurrency, for example to notice that two users edited the same document without seeing each other's change, one number per event isn't enough. You need one number per process. That's §07.

Vector clocks

The rules

A vector clockVector clockOne counter per machine, kept by every machine and sent with every message. Comparing two vectors tells you whether one event happened-before the other or whether they were concurrent — which a single number can't. generalizes the Lamport counter. With three processes A, B and C, every process keeps a list of three numbers, [A, B, C], all starting at 0:

1. Local event or send: add 1 to your own entry. A send attaches a copy of the whole vector to the message.
2. Receive: take the bigger number in each position (yours vs. the message's), then add 1 to your own entry.

Entry B in A's vector means: “I know about the first that many events on B.” Receiving a message passes on everything the sender knew, which is why you take the maximum position by position.

Worked example. A1 = [1,0,0], A2 = [2,0,0] (sent as m1). B1 = [0,1,0]. When m1 arrives at B: max of [0,1,0] and [2,0,0] is [2,1,0]; add 1 to B's entry → B2 = [2,2,0]. B2 now “knows” A's first two events, which is exactly its past.

Comparing two vectors

For example, A3 = [3,0,2] and B3 = [2,3,0]. A3 is bigger in A's entry, B3 in B's entry. Each knows something the other doesn't: concurrent. Unlike Lamport numbers, this comparison is exact in both directions: x → y if and only if x's vector is ≤ y's (and different).

Switch between the three modes on the same diagram. Click two events in vector mode and the box explains the comparison; flip to Lamport with the same two selected to see what a single number loses:

What it's for: detecting conflicts

The classic use is spotting conflicting writes in a database with several copies. Each stored value carries a vector. When two versions of the same key meet:

• If one vector is ≤ the other, the newer version simply replaces the older one: its writer had already seen the older value.
• If they're concurrent, two clients wrote without seeing each other's change. That's a real conflict. Keep both versions (called siblingsSiblings (conflicting versions)Two or more versions of the same key, written concurrently, that a database keeps side by side until the app (or a user) merges them into one.) and let the app merge them, instead of silently throwing one away.

The Replication topic shows where these conflicts come from and the strategies for resolving them. Databases usually attach a version vectorVersion vectorA small set of counters, one per replica, attached to a value. Comparing two vectors tells you whether one version came after the other or whether they were written concurrently. to each value: the same comparison, but counting something different. A vector clock ticks on every event of a process; a version vector ticks only when a replicaReplicaA full copy of a database kept on another machine and updated as the original changes, for reading load and for taking over if the original dies. writes that key, so it has one counter per replica that wrote the value. Riak, for example, uses version vectors.

The cost: one entry per process

A vector clock has one entry per participant. With 3 servers that's tiny. With every browser tab or phone as a participant, vectors grow to thousands of entries and travel with every message and every stored value. Real systems fight this:

• Count servers, not clients. Only the replicas that coordinate writes get entries.
• Prune old entries. Riak historically trimmed vectors past a size limit. That's dangerous: once an entry is gone, two versions can look concurrent when they weren't (a false conflict), or ordered when they weren't (silently losing a write).
• Better encodings, such as dotted version vectors, keep the size proportional to the replicas while still detecting conflicts correctly.

Building them in Go and JavaScript

Logical clocks are small enough to write yourself, and writing them is the best way to remember the rules. All the code below was run with Go 1.24 and Node 22, and every test passes, including under Go's race detector (go test -race, which reports two goroutines touching the same memory unsafely).

Lamport clock

package clocks

import "sync"

// Lamport is a Lamport clock: one counter per node.
type Lamport struct {
	mu   sync.Mutex // a server handles many requests at once
	time uint64
}

// Tick: call for every local event and before every send.
// The result is the event's timestamp (attach it to the message).
func (l *Lamport) Tick() uint64 {
	l.mu.Lock()
	defer l.mu.Unlock()
	l.time++
	return l.time
}

// Recv: call when a message arrives, with the timestamp it carried.
func (l *Lamport) Recv(remote uint64) uint64 {
	l.mu.Lock()
	defer l.mu.Unlock()
	l.time = max(l.time, remote) + 1 // ← jump past the sender, then count this event
	return l.time
}

Vector clock

package clocks

import (
	"maps"
	"sync"
)

// VClock maps a node ID to how many of that node's events we know about.
type VClock map[string]uint64

type Order int

const (
	Equal      Order = iota // 0
	Before                  // 1
	After                   // 2
	Concurrent              // 3
)

// String makes test failures print "Before" instead of "1".
func (o Order) String() string {
	return [...]string{"Equal", "Before", "After", "Concurrent"}[o]
}

// Compare says how a relates to b.
func Compare(a, b VClock) Order {
	aAhead, bAhead := false, false
	for id, n := range a {
		if n > b[id] { // a missing key reads as 0
			aAhead = true
		}
	}
	for id, n := range b {
		if n > a[id] {
			bAhead = true
		}
	}
	switch {
	case aAhead && bAhead:
		return Concurrent // each knows something the other doesn't
	case aAhead:
		return After
	case bAhead:
		return Before
	default:
		return Equal
	}
}

// Node owns one vector clock. Safe for concurrent use.
type Node struct {
	mu    sync.Mutex // Go maps crash on concurrent writes, so guard the clock
	id    string
	clock VClock
}

func NewNode(id string) *Node { return &Node{id: id, clock: VClock{}} }

// Event: a local event or a send. Returns a copy to store or attach to a message.
func (n *Node) Event() VClock {
	n.mu.Lock()
	defer n.mu.Unlock()
	return n.tick()
}

// Recv merges the sender's vector, then counts the receive as an event.
func (n *Node) Recv(remote VClock) VClock {
	n.mu.Lock()
	defer n.mu.Unlock()
	for id, c := range remote {
		n.clock[id] = max(n.clock[id], c) // ← element-wise max
	}
	return n.tick() // not Event(): we already hold the lock
}

// tick adds 1 to our own entry. The caller must hold n.mu.
func (n *Node) tick() VClock {
	n.clock[n.id]++
	return maps.Clone(n.clock)
}

Tests

The tests replay the start of the §05 scenario: A has two events (the second is a send), B has one local event and then receives.

package clocks

import (
	"sync"
	"testing"
)

func TestLamport(t *testing.T) {
	var a, b Lamport // zero value = clock at 0, ready to use

	a.Tick()        // A: local event → 1
	msg := a.Tick() // A: send        → 2
	b.Tick()        // B: local event → 1

	if got := b.Recv(msg); got != 3 { // max(1, 2) + 1
		t.Fatalf("receive = %d, want 3", got)
	}
}

func TestVector(t *testing.T) {
	a, b := NewNode("A"), NewNode("B")

	a1 := a.Event()   // {A:1}
	msg := a.Event()  // {A:2}        sent to B
	b1 := b.Event()   // {B:1}        B hasn't heard from A yet
	b2 := b.Recv(msg) // {A:2, B:2}

	tests := []struct {
		name string
		x, y VClock
		want Order
	}{
		{"send before receive", msg, b2, Before},
		{"a1 before a2", a1, msg, Before},
		{"b1 and a1 are concurrent", b1, a1, Concurrent},
		{"receive after b1", b2, b1, After},
	}
	for _, tc := range tests {
		if got := Compare(tc.x, tc.y); got != tc.want {
			t.Errorf("%s: got %v, want %v", tc.name, got, tc.want)
		}
	}
}

// 50 goroutines use one node at once, like requests in a server.
func TestConcurrentUse(t *testing.T) {
	n := NewNode("A")
	var wg sync.WaitGroup
	for i := 0; i < 50; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			n.Event()
			n.Recv(VClock{"B": 3})
		}()
	}
	wg.Wait()
	if got := n.Event()["A"]; got != 101 { // 50 events + 50 receives + this one
		t.Fatalf("A = %d, want 101", got)
	}
}

$ go test -race -v ./...
=== RUN   TestLamport
--- PASS: TestLamport (0.00s)
=== RUN   TestVector
--- PASS: TestVector (0.00s)
=== RUN   TestConcurrentUse
--- PASS: TestConcurrentUse (0.00s)
PASS

$ node --test
# tests 2
# pass 2
# fail 0

Carrying the clock in messages

A clock only works if its value travels with every message: in an HTTP header, in gRPC metadataMetadata (gRPC)Key–value pairs sent with a call, outside the message itself — gRPC's name for HTTP headers. Used for auth tokens, request IDs and tracing info., or as a field in the message body. The pattern is always the same: call Tick() (or Event()) just before sending and put the result in the message; call Recv(…) with the received value before doing anything else with the message. Forget one path (a background job, a retry) and causality silently breaks for that path.

Hybrid logical clocks (HLC)

Why not just pick one?

So far we have two kinds of timestamp, each with a flaw:

A database wants both. It wants to answer “show me the data as of 12:00” (which needs real time) while guaranteeing that a write which caused another write gets a smaller timestamp (which needs causality). In 2014, Sandeep Kulkarni and colleagues published the hybrid logical clockHybrid logical clock (HLC)A timestamp made of a physical-time part (milliseconds) plus a small counter. It stays close to real wall-clock time, never goes backwards, and still respects happened-before like a Lamport clock. Used by CockroachDB, MongoDB and YugabyteDB., which gets both.

The algorithm

An HLC timestamp is a pair (l, c). l is a wall-clock time in milliseconds: the biggest physical time this node has heard of. c is a counter that orders events sharing the same l. Timestamps compare by l first, then c. Each node also reads its physical clock, pt.

• Local event or send: l = max(l, pt). If l didn't change (the physical clock hasn't passed it yet), add 1 to c; otherwise set c = 0.
• Receive a message carrying (ml, mc): l = max(l, ml, pt). Then set c based on which of the three won:
  • l equals both the old l and ml → c = max(c, mc) + 1
  • only the old l → c = c + 1
  • only ml → c = mc + 1
  • pt (the physical clock is ahead of both) → c = 0

Compare it with Lamport's rule, max(mine, received) + 1: the HLC does the same thing, except that the physical clock also takes part in the max, and “+1” happens in the counter instead of in the time. Step through it. Server A's clock is 100 ms fast:

Step 4 is the whole point. B's physical clock said 110, earlier than the send it was receiving (206). A plain timestamp would have recorded an effect before its cause. The HLC took 206 from the message and bumped the counter, so the receive is ordered after the send, and its l is still a real wall-clock reading (A's), not an arbitrary number.

The guarantees

• Respects happened-before: if a → b, then HLC(a) < HLC(b), like a Lamport clock.
• Never goes backwards, even if NTP steps the physical clock back: l only grows.
• Stays close to real time: l is always at least the node's own physical clock, and at most the fastest clock in the cluster. So it's never further from physical time than the worst clock skewClock skewThe difference between two machines' clocks. Even with time sync (NTP) it is often milliseconds, and can be much more after a fault, so timestamps from different machines can't reliably order events..
• Small: the counter only grows while the physical clock is behind l, so it stays small, and the pair fits in 64 bits (the original paper used 48 bits of time and 16 bits of counter). Our Go struct below is bigger, for readability; real systems pack it.

In Go

The physical clock is passed in as a function, so tests can control it (§11 explains why):

package clocks

import (
	"fmt"
	"sync"
	"time"
)

// Timestamp is an HLC reading: a wall-clock part plus a tie-breaking counter.
type Timestamp struct {
	Wall    int64  // milliseconds since the Unix epoch (close to real time)
	Logical uint32 // counts events that share the same Wall value
}

func (a Timestamp) Less(b Timestamp) bool {
	return a.Wall < b.Wall || (a.Wall == b.Wall && a.Logical < b.Logical)
}

type HLC struct {
	mu        sync.Mutex
	now       func() int64  // physical clock, injectable for tests
	maxOffset time.Duration // reject peers whose clock is further ahead than this
	last      Timestamp
}

func NewHLC(now func() int64, maxOffset time.Duration) *HLC {
	return &HLC{now: now, maxOffset: maxOffset}
}

// Now: for a local event or a send.
func (h *HLC) Now() Timestamp {
	h.mu.Lock()
	defer h.mu.Unlock()
	pt := h.now()
	if pt > h.last.Wall {
		h.last = Timestamp{Wall: pt} // physical clock moved on: use it, counter back to 0
	} else {
		h.last.Logical++ // clock didn't move (or went back): stay put, count up
	}
	return h.last
}

// Update: for a receive. Never moves backwards, never falls behind the sender.
func (h *HLC) Update(remote Timestamp) (Timestamp, error) {
	h.mu.Lock()
	defer h.mu.Unlock()
	pt := h.now()
	if d := time.Duration(remote.Wall-pt) * time.Millisecond; d > h.maxOffset {
		return Timestamp{}, fmt.Errorf("remote clock %v ahead of ours, max is %v", d, h.maxOffset)
	}
	old := h.last
	wall := max(old.Wall, remote.Wall, pt)
	var logical uint32
	switch {
	case wall == old.Wall && wall == remote.Wall:
		logical = max(old.Logical, remote.Logical) + 1
	case wall == old.Wall:
		logical = old.Logical + 1
	case wall == remote.Wall:
		logical = remote.Logical + 1
	default: // our physical clock is ahead of both: counter resets
		logical = 0
	}
	h.last = Timestamp{Wall: wall, Logical: logical}
	return h.last, nil
}

And a test with a fake physical clock, covering step 4 of the stepper, a clock stepped backwards, and a peer whose clock is far ahead:

package clocks

import (
	"testing"
	"time"
)

// fakeClock is a physical clock the test controls.
type fakeClock struct{ ms int64 }

func (f *fakeClock) now() int64 { return f.ms }

func TestHLCFollowsSenderWithSlowClock(t *testing.T) {
	aClock, bClock := &fakeClock{ms: 206}, &fakeClock{ms: 106} // B is ~100 ms behind
	a := NewHLC(aClock.now, 500*time.Millisecond)
	b := NewHLC(bClock.now, 500*time.Millisecond)

	sent := a.Now() // {206 0}
	bClock.ms = 110
	got, err := b.Update(sent)
	if err != nil {
		t.Fatal(err)
	}
	if got != (Timestamp{Wall: 206, Logical: 1}) || !sent.Less(got) {
		t.Fatalf("got %v, want {206 1}, after %v", got, sent)
	}
}

func TestHLCNeverGoesBackwards(t *testing.T) {
	clk := &fakeClock{ms: 1000}
	h := NewHLC(clk.now, time.Second)
	first := h.Now()
	clk.ms = 400 // NTP stepped the clock back 600 ms
	if second := h.Now(); !first.Less(second) {
		t.Fatalf("%v is not after %v", second, first)
	}
}

func TestHLCRejectsFarFutureClock(t *testing.T) {
	h := NewHLC((&fakeClock{ms: 1000}).now, 500*time.Millisecond)
	if _, err := h.Update(Timestamp{Wall: 5000}); err == nil {
		t.Fatal("expected an error for a peer 4 s ahead")
	}
}

aClock.now (without parentheses) is a method value: the method bound to that particular aClock, passed around as a plain func() int64. All three tests pass with go test -race.

Who uses it

• CockroachDBCockroachDB / TiKVDistributed SQL / key-value databases that split data into many small ranges and run one Raft group per range, so each range has its own leader and replicated log. stamps every transaction with an HLC timestamp. It also enforces a maximum clock offsetMaximum clock offsetA configured promise that no two machines' clocks differ by more than this (CockroachDB has long defaulted to 500 ms). Correctness depends on it, so nodes check it and shut down rather than break it. (500 ms by default for many years; it's a start-up flag, so check your version's docs): a node whose clock is too far from the others shuts itself down rather than risk wrong answers. The maxOffset check above is a tiny version of the same idea.
• MongoDBMongoDBA popular document database: it stores JSON-like documents in collections instead of rows in tables. uses a hybrid logical clock for its “cluster time” (since version 3.6), which powers causally consistent sessions: a client that wrote something is guaranteed to read it back, even from another replica (when it uses “majority” read and write concern, MongoDB's settings for how many replicas must confirm).
• YugabyteDB uses HLC timestamps for its transactions, similar to CockroachDB.

See the gap, and the cheapest fix when the client is involved: let the client carry the timestamp from one request to the next.

TrueTime and Spanner: waiting out the uncertainty

An honest clock

Every clock reading is really “12:00:00.205, give or take a few milliseconds”. Normal APIs drop the “give or take”. TrueTimeTrueTimeGoogle's clock API used by Spanner. Instead of one time, it returns an interval [earliest, latest] that is guaranteed to contain the true time. GPS receivers and atomic clocks in each data center keep the interval only a few milliseconds wide., the clock API inside Google's SpannerSpannerGoogle's globally distributed SQL database (also sold as Cloud Spanner). It uses TrueTime to give transactions timestamps that match real time order across the whole planet. database, keeps it:

TT.now()  →  [earliest, latest]   // the true time is guaranteed to be in here

The half-width of the interval is called ε (epsilon)ε (TrueTime uncertainty)How wrong TrueTime's clock might be: half the width of the [earliest, latest] interval. If ε = 4 ms, the true time is within 4 ms either side of the reading., the clock uncertainty. To keep ε small, every Google data center has time servers with GPS receivers and atomic clocksAtomic clockA clock that counts the vibrations of atoms (caesium or rubidium). Extremely stable — off by far less than a microsecond per day. GPS satellites carry them, which is why a GPS receiver is also a very good time source. (two kinds, because they fail for different reasons). Every machine polls them about every 30 seconds. Between polls, the machine assumes its own clock may drift at a pessimistic worst-case rate (200 µs per second in the 2012 paper), so ε grows until the next poll and then drops again. The paper reported ε of roughly 1–7 ms; Google has since said it's usually much lower.

The problem it solves

Spanner promises that if transaction T1 finishes before T2 starts (in real time, anywhere on Earth), then T1 gets a smaller commit timestamp. That matters because Spanner lets you read “the database as of timestamp t”: if T2's timestamp were smaller, a read could show T2's effects without T1's, even though T2 happened afterwards. Google calls this external consistency; elsewhere it's called strict serializabilityStrict serializabilitySerializable transactions (as if run one at a time) plus linearizability (that one-at-a-time order respects real time). What Spanner and CockroachDB-style databases aim for., the transaction version of linearizabilityLinearizabilityThe strongest consistency guarantee: the system behaves as if there were one copy and every operation happened at a single instant. Once any read sees a new value, no later read sees the old one. (the CAP topic explains linearizability and puts both on its ladder of consistency models).

Notice that an HLC can't promise this (§09): if T1's client tells a friend by phone, and the friend starts T2 on another server, no message connected the two servers. Only real time did.

Commit wait

Spanner's rule, called commit waitCommit waitSpanner's trick: after choosing a commit timestamp, wait until TrueTime says that timestamp is definitely in the past before telling anyone the transaction committed. The wait is about twice the clock uncertainty., is short:

1. When T1 is ready to commit, choose its timestamp s = TT.now().latest: a time that's surely not in the past yet.
2. Wait until TT.now().earliest > s: until s is surely in the past in true time. From then on, every server's TT.now().latest (which is never below the true time) is already beyond s.
3. Only then tell the client “committed”.

Anyone who hears about T1 after step 3, and then starts T2 anywhere, gets TT.now().latest as T2's timestamp, and that is already past s. So T2's timestamp is bigger. The wait is about 2ε: ε for latest to be reached, plus ε until even earliest passes it. Try it with worst-case clocks (server 1 is ε fast, server 2 is ε slow, both within their promise):

Without the wait, a 4 ms uncertainty is enough to give T2 a smaller timestamp. With it, every commit costs about 2ε of extra latency, and Spanner overlaps that wait with other work it has to do anyway (like replicating the commit, which takes a network round trip).

Without atomic clocks: CockroachDB's uncertainty intervals

CockroachDBCockroachDB / TiKVDistributed SQL / key-value databases that split data into many small ranges and run one Raft group per range, so each range has its own leader and replicated log. runs on ordinary servers, so its uncertainty is the configured maximum clock offsetMaximum clock offsetA configured promise that no two machines' clocks differ by more than this (CockroachDB has long defaulted to 500 ms). Correctness depends on it, so nodes check it and shut down rather than break it.: 500 ms by default for many years (it's a start-up flag, so check your version's docs). Waiting 2 × 500 ms on every commit would be far too slow. So it moves the problem to reads:

• A transaction reads at its HLC timestamp t (§09).
• If it finds a value written at a timestamp slightly above t, between t and t + max offset, it can't tell whether that write really happened before or after it started: the writer's clock might simply have been ahead. That window is the uncertainty intervalUncertainty intervalThe window of times in which you can't tell whether another machine's event was before or after yours, because the clocks may differ by up to the maximum offset..
• Instead of guessing, the reader restarts at a higher timestamp that includes the write. Values written beyond t + max offset are definitely later and are ignored.

Drag the write along the timeline:

So the cost is paid only when reads and writes on the same data overlap in time, instead of on every commit. The trade-off: CockroachDB guarantees serializable transactions (the result is as if transactions ran one at a time) and no stale reads of a single key, but it doesn't make Spanner's promise for transactions on unrelated keys connected only by real time (like the phone call above). And correctness depends on the max offset really holding, which is why nodes check their offset against each other and shut down when it's exceeded.

Staff-level: time in real systems

Logs and traces across services

Logs from one machine, sorted by timestamp, are in the right order (as long as the clock wasn't stepped). Logs from many machines, sorted by timestamp, are only roughly right: events less than the clock skew apart can appear swapped, exactly like §01.

• Follow the request, not the clock. Give every incoming request an ID and pass it to every service it touches (a request ID or trace ID header). To reconstruct what happened, filter by ID, then order by cause: which call made which.
• Distributed traces have the parent built in. Each spanTracing / spanDistributed tracing records the path of one request through your code and services. Each timed step (one middleware, one DB query) is a span, so you can see where the time went. records its parent span. That's happened-before, recorded explicitly. With clock skew you'll still see oddities like a child span that starts before its parent; that's the skew between the two machines, not a bug in your code. Some trace viewers (Jaeger, for one) can adjust for it.
• Within one process, add a sequence number to each log line if exact order matters; it's cheaper than trusting timestamps at microsecond resolution.

More on logs, metrics and traces in the Observability topic.

Order by sequence numbers, not by time

When one component is in charge of order, let it hand out numbers: a database sequence, an offset within one partitionPartition (Kafka)One ordered slice of a topic. Messages with the same key always land in the same partition, so they stay in order relative to each other. of KafkaKafkaApache Kafka: a durable, ordered log of events, split into partitions, that many programs can publish to and read from at their own pace., a Postgres LSNLSN (log sequence number)Postgres's name for a position in the write-ahead log, written like 0/3000148. It only ever grows, so “has this replica replayed past LSN X?” is a simple comparison.. These are exact and cheap, because a single writer assigns them. The cost is that single writer.

For IDs created on many machines at once, the Sharding topic covers UUIDv7UUIDv7A UUID whose first 48 bits are the current time in milliseconds, followed by random bits. New IDs sort roughly by creation time, which keeps B-tree indexes compact. and Snowflake IDsSnowflake IDA 64-bit ID made of a timestamp, a machine number and a per-millisecond counter, invented at Twitter. Unique without coordination, sortable by time, and half the size of a UUID.. Both start with a timestamp, so they sort roughly by creation time, which is great for database indexes. But they inherit their machine's clock:

• Two IDs created on different machines within the clock skew can sort in the wrong order. Never use them to decide “which write came first” across machines.
• If the clock steps backwards, a generator could produce an ID smaller than its previous one, or even a duplicate. Twitter's original Snowflake generator refused to create IDs until the clock caught up again; check what your library does.

Leases: where a clock bug becomes a correctness bug

A leaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”. is a lock with an expiry: “node 1 is the leader until 10 seconds from now.” Many leader-election and locking systems use them (the Consensus topic shows how leaders are chosen). A lease is only safe if the holder stops acting before anyone else thinks the lease has expired. That depends on:

• Clock rate, not clock value: measure the lease with the monotonic clock (§03), as a duration from when it was granted. Never compare wall-clock times from two machines.
• A safety margin for drift: if clocks can run 200 ppm apart, a 10 s lease can look about 2 ms longer to one side. The holder should give up a little early; the next holder should start a little late.
• Pauses: a garbage-collectionGarbage collector (GC)Part of the language runtime that automatically frees memory nobody can reach anymore. Anything still referenced — like a stuck goroutine — is never freed. pause, a VM migration or a laptop lid can freeze a process for seconds. It wakes up still believing it holds the lease. No clock fixes that, which is why writes should also carry a fencing tokenFencing tokenA number that increases every time a new leader is chosen or a lock/lease is granted. Storage remembers the highest number it has seen and rejects writes carrying an older one, so a paused old holder can't overwrite the new holder's work. that the storage checks (Replication §08 explains fencing). Step through it, then switch fencing off:

UTC hygiene, in brief

• Store and send UTC. In Postgres use timestamptz (it stores an absolute instant); in JSON use RFC 3339RFC 3339 / ISO 8601 timestampThe standard text format for a moment in time, like 2026-10-30T12:00:00Z. The Z means UTC; otherwise an offset such as +02:00 is written at the end. strings with a Z or an explicit offset (2026-10-30T12:00:00Z). Go's time.Time and JavaScript's toISOString() both produce this.
• Convert to local time only for display, using the user's time zone name (Europe/Berlin), not a fixed offset. Offsets change with daylight saving.
• Schedule “every day at 09:00 local” in local time, with a time-zone database, not as “every 24 hours”: some days are 23 or 25 hours long.

Testing code that uses time

Code that calls time.Now() or Date.now() directly can't be tested quickly: to test a 10-minute expiry you'd wait 10 minutes. Instead, inject the clock, as the HLC in §09 does with its now field. That's dependency injection applied to time, and in tests you pass a fake clockFake clockA test double for time: code asks it for “now” instead of the real clock, and the test moves it forward (or backward) by hand. Makes timeouts and expiry testable instantly and deterministically. that you move by hand.

package lease

import (
	"testing"
	"testing/synctest" // stable since Go 1.25
	"time"
)

func TestFakeTime(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		start := time.Now()
		time.Sleep(time.Hour) // returns instantly: the fake clock jumps ahead
		if got := time.Since(start); got != time.Hour {
			t.Fatalf("got %v", got)
		}
	})
}

Both ran and passed (the Go one on Go 1.26). Go's testing/synctest package (stable since Go 1.25; Go 1.24 had an experimental synctest.Run behind GOEXPERIMENT=synctest) runs the function in a “bubble” where time.Now, time.Sleep and timers use a fake clock that jumps forward whenever every goroutine in the bubble is waiting. Node's mock.timers (still marked experimental in Node 22, so it prints a warning) fakes Date and, optionally, setTimeout. The simplest option of all works in both languages: pass a now function into your code and let the test supply one that returns whatever it wants.

Pitfalls gallery

const start = Date.now();
await callUpstream();
const ms = Date.now() - start; // can be negative after a step

const start = performance.now();
await callUpstream();
const ms = performance.now() - start; // monotonic

Check yourself

Predict the answer before you click. Each explanation points to the section that covers it.

start := time.Now()
doWork() // takes 1 s; meanwhile NTP steps the clock back 2 s
elapsed := time.Now().Sub(start)
fmt.Println(elapsed.Round(time.Second))

Cheat sheet