Design a Chat System

The 30-second version

That sentence is the whole design. The rest of this page explains why each part is there and what goes wrong without it.

The whole system on one picture

Phones keep a long-lived WebSocketWebSocketA long-lived two-way connection between browser and server, used for chat and live updates. It starts as a normal HTTP request, then the server takes over (“hijacks”) the raw connection. connection open to a gatewayGateway (connection server)A server whose main job is to hold users' open WebSocket connections and pass messages between those sockets and the rest of the backend. It knows who is connected to it, but keeps no chat history., a server whose job is just to hold those connections. When Alice sends a message, the chat service numbers it, writes it to the message store, and looks up which gateway Bob is connected to. If Bob isn't connected at all, it sends a Push notificationPush notificationA message shown on a phone or browser even when the app isn't open. Your server can't reach the device directly; it hands the message to Apple's APNs, Google's FCM or a browser vendor's push service, which delivers it. instead.

What you'll learn

The page follows the usual system design arc: requirements (§02), a rough capacity estimate (§03), the design piece by piece (§04–§09), what breaks as traffic grows (§10), what happens when parts die (§11), and what to monitor (§12).

Requirements: what are we building?

“Build a chat app” could mean a two-person messenger or a 100,000-person live chat during a football match. Those need very different designs, so the first step is to pin down the scope. Here is the one we'll design.

Functional: what users can do

Non-functional: how well it must work

Out of scope (say it out loud)

• Huge rooms and broadcast channels with tens of thousands of members (large groupsLarge group / channelA conversation with thousands of members. Pushing every message to every member's devices is too expensive, so such rooms are usually read on demand instead (fan-out on read).). They need a different delivery strategy, sketched in §10.
• Voice and video calls. They use different protocols that stream audio and video live.
• Images and files. In a typical design these go to Object storageObject storageA cloud service that stores files ("objects") by key in buckets over HTTP, cheaply and almost without limit (Amazon S3, Google Cloud Storage). Not a disk you mount. and the message just carries a link. Design File Sync covers uploads.
• End-to-end encryptionEnd-to-end encryption (E2EE)Messages are encrypted on the sender's device and only decrypted on the recipients' devices. The server stores and forwards ciphertext it cannot read. It changes many design details, so it is out of scope here.. It changes how keys, devices and history work, and deserves its own page. The delivery machinery on this page still applies: the server just moves bytes it can't read.
• Search, spam filtering and moderation. They matter in real products, but they sit beside the core path rather than on it.

How big is it? A rough estimate

Every number below is a guess with round numbers, labelled approximate. The goal is to be within about 10× of reality, which is enough to tell “one server” from “a thousand servers”.

Step 1: messages per second

Assume 100 million daily active users (people who open the app at least once a day), each sending 40 messages a day.

• Messages per day: 100,000,000 × 40 = 4 billion.
• A day has 86,400 seconds; round it to 100,000 to make the maths easy.
• Average rate: 4,000,000,000 ÷ 100,000 = 40,000 messages per second.
• People chat in the evening, not at 4 a.m. Assume the busiest hour is 3× the average (the peak factorPeak factorHow much busier the busiest hour is than the daily average. Chat traffic follows people's waking hours, so a factor of 2–3× is a common planning guess.): 40,000 × 3 = 120,000 messages per second at peak.

Each message is delivered to more than one device: the recipient's phone and laptop, the sender's other devices, several people in a group. Assume 3 deliveries per message on average: 120,000 × 3 = 360,000 deliveries per second at peak. That's the outgoing load, measured in the same units as QPSQPS (queries per second)How many requests a system handles each second, the basic unit of load in capacity planning. 1 million requests a day is only about 12 QPS on average, but traffic is never spread evenly, so designs plan for the peak..

Step 2: open connections and gateways

Not everyone is online at once. Assume 20% of daily users have the app open at the busiest moment: 100,000,000 × 0.2 = 20 million concurrent connectionsConcurrent connectionsHow many devices hold an open connection at the same moment. It is always much smaller than the number of daily users, because most people are not online all day..

How many fit on one server? The WebSockets scaling section estimates very roughly 20–100 KB of memory per idle connection (measure your own). Take 100,000 connections per gateway (100,000 × 50 KB ≈ 5 GB of memory, which is comfortable).

• 20,000,000 ÷ 100,000 = 200 gateways if every one were completely full.
• Never run them full: when one dies, its users reconnect to the others, which need room for them. Keep them half full: 200 × 2 = about 400 gateways.

Step 3: the hidden load, heartbeats

Every open connection sends a small HeartbeatHeartbeat (application)A small “are you still there?” message your own protocol sends every few seconds. If a few heartbeats go unanswered, the program treats the other side as gone. now and then, so both sides know it's still alive (§07 explains why). At one heartbeat every 30 seconds:

20,000,000 ÷ 30 ≈ 670,000 heartbeats per second, nearly twice the 360,000 real deliveries. Keeping track of who is online costs more traffic than the chat itself. Remember that number when we get to presence.

Step 4: storage

A text message is about 100 bytes. Add IDs, the sequence number, timestamps and index overhead, and call it 200 bytes stored per message.

• Per day: 4,000,000,000 × 200 B = 800 GB.
• Per year: 800 GB × 365 ≈ 290 TB.
• Stored as 3 copies (so a dead disk loses nothing): 290 TB × 3 ≈ 880 TB a year, close to a petabyte, and growing every year you keep history.

Outgoing network traffic, by comparison, is small: 360,000 deliveries × ~500 bytes on the wire ≈ 180 MB/s at peak, about 1.4 Gbit/s across 400 gateways.

Connections: gateways and finding the right one

The simplest design that works

For the first 10,000 users, one server is enough. Each phone opens a WebSocket to it. When Alice sends a message, the server saves it in Postgres, finds Bob's socket in its own memory, and writes the message into it. If Bob isn't connected, the message waits in the database until he is.

This is a fine design, and you should start here. It has two problems we'll fix in this section: it's a single point of failureSingle point of failure (SPOF)One component that, if it breaks, takes the whole system down with it — because everything depends on it and nothing can take its place., and it can't grow beyond one machine.

Why WebSockets

A server can't send anything over plain HTTP until the client asks. Chat needs the opposite: Bob's phone must hear about Alice's message the moment it exists. A WebSocketWebSocketA long-lived two-way connection between browser and server, used for chat and live updates. It starts as a normal HTTP request, then the server takes over (“hijacks”) the raw connection. keeps one connection open in both directions (Full duplexFull duplexBoth sides can send at the same time, independently — like a phone call. Half duplex is a walkie-talkie: one talks while the other listens. HTTP request/response is closer to taking turns.), so either side can send at any time. The WebSockets topic explains the protocol, and its comparison section shows why chat, where both sides talk often, is the classic case for it over server-sent events or long polling.

Gateways: servers that hold connections

Once there is more than one server, split them into two kinds:

• Gateways hold WebSocket connections. A gateway checks the user's login once when the connection opens, then passes messages between sockets and the services behind it. It knows who is connected to it, and nothing about chats.
• Chat service instances do the real work: number each message, store it, and decide who receives it. They keep nothing in their own memory between requests, so any instance can handle any message.

A gateway is a stateful serverStateful serverA server that holds something in its own memory that other servers don't have — here, live connections. If it dies, that state is gone, and the clients attached to it must reconnect somewhere else.: it holds live connections no other server has, and when it dies those connections die with it. Keeping gateways small and simple pays off. They rarely need new releases, so they are rarely restarted, and a restart is what disconnects their users. Chat logic changes every week, but it lives in the stateless services, which can be redeployed freely.

The message is on the wrong gateway

With 400 gateways, Alice and Bob are almost never on the same one. Try three strategies for getting the message across:

The usual answer is a session registrySession registryA shared lookup table (often in Redis) that answers “which gateway is user X's phone connected to right now?”. Gateways add an entry when a device connects and remove it when the device leaves; entries also expire on their own.: a shared table, often in RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages)., from user to the gateways holding that user's devices.

sessions:bob  →  { phone-1: gw-17,  laptop-2: gw-203 }     expires 90 s after the last heartbeat

• When a device connects, its gateway adds an entry. When it disconnects cleanly, the gateway removes it.
• When a gateway crashes, it can't clean up. So entries expire on their own unless the device's heartbeats keep refreshing them.
• One user can have several entries: one per device. Delivering to Bob means delivering to every device listed, which is how multi-device works.

The other common design is pub/sub, from WebSockets → scaling: when Bob connects, his gateway subscribes to a topic like user:bob on a Message bus / brokerMessage bus / brokerA separate server (Kafka, RabbitMQ, NATS, Google Pub/Sub, SNS/SQS) that stores messages from publishers and delivers them to subscribers in other programs — the Observer pattern across a network., and the chat service publishes Bob's messages there. The broker then does the “where is Bob?” lookup. Either way, the routing state is a cache that can be rebuilt: every gateway knows exactly which devices it holds and can re-register them all after a registry failure (§11).

Load-balancing connections that last for hours

A Load balancerLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped. in front of the gateways picks one gateway per connection, not per message. Usually it's a layer-4Layer 4 (L4) load balancerA load balancer that works at the TCP/UDP level. It only sees IP addresses and ports, picks a backend once per connection, and copies bytes back and forth without understanding them. Examples: AWS NLB, Linux IPVS. balancer that just forwards TCP. Two consequences, both covered in Load balancing → long-lived connections:

• New gateways start empty. After you add servers, existing connections stay where they are, so old gateways stay full and new ones idle. Balancers send new connections to the gateway with the fewest open connections to even this out over time.
• Deploys must be gentle. Restarting a gateway drops 100,000 connections. DrainConnection drainingTaking a backend out of rotation gently: the load balancer stops giving it new requests but lets the ones already running finish (up to a time limit) before the instance is stopped. it instead: stop accepting new connections, then close the existing ones at random moments over half a minute. Clients reconnect with random delays (WebSockets → reconnecting) and land on other gateways. Without this, you get a Thundering herdThundering herdMany clients hitting a server at the same moment — e.g. all retrying together right after an outage — which knocks it over again. of 100,000 reconnects in the same second.

You don't need sticky sessionsSticky session (session affinity)Sending all requests from the same user to the same backend, usually by setting a cookie that names the backend or by hashing the client's IP address. for WebSockets: a connection is already pinned to one gateway for its whole life, and the registry handles everything that crosses gateways.

The life of one message

Step through it, then switch scenarios. Each failure scenario breaks a different step, and the design recovers in a different way:

The four rules behind the stepper

1. The client names every message. Before sending, the app creates a client message IDClient message IDA random ID (usually a UUID) the sending app creates for each message before it sends it. If the app has to retry, it sends the same ID, so the server can tell “new message” from “the same message again”., a random UUIDUUID (v4 / v7)A 128-bit ID like 3f2b…-…. Version 4 is fully random; version 7 starts with a timestamp, so new IDs are roughly in increasing order.. If the connection drops and the app isn't sure the message arrived, it sends it again with the same ID. The server remembers which IDs it has stored and treats a repeat as “already done”. This is an idempotency key, the technique explained in Idempotency → keys, applied to chat.

2. The server numbers every message. Each conversation has its own counter. The first message is #1, the next #2, and so on. This sequence numberSequence numberA number that goes up by one with every message in a stream (1, 2, 3…). A client that remembers the last number it saw can spot gaps and duplicates and ask “send me everything after 41”. decides the order everyone sees, and it lets devices notice anything they missed (§06). The client can't choose it: two phones would pick the same number.

3. Store first, then ack. The ackAcknowledgement (ack)A “got it, done” message from a receiver back to the sender (or to a message broker). If no ack arrives in time, the sender assumes the message was lost and sends it again. (“stored as #42”) is a promise: this message will never be lost. So it's sent only after the write is durableDurabilityThe promise that once the database says “committed”, the data survives a crash or power cut — usually because it was written to disk (and fsynced) before replying.: on disk, and on more than one machine, so that one dead server can't take it away (§11 shows what goes wrong otherwise). Ack first and crash a moment later, and Alice sees ✓ for a message that no longer exists.

4. Deliver after storing, and don't make the sender wait for it. Once the message is stored, the chat service sends it to every device that should get it: all of Bob's devices, and Alice's other devices, so her laptop shows what she typed on her phone. Devices with no connection get a push notification instead. Alice doesn't wait for any of this: Bob's phone might be switched off for a week.

There's a gap between rules 3 and 4: what if the chat service stores #42 and crashes before delivering it? Alice's retry is fine (she gets her ack), but nothing on Bob's side failed, so nothing makes his app ask for #42, and an offline Bob gets no notification. The fix is an Outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does.: in the same database transaction as the message, write a row saying “deliver #42”. A worker picks up unfinished rows and does the delivery and the push, retrying until it's done. Try the “chat service crashes” scenario above. The Queues topic explains why such a worker must expect to repeat work, and apps also run a slow background sync every few minutes as a last line of defence (§06).

The send path in code

Here is the heart of the chat service: number and store a message, at most once per client message ID. A map and a lock stand in for the database. (New to Go? The 90-second primer is in Async / Await §02.)

package main

import (
	"fmt"
	"sync"
)

type Message struct {
	Seq      int64
	Sender   string
	ClientID string
	Text     string
}

type ChatService struct {
	mu      sync.Mutex
	lastSeq map[string]int64     // conversation → last sequence number handed out
	seen    map[string]int64     // "sender/clientID" → seq it got the first time
	log     map[string][]Message // conversation → messages, in seq order
}

// Send stores a message once, however many times the client retries it.
func (s *ChatService) Send(conv, sender, clientID, text string) (seq int64, dup bool) {
	s.mu.Lock() // one sender at a time (a DB transaction in real life)
	defer s.mu.Unlock()

	key := sender + "/" + clientID
	if seq, ok := s.seen[key]; ok {
		return seq, true // ← a retry: same answer, no second message
	}
	s.lastSeq[conv]++ // ← the server, not the client, decides the order
	seq = s.lastSeq[conv]
	s.log[conv] = append(s.log[conv], Message{seq, sender, clientID, text})
	s.seen[key] = seq
	return seq, false // only now is it safe to ack: the message is stored
}

func main() {
	chat := &ChatService{lastSeq: map[string]int64{}, seen: map[string]int64{}, log: map[string][]Message{}}
	fmt.Println(chat.Send("alice:bob", "alice", "c-1", "Lunch?"))
	fmt.Println(chat.Send("alice:bob", "alice", "c-2", "Pizza?"))
	fmt.Println(chat.Send("alice:bob", "alice", "c-2", "Pizza?")) // ack was lost → retry
	fmt.Println(chat.Send("alice:bob", "bob", "c-1", "Yes!"))     // Bob's own c-1 is a different key
	fmt.Println("stored:", len(chat.log["alice:bob"]), "messages")
}

Real output from Go 1.24 and Node 22:

1 false
2 false
2 true
3 false
stored: 3 messages

The retry of c-2 gets seq 2 again, flagged as a duplicate, and nothing new is stored. Bob's c-1 is stored normally, because the key includes the sender: two phones may well pick the same ID.

The same thing in a real database

In production, the counter and the messages live in the database, and the numbering and the duplicate check happen in one statement. Here is a Postgres version (the table design is in §09):

WITH next AS (
  UPDATE conversations SET last_seq = last_seq + 1   -- locks this row: other senders wait here
  WHERE id = 7
  RETURNING last_seq
)
INSERT INTO messages (conversation_id, seq, sender_id, client_msg_id, body)
SELECT 7, last_seq, 1, 'a1b2c3d4-0000-4000-8000-000000000001', 'Lunch?' FROM next
RETURNING seq;

Running it twice against Postgres 16 (real output, trimmed):

 seq
-----
   1
(1 row)

ERROR:  duplicate key value violates unique constraint "messages_conversation_id_sender_id_client_msg_id_key"

The WITH … AS (…) part runs first, bumps the counter and passes the new number to the INSERT. The second run breaks a UNIQUE (conversation_id, sender_id, client_msg_id) rule on the table, so Postgres rejects the whole statement, including the counter bump. last_seq stays 1, so the retry leaves no hole in the numbering. The app catches that error, reads the existing row's seq (in a new transaction, or after rolling back to a SAVEPOINT, because Postgres refuses further statements in a transaction that hit an error), and returns it as the ack.

Why not ON CONFLICT DO NOTHING? Because then the statement succeeds: the counter bump is committed while the insert is skipped, leaving a hole at #2 that every device would wait for. Letting the error abort the whole statement is what keeps the numbers gap-free. Atomic statements and transactions are covered in Transactions.

What the devices can rely on

The network between the server and a phone can still drop or repeat things, so the server delivers each message at least once (At-least-once deliveryAt-least-once deliveryA delivery promise where a message is never lost but may arrive more than once (for example, re-sent after a crash). Receivers must therefore handle duplicates safely.): if it isn't sure a device got it, it sends it again. The device drops any number it already has. Together that looks like exactly-once to the user. It's the same pattern as Idempotency → delivery guarantees, with the sequence number doing the DeduplicationDeduplicationRecognising that a message or request was already handled (usually by an ID you remember) and skipping it the second time..

Ordering: the order people expect

Why timestamps fail

The obvious design is to stamp each message with the time it was sent and sort by that. Try it with a phone whose clock is wrong:

Every computer's clock drifts, and phones can be set by hand. The Clocks topic explains why no two clocks agree. That difference between clocks is called Clock skewClock skewThe difference between two machines' clocks. Even with time sync (NTP) it is often milliseconds, and can be much more after a fault, so timestamps from different machines can't reliably order events., and the time a device believes it is, is its wall-clock timeWall-clock timeThe time of day your computer believes it is (like 2026-10-30 12:00:01). It can jump backwards or forwards when it's corrected, so it's unreliable for measuring durations or ordering events across machines..

Using the server's clock instead is better, but still not enough:

• Many servers, many clocks. Alice's message is stamped by one chat service instance, Bob's reply by another. If their clocks differ by 20 ms and the reply came 5 ms later, the reply sorts first.
• Ties. Two messages in the same millisecond have no defined order.
• Gaps are invisible. If you have messages stamped 12:00:01 and 12:00:09, was there one at 12:00:05? A timestamp can't tell you. A number can: if you have #41 and #43, you are missing #42.

One counter per conversation

So each conversation gets its own counter, and the chat service hands out the next number as it stores each message (§05). Everyone in the conversation sees messages in number order. That's a Total orderTotal orderAn ordering where every pair of items can be compared, so there is always a definite first and second — like numbering every message in a chat 1, 2, 3. A partial order leaves some pairs with no defined order. inside the conversation: any two messages can be compared, and everyone agrees which came first.

We don't order messages across different conversations, and nobody needs us to. Whether your message to your mum came before or after a message in your work group doesn't matter. Keeping the counter per conversation means no single counter has to handle the whole system's 120,000 messages per second.

Why messages still arrive out of order

A single WebSocket runs over TCP, which delivers bytes in order. So how can Bob's phone ever get #44 before #43?

• Several senders, several paths. Alice's #43 and Carol's #44 are stored by two different chat service instances, which then deliver them in parallel. If the instance holding #43 is a few milliseconds slower, #44 reaches Bob's gateway first.
• History and live messages race. When Bob opens a conversation, his app fetches recent history while new messages are already arriving live. The two streams overlap.
• Reconnects. A message sent while Bob's connection was down never reaches him at all. His app only notices when #45 arrives next.

The client must expect three things: messages out of order, messages twice, and messages missing. Here are all three:

The client-side fix in code

The client keeps the last number it has shown with no holes before it, plus a small waiting area for messages that arrived early.

package main

import (
	"fmt"
	"strings"
)

type Msg struct {
	Seq  int64
	Text string
}

type Inbox struct {
	LastSeq int64         // everything up to here is on screen, with no holes
	pending map[int64]Msg // arrived early, waiting for the hole to fill
	Shown   []string
}

func (in *Inbox) Receive(m Msg) string {
	if _, held := in.pending[m.Seq]; m.Seq <= in.LastSeq || held {
		return "duplicate, dropped"
	}
	in.pending[m.Seq] = m
	for { // ← release every message that's now in order
		next, ok := in.pending[in.LastSeq+1]
		if !ok {
			break
		}
		in.LastSeq++
		in.Shown = append(in.Shown, next.Text)
		delete(in.pending, in.LastSeq)
	}
	if len(in.pending) > 0 {
		return fmt.Sprintf("gap: waiting for %d", in.LastSeq+1)
	}
	return "in order"
}

func main() {
	in := &Inbox{pending: map[int64]Msg{}}
	for _, seq := range []int64{1, 2, 4, 3, 3, 6} { // the network's order, not the server's
		fmt.Printf("got %d: %s\n", seq, in.Receive(Msg{seq, fmt.Sprintf("m%d", seq)}))
	}
	fmt.Println("on screen:", strings.Join(in.Shown, " "))
	// On reconnect: ask the server for everything after LastSeq.
	fmt.Printf("sync since %d → server sends 5\n", in.LastSeq)
	fmt.Println("got 5:", in.Receive(Msg{5, "m5"}))
	fmt.Println("on screen:", strings.Join(in.Shown, " "))
}

Both print the same (real output):

got 1: in order
got 2: in order
got 4: gap: waiting for 3
got 3: in order
got 3: duplicate, dropped
got 6: gap: waiting for 5
on screen: m1 m2 m3 m4
sync since 4 → server sends 5
got 5: in order
on screen: m1 m2 m3 m4 m5 m6

Filling gaps: “sync since N”

A gap is usually just a slow message, so the client waits a moment (say a second or two). If the hole is still there, it asks the server: “for conversation 7, send me everything after #4”. The server reads that range from the store, which is cheap because messages are stored sorted by number (§09).

The same request handles every way of missing messages:

• Reconnecting after a dropped connection or a crashed gateway (§05's stepper).
• Opening the app after it was closed, while notifications came in.
• A brand new device, which simply starts from #0 (or from “the last 50 messages”, for a quick start).

The number each device has stored up to is its sync cursorSync cursorThe last sequence number a device has safely stored for a conversation. When the device reconnects, it sends its cursor and the server replies with every message after it.. One mechanism instead of many special cases is a big reason this design is reliable: the gap-filling code runs every time someone opens the app, so it's well tested.

Presence: who's online, approximately

How a server knows you're online

A connected app sends a small HeartbeatHeartbeat (application)A small “are you still there?” message your own protocol sends every few seconds. If a few heartbeats go unanswered, the program treats the other side as gone. every so often, say every 30 seconds. The server remembers the time of the last one. “Online” means “heard from recently”: if nothing arrives for two intervals (60 s), the user is shown as offline.

Why not just watch the connection? Because a connection can die silently. A phone that drives into a tunnel doesn't send a goodbye. Its TCP connection looks open to the server for minutes, until a write fails or a timeout fires (the WebSockets topic shows why). Heartbeats turn “I haven't heard anything” into a decision after a time you choose.

Try both exits. A clean close (the user leaves the app) can send “going offline” first, so it shows at once. A lost signal can't send anything, so the server only notices the silence, and for up to two intervals Alice sees a “ghost”. A shorter interval makes the ghost shorter but multiplies the traffic. At 20 million connections, 30 s means about 670,000 heartbeats per second (§03); 10 s means 2 million.

Storing presence

Presence changes constantly and is worthless after a minute, so it doesn't belong in the message database. A typical design keeps it in memory (or RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages).) as one key per user with a short expiry timeTTL (time to live)An expiry time attached to a piece of data: “delete this row 7 days after it was written.” Caches and DNS use TTLs so stale entries disappear on their own.:

presence:bob  →  "online"     expires in 60 s; every heartbeat resets the timer

Each heartbeat sets the key again with a fresh 60-second expiry. If the heartbeats stop, the key disappears by itself, and a missing key means “offline”. When a key disappears, store “last seen at 12:03” somewhere longer-lived, for the “last seen 5 min ago” label. The same heartbeat can refresh the session registry entry from §04, so one small message does both jobs.

The expensive part: telling everyone

Knowing Bob is online is cheap. Telling the people who care is not. Suppose every user has 500 contacts, and the app shows a green dot next to each.

• Bob comes online → 500 contacts might want to know → up to 500 presence updates.
• Typical users go online and offline a dozen or two times a day (each time they open and close the app).
• 100 million users × 20 changes a day × 500 contacts = 1 trillion potential updates a day, about 10 million per second. That's far more than the 40,000 messages per second we're actually here to deliver.

This is Fan-outFan-outDelivering one message to many recipients — one chat message pushed to all 500 people in the room. again, and the fixes all reduce how many people get told:

• Only tell people who are looking. Push presence only to contacts who are online right now and have the app open. That's usually a small fraction of 500.
• Subscribe to what's on screen. The app says “I'm showing Bob, Carol and Dan” and gets updates only for them. Scroll, and it changes the list.
• Pull instead of push for big lists. For a group of 500, show “23 online”, fetched when the group is opened, instead of 500 live dots.
• Smooth out flapping. Wait a few seconds before announcing “offline”, so a phone switching from Wi-Fi to mobile data doesn't send offline → online to everyone.

Receipts and multiple devices

Three ticks, three moments

The delivered and read receiptsDelivery / read receiptA small message going back to the sender: “delivered” means the message reached the recipient's device; “read” means the recipient opened the conversation and saw it. In apps these show as ticks next to the message. are sent by Bob's app as small messages through Bob's gateway and the chat service, and delivered to Alice's devices the same way a chat message is. They can be lost, duplicated or delayed like any other message, and they're fixed the same way.

Watermarks instead of one receipt per message

Bob comes back from a flight to 200 unread messages across a group. Sending 200 “delivered” receipts and then 200 “read” receipts is 400 messages, each fanned out to every sender's devices. Instead, because the conversation is numbered, Bob's app sends:

{ "type": "read", "conversation": 7, "upTo": 4031 }

That's a watermarkWatermark (receipts)One number that stands for a whole range: “delivered up to seq 42” or “read up to seq 40”. Because messages in a conversation are numbered in order, one watermark replaces a separate receipt for every message.: one number that means “everything up to here”. It only ever moves forward, so receipts that arrive late or twice are harmless: the server keeps the larger value.

Receipts in groups

In a group, each member has their own two watermarks per conversation: “delivered up to” and “read up to”. The server stores them as one small row per member, something like (conversation 7, carol) → delivered 4031, read 4020.

To draw ticks on Alice's message #4025, her app compares 4025 with each member's watermarks: “read by 3 of 5”. In a 500-person group, pushing every member's every watermark change to every sender is the presence fan-out problem again. That's why big groups often only show receipts when you tap a message, fetched on demand.

Many devices, one account

Bob has a phone and a laptop. Each device:

• has its own connection, gateway and registry entry (§04), and gets every message in every conversation, including the ones Bob sent from the other device;
• keeps its own sync cursorSync cursorThe last sequence number a device has safely stored for a conversation. When the device reconnects, it sends its cursor and the server replies with every message after it. per conversation, because the laptop may have been asleep for a week while the phone was in use, so each catches up from its own point (§06);
• shares one read watermark with the others, because the read watermark belongs to Bob, not to a device. When Bob reads on his phone, the server sends “Bob read up to #42” to his laptop too, and the laptop clears its unread badge.

The delivered receipt is sent when the first of Bob's devices gets the message: Alice cares that it reached Bob, not which device it reached.

Storing messages

Start from the access patterns

Before choosing a table, list how the app reads and writes. Each one is an Access patternAccess patternOne specific way your app reads or writes data, written as a sentence — “get a customer's last 10 orders, newest first”. Listing them is the first step of choosing and designing a database.:

1. Append a message to a conversation and give it the next number (§05). About 120,000 per second at peak.
2. Read the newest 50 messages of a conversation, when it's opened.
3. Read everything after #N in a conversation, to fill a gap or sync a device (§06).
4. Scroll back: read the 50 messages before #N.
5. Find a message by (sender, client message ID), to catch retries (§05).

All five are about one conversation at a time, and patterns 2–4 read a range of numbers in it. Nothing asks for “every message sent at 3 p.m. across all chats”.

The table

CREATE TABLE conversations (
  id       bigint PRIMARY KEY,
  last_seq bigint NOT NULL DEFAULT 0          -- the last number handed out
);

CREATE TABLE messages (
  conversation_id bigint      NOT NULL,
  seq             bigint      NOT NULL,
  sender_id       bigint      NOT NULL,
  client_msg_id   uuid        NOT NULL,
  body            text        NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (conversation_id, seq),                    -- ← a conversation's messages sit together, in order
  UNIQUE (conversation_id, sender_id, client_msg_id)     -- ← a retry can't create a second row
);

The primary key (conversation_id, seq) is the important line. The database keeps rows sorted by it, so all of conversation 7's messages sit next to each other in number order. “Everything after #4,000 in conversation 7” is then one short Range scanRange scanReading all keys between two bounds in sorted order — e.g. every event for user 42 from 9:00 to 10:00 — instead of looking up one key.: WHERE conversation_id = 7 AND seq > 4000 ORDER BY seq LIMIT 100.

created_at is still stored, for showing “12:03” next to a bubble. It's just never used for ordering.

Splitting it across machines

At ~880 TB a year (§03), no single database holds this. The data is shardedShardOne piece of a split-up dataset, together with the database server that stores it. “Tenant 42 is on shard 3” means its rows live on that server.: split across many database servers. The Shard keyShard key (partition key)The column whose value decides which shard a row goes to, e.g. tenant_id. Queries that include it can go straight to one shard; queries without it have to ask all of them. is conversation_id, for reasons that fall straight out of the access patterns (the Sharding topic explains the general rule):

• Every read touches one shard. All five patterns name a conversation, so the app knows exactly which shard to ask.
• The counter lives with its messages. The conversation's last_seq row and its messages are on the same shard, so numbering and storing stay one local transaction, exactly the statement from §05.
• Load spreads evenly. There are billions of conversations, most of them small and quiet.

Sharding by user would be wrong here: a 1:1 conversation belongs to two users, so either every message is stored twice, or reading a conversation means asking two shards.

Busy conversations are hot partitions

Sharding by conversation spreads conversations evenly, not traffic. A lively 500-person group might send 50 messages a second, all landing on one shard, all queuing on one last_seq row. That's a Hot partitionHot partitionOne partition (one slice of the data or of a message stream) getting far more traffic than the others, usually because one key — a huge customer, a viral chat, a null key — is very common. The one machine or consumer handling it becomes the bottleneck.: one piece of the data getting far more traffic than the rest (Sharding → hot keys). The usual answers:

• Cap group size. This is one reason chat apps limit groups to a few hundred or thousand members.
• Number in memory. Give one chat service instance ownership of each busy conversation and let it hand out numbers from memory, writing them in batches (§11 covers how to do this safely).
• Split very long histories by time. For a conversation with years of history, make the partition key (conversation_id, month), so no single partition grows forever.

Hot and cold

Messages from today get read constantly. Messages from three years ago are read when someone scrolls a very long way back, or searches. Keeping both on the same fast SSD databases wastes money. The usual answer is hot and cold storageHot vs cold storageHot storage is fast and expensive (memory, SSD databases) and holds data that is read often, like this week's messages. Cold storage is slow and cheap (object storage, compressed files) and holds old data that is rarely read.:

• Hot: recent messages (say the last 90 days) in the sharded database, often with the newest few hundred per busy conversation also cached in memory.
• Cold: older messages compacted into large compressed files, one batch per conversation and time range, in Object storageObject storageA cloud service that stores files ("objects") by key in buckets over HTTP, cheaply and almost without limit (Amazon S3, Google Cloud Storage). Not a disk you mount.. Reading them is slower (tens or hundreds of milliseconds), which is fine for a deep scroll.

The app doesn't need to know. The history API reads hot storage first and, when the user scrolls past the oldest hot message, fetches from cold storage.

What breaks as traffic grows

The steps, in words

10k → 100k users: more than one server. The first step is about staying up more than about capacity: one server is a single point of failure, and every deploy disconnects everyone. Two or three servers fix that, and immediately need a session registrySession registryA shared lookup table (often in Redis) that answers “which gateway is user X's phone connected to right now?”. Gateways add an entry when a device connects and remove it when the device leaves; entries also expire on their own. (§04). The cost: a new component sitting on every message's path, which must be fast and replicated.

100k → 1M: split by job. Gateways (hold sockets), chat service (order, store, route), presence and push become separate fleets, because they scale with different things. Gateways scale with connections, the chat service with messages per second, presence with connections × contacts. Push and receipts move behind a queue, so a slow Apple or Google push service can't hold up sending. The cost: more deployables, and a queue to watch for growing backlog.

1M → 10M: the database. About 12,000 writes per second at peak and tens of terabytes a year outgrow one primary database. Shard by conversation (§09). The cost: no more queries across all conversations without a separate system for them (search, analytics), and moving shards around as they grow (the Sharding topic covers resharding).

10M → 100M: everything is a fleet. Hundreds of gateways, a sharded registry and presence service, cold storage for old messages. At this size, rare events become hourly ones: a gateway dies every day, a shard fails over every week. Recovery has to be automatic (§11).

Large groups: stop pushing

Our design pushes every message to every member's devices. That's fan-out on write: the work happens when the message is sent. For a 5-member group it's 5 deliveries. For a 50,000-member community channel, one message means 50,000 deliveries, most to people who will never scroll that far.

Large rooms flip it to fan-out on read: store the message once, push only a tiny “channel 9 has new messages up to #88,120” hint to members who have that channel open, and let each client fetch the messages when it actually displays the channel. Unread counts are computed from the member's read watermark (§08) and the channel's latest number. Same storage, same sequence numbers, different delivery. Design a News Feed explores this trade-off in depth. Apps often show the switch openly: big channels have no per-message read receipts and no per-member presence.

When things die

A gateway dies

100,000 connections vanish at once. Each client notices (the socket closes, or heartbeats stop being answered) and reconnects. Two things make this safe:

• The gateway held nothing unique. Every message it was carrying is in the store, so reconnecting and syncing since the last number recovers everything (§05's “gateway crashes” scenario).
• Spare capacity and random delays. The other gateways need room for 100,000 extra connections; that's why §03 kept them half full. Reconnects are spread over several seconds with JitterJitterA small random amount added to a delay or timer — a retry wait, a reconnect, a connection's maximum age — so that many clients don't all act at the same instant and stampede a server. (random delays), so they don't all hit at once as a Thundering herdThundering herdMany clients hitting a server at the same moment — e.g. all retrying together right after an outage — which knocks it over again.. Each reconnect also costs a TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it. handshake and a login check, so a burst of reconnects is CPU-heavy for the gateways and the auth service.

A send is duplicated, or the ack is lost

These are the same problem seen from two sides. The sender can't tell “never arrived” from “arrived, but the answer got lost”, so it retries, and the server uses the client message ID to make the retry harmless (§05). The one rule that must hold: the duplicate check and the store happen together, in one database statement or transaction. Check first and store later, and two retries arriving at the same moment can both pass the check.

The chat service dies between storing and delivering

The sender is covered: no ack, so it retries, and the dedup check returns the stored #42. The recipients are not: nothing failed on their side, so nothing makes them ask. That's why §05 writes an Outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does. row (“deliver #42”) in the same transaction as the message. A worker retries unfinished outbox rows until delivery and push are done, so a crash can delay delivery but not skip it. This is the general outbox pattern from Sagas → messaging, and the “chat service crashes” scenario in §05's stepper walks through it.

A database primary fails over

When the database server holding a shard dies, a ReplicaReplicaA full copy of a database kept on another machine and updated as the original changes, for reading load and for taking over if the original dies. is promoted to take its place (FailoverFailoverSwitching to a standby (a replica) when the primary server fails, so the system keeps working — automatically or by an operator.). Whether that loses messages depends on when the ack was sent:

With asynchronous replication, an acked message can exist only on the machine that just died. Worse, the promoted replica doesn't know its number was used, so it hands #43 out again, and devices that already have the old #43 never fetch the new one. The fix is to ack only after the write is on a second machine: Synchronous replicationSynchronous replicationThe primary database waits until a replica (a copy on another machine) confirms it has a write before telling the app “committed”. Slower by one network round trip, but a committed write survives the primary dying. With asynchronous replication, the default in many databases, the primary answers first and copies a moment later., or a QuorumQuorumA majority (or another agreed minimum) of replicas that must answer before a read or write counts — e.g. 2 of 3. Any two majorities overlap, so a read quorum always includes at least one replica that saw the latest write quorum. write (2 of 3 copies). It costs a network round trip on every send, which is cheap next to the promise it protects. Also fenceFencingMaking sure an old leader can no longer write before (or while) a new one takes over: powering it off, cutting its access, or having storage reject its requests. the old primary so it can't come back and keep accepting writes. The Replication topic covers both modes in depth.

The registry or presence dies

Both are caches that can be rebuilt. Gateways know which devices they hold and re-register them; heartbeats refill presence within one interval. While they're down, the system degrades instead of failing (Graceful degradationGraceful degradationDesigning a system so that when a part fails, it keeps working with less — showing a default list instead of personalized recommendations — instead of failing completely.): messages are still stored, and arrive at the next sync instead of instantly; green dots disappear. Clients also run a slow background “sync since N” every few minutes, so even a long routing outage delays messages instead of hiding them.

Split brain on numbering

The worst failure is quiet: two sequencersSequencerWhatever hands out sequence numbers for a conversation: 1, 2, 3… It must give each number exactly once, so two servers must never both be allowed to number the same conversation. both believe they own the numbering of conversation 7. Say instance A hands out numbers from memory (the hot-conversation trick from §09), freezes for 30 seconds in a long pause, and the system gives conversation 7 to instance B. A wakes up, unaware, and both hand out #43 for different messages. That's Split brainSplit brainTwo nodes both believe they are the leader and both accept writes, so the data forks into two histories. Usually caused by a failover while the old leader was still alive.: two leaders, two histories. Devices that got A's #43 never ask for B's, because they think they already have #43.

The Replication topic describes the same danger for database leaders. The fixes are the same:

• Let the database referee. PRIMARY KEY (conversation_id, seq) means a second #43 can't be stored: the losing write fails and its sender retries, getting a fresh number. The §05 statement goes further: the database itself does the numbering, so there's no second owner at all. If numbering lives in the database, the chat service can't cause split brain. The database still can, during a failover, unless acks wait for a second machine as shown above. Together, this is the simple default.
• If you number in memory, use a lease with an epoch. The owner of a conversation holds a LeaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”. (the Consensus topic shows how such leaders are chosen safely): permission that expires unless renewed, with an epoch number that goes up on every handover (A had epoch 6, B gets 7). Every write carries the epoch, and the store rejects writes from an old one. That's FencingFencingMaking sure an old leader can no longer write before (or while) a new one takes over: powering it off, cutting its access, or having storage reject its requests.: a frozen ex-owner can wake up, but it can no longer write.

With an owner_epoch column added to §09's conversations table, A's write looks like this:

WITH next AS (
  UPDATE conversations
  SET    last_seq = last_seq + 1
  WHERE  id = 7
    AND  owner_epoch = 6   -- ← A still thinks it's the owner; B has already set epoch 7
  RETURNING last_seq
)
INSERT INTO messages (conversation_id, seq, sender_id, client_msg_id, body)
SELECT 7, last_seq, 1, 'a1b2c3d4-0000-4000-8000-000000000043', 'See you there' FROM next
RETURNING seq;             -- 0 rows → A learns it lost ownership and stops

This is a Conditional writeConditional writeA write that only happens if the data is still in the state you expect, e.g. UPDATE … WHERE status = 'paid' or WHERE version = 6. A repeat finds the condition false and changes nothing.: it only happens if the data is still the way the writer expects. It is the same one-statement shape as §05's send.sql, with the epoch added to the WHERE. If the epoch is stale, the UPDATE matches zero rows, so next is empty and the INSERT copies nothing: a rejected owner can't store a message either.

Don't split this into a plain UPDATE followed by a separate INSERT, even inside one transaction: an UPDATE that matches zero rows is not an error, so the INSERT would still run and store a message with a stale number. If you do use two statements, check the UPDATE's row count and ROLLBACK when it is 0.

What to monitor

The user-facing numbers

These are the ones worth an SLOSLO (service level objective)A reliability target you measure, like “99.9% of requests succeed in under 300 ms over 30 days”. The measured number (here, the share of fast successful requests) is the SLI; the 0.1% you're allowed to miss is the error budget. (a target like “99.9% of sends get their ✓ within 500 ms over 30 days”). The Observability topic explains how to set and alert on them.

Clients have to report the last two, because only the device knows what it received. Apps send small counters (“gaps seen”, “gaps filled”, “duplicates dropped”) with their regular telemetry.

The numbers that explain why

• Connections per gateway, and the reconnect rate. A gateway far fuller than the others means the load balancer isn't evening things out (§04). A spike in reconnects across all gateways means something upstream broke, or a deploy isn't draining gently.
• Duplicate rate: retries the server recognised by client message ID. A steady trickle is normal on mobile networks. A jump means acks are being lost, often a gateway or network problem.
• Registry lookup latency and “not found” rate. Slow lookups slow every message. A sudden rise in “not found” for users who are actually online means registry entries are expiring too early.
• Queue lag for push and receipts (Consumer lagConsumer lagHow far a subscriber of a message broker is behind the newest message — e.g. 5,000 messages or 3 minutes. Growing lag means the subscriber is too slow.): how far behind the push workers are. Growing lag means notifications are going out late.
• Push provider errors, split by Apple and Google. Many errors mean expired device tokens (the app was uninstalled), which you should clean up, not retry.
• Per-shard write latency. One slow shard among many usually means a hot conversation (§09).
• Heartbeats per second and presence update fan-out. Remember from §03 that presence can cost more than messages. Watch it so a client release that halves the heartbeat interval doesn't double your bill unnoticed.

Check yourself

Answer before you click. Each one is covered above: Q1 and Q4 in §05, Q2 and Q3 in §06, Q5 in §07, Q6 in §03.

Cheat sheet