Design a Notification System

The 30-second version

Almost every app sends notifications: a Push notificationPush notificationA message shown on a phone or browser even when the app isn't open. Your server can't reach the device directly; it hands the message to Apple's APNs, Google's FCM or a browser vendor's push service, which delivers it. when your food is on the way, an email with a password-reset link, a text message with a login code, a red badge on the bell icon inside the app. Each of these routes is a channelChannel (notifications)One way of reaching a person: push to their phone, email, SMS text message, or an in-app message. Each channel has its own providers, costs, speed and failure modes.. Big apps don't let every team call Apple, Google and an email company on their own. They build one notification service that everyone calls.

Here's the whole design on one line. You'll build it up piece by piece:

Why it's harder than "call the email API"

Sending one message is one HTTP call. The hard part is everything around it:

• Providers fail. Apple, Google, email and SMS companies (the providers) time out, slow down and return errors. You must retry, but carefully.
• Retries cause duplicates. If a call timed out, maybe the message went out anyway. Retry blindly and the user gets two texts with two different login codes.
• Volume is spiky. A marketing message to 10 million users arrives all at once, right when someone else needs a password reset now.
• People get annoyed. Too many messages, or a buzz at 3 am, and users turn notifications off for good.

The answer, in one paragraph

The API accepts a request (with an Idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once. so repeats are recognised), checks the user's preferences, and puts one message per channel into a queue (a waiting line of messages kept on a server, like the post office's bins), then answers right away. Workers (background programs that loop forever) take messages from the queues and call the providers. Separate queues per channel and per urgency keep slow or bulky work from blocking urgent work. Failed sends are retried with growing waits, hopeless ones are parked in a dead-letter queueDead-letter queueWhere a job goes after it has failed too many times, so it stops retrying forever and a human can look at it., every worker checks for duplicates before sending, and rate limits protect both users and providers. Status reports that come back from providers later tell you what was actually delivered.

The route through this page

What are we building?

Imagine a large shopping app. The order service, the login service and the marketing team all need to reach users. Our job is the one service they all call.

Functional: what it does

1. Send on four channels. PushPush notificationA message shown on a phone or browser even when the app isn't open. Your server can't reach the device directly; it hands the message to Apple's APNs, Google's FCM or a browser vendor's push service, which delivers it. (to phones, through Apple's APNs, the Apple Push Notification service, and Google's FCM, Firebase Cloud Messaging), email, SMS text messages, and in-appIn-app notificationA message stored in your own database and shown inside your app (the bell icon with a red badge). No outside provider is involved: it appears the next time the user opens the app, or right away over a live connection. messages (the bell icon).
2. Triggered by other services through an API: "send order_shipped to user 42 with order number 1042".
3. Respect preferencesNotification preferencesWhat each user has allowed: which channels (push yes, SMS no), which kinds (order updates yes, promotions no), and when. An opt-out is a “no” that the system must honour every time.. Users choose channels and kinds of message. An opt-out ("no promotions") must win every time.
4. TemplatesTemplate (notifications)A stored message with blanks, like “Hi {{name}}, your order {{id}} shipped”. Callers send the template name and the values; the notification service fills them in per channel and per language.. Callers send a template name plus values; we fill in the text per channel and language. Teams shouldn't paste HTML emails into API calls.
5. Scheduled and bulk sends. "Send this at 9 am tomorrow", or "send this to these 10 million users".
6. Status. Callers and support staff can ask: was it sent? Was it delivered?

Two very different kinds of message

The single most important observation in this design: not all notifications are equal.

Every later decision (separate queues, who gets dropped first under load, which limits apply) follows from this table.

Non-functional: how well it does it

• Never lose a transactional notification. Once the API says "accepted", the message must survive crashes and restarts.
• No duplicates. Two "your order shipped" pushes is sloppy; two login codes is confusing and costs money (SMS is paid per message).
• Don't spam. Per-user caps on marketing, no promos during the night in the user's own time zone.
• Fast for transactional. Say, 99% of password resets handed to the provider within 5 seconds of the request (an example target; you'd agree on yours with the product team).
• Survive a provider outage. If the SMS company goes down, login codes should still arrive, perhaps through a second company.
• Scale to big blasts without hurting everyday traffic.

Out of scope (say it out loud)

• Writing the marketing campaigns, and choosing who is in a segment ("users who bought shoes"). We receive the final list.
• Building our own email servers or SMS network. We use outside providers.
• Chat between users. That's a different system: see Design a Chat System.

How big is it? A rough estimate

All numbers below are approximate and invented for a large, typical shopping app. They are not any real company's figures, and the provider limit is illustrative: real limits depend on your provider, contract and setup, so look them up in your provider's documentation.

Step 1: the inputs

• 50 million daily active users (DAU)DAU (daily active users)How many different people use the app on a given day. Capacity estimates often start from it..
• Per user per day, on average: 4 pushes, 0.5 emails, 0.05 SMS (texts cost money, so they're used for login codes only) and 2 in-app messages.
• A day has 86,400 seconds. The busiest hour is about 3× the average: the peak factorPeak factorHow much busier the busiest hour is than the daily average, e.g. 3×. Systems are sized for the peak, not the average..
• One marketing blast: 10 million users, and the marketing team would like it out within 10 minutes.
• Our push provider setup accepts about 5,000 pushes a second (illustrative).

Step 2: everyday traffic per channel

Push: 50,000,000 × 4 = 200,000,000 a day. 200,000,000 ÷ 86,400 ≈ 2,300 per second on average, × 3 ≈ 7,000/s at peak.

Email: 50,000,000 × 0.5 = 25,000,000 a day ≈ 290/s, ≈ 900/s at peak.

SMS: 50,000,000 × 0.05 = 2,500,000 a day ≈ 29/s, ≈ 90/s at peak.

In-app: 50,000,000 × 2 = 100,000,000 a day ≈ 1,200/s. These are just rows written to our own database, no provider involved.

Total: about 330 million notifications a day, about 3,800/s on average and 11,000/s at peak. That's busy, but each send is a small HTTP call. A few dozen worker machines handle it.

Step 3: the blast (the scary number)

10,000,000 pushes in 10 minutes = 10,000,000 ÷ 600 s ≈ 16,700 per second.

That's more than twice our normal push peak (7,000/s), arriving in one go. Worse, the provider only accepts 5,000/s:

10,000,000 ÷ 5,000 per second = 2,000 s ≈ 33 minutes, no matter how many workers we add.

So the blast can't go out in 10 minutes. It must wait in a queue and drain at the provider's pace. Now picture a user who taps "forgot password" one second after the blast was queued. If their message joins the same line, it waits behind up to 10 million others: up to 33 minutes for a password reset. That one number explains most of §05.

Step 4: storage and bandwidth

Notification log (who, what, which channel, status, timestamps): about 500 bytes a row. 330,000,000 × 500 bytes ≈ 165 GB a day. Kept for 30 days: 165 GB × 30 ≈ 5 TB. A shardedShardOne piece of a split-up dataset, together with the database server that stores it. “Tenant 42 is on shard 3” means its rows live on that server. database (see Sharding) or a log store, not a single small server.

Dedupe memory (§07): remember every sent ID for 24 hours, about 100 bytes each. 230,000,000 provider sends × 100 bytes ≈ 23 GB. Fits in memory on a small RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages). cluster.

Bandwidth: a push is small, say 1 KB: 7,000/s × 1 KB ≈ 7 MB/s. An HTML email is more like 50 KB: 900/s × 50 KB ≈ 45 MB/s ≈ 360 Mbit/s. Email, not push, dominates the bytes.

Play with the inputs

Change the inputs and watch which outputs move. The widget doesn't round the inputs first, so with the defaults it shows slightly different figures (6,900/s instead of 7,000, 4.9 TB instead of 5 TB). Same estimate, less rounding. Try a 50-million-user blast, or halving the provider limit.

Version one, and what breaks

Version zero: call the provider inline

The tempting first try: the order service calls the email provider right inside its "ship order" code.

This fails quickly. If the provider takes 3 seconds, so does shipping the order. If the provider is down, either the order fails (bad) or the email is silently lost (also bad). And every team re-implements preferences, templates and retries. So we pull sending out into its own service, and make it asynchronousAsynchronous (async)Starts now but finishes later. Your code keeps going, and is told when the result is ready. (the caller doesn't wait for the send).

Version one: a table and a worker

1. The notification API writes one row per message into a notifications table with status = 'pending', and answers 202 Accepted ("got it, will do"), not "sent".
2. A worker process loops: grab some pending rows, call the provider, mark each row sent or failed.

-- Grab up to 100 pending rows that no other worker is holding right now.
SELECT id, user_id, channel, payload
FROM notifications
WHERE status = 'pending' AND send_after <= now()
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;   -- ← rows another worker has locked are skipped, not waited for

One detail: don't hold that transaction open while calling providers, or rows stay locked for the whole slow batch. Claim the rows instead (UPDATE … SET status = 'sending' on the selected IDs), commit, then send and update each row afterwards. Store a claimed_at time too, so rows left in sending by a crashed worker can be picked up again after a timeout.

This is honestly a good design for a small app. Writes are durable (in the database), restarts are safe (unfinished rows stay pending), scheduled sends are one column. At 1 million notifications a day (about 12 a second), one database and a couple of workers are plenty.

Grow it: 10× and 100×

Drag the slider. Components appear as traffic grows, and the current bottleneck is highlighted in red.

At 10× (10M a day: 10,000,000 ÷ 86,400 ≈ 115/s, × 3 ≈ 350/s at peak), the first break is the worker. A provider call takes, say, 100 ms, so one worker sending one message at a time manages about 10 a second. 350 ÷ 10 = 35 workers, or fewer workers that each send many messages at once. The second break: one loop for all channels means a slow email provider holds up pushes. So: one pool of workers per channel.

At 100× (100M a day ≈ 1,200/s, ≈ 3,500/s at peak), the table itself breaks. Every message is inserted, polled, locked and updated several times: say 4 writes each, so 3,500 × 4 ≈ 14,000 writes a second, all on one database. Meanwhile a blast like the one in §03 fills the table with 10 million rows that the workers must chew through before reaching a password reset. §03's full numbers (330M a day, ~11,000/s at peak, about 330×) only make this worse. The fixes, each covered in its own section:

• A Message brokerMessage brokerA server that accepts messages from producers and hands them to consumers later — RabbitMQ, Kafka, SQS, NATS. It decouples who sends from who handles, and buffers bursts. (Kafka, RabbitMQ, SQS…) with one queue per channel replaces polling the table (§05).
• Priority lanes: separate queues for transactional and bulk (§05).
• Rate limiters in front of each provider, and per-user caps (§08).
• A dedupe store and retry queues with growing waits (§06, §07).
• A second provider per channel for failover (§09).

The pipeline, one notification at a time

The request

The order service calls us like this. The Idempotency-Key header is a unique ID the caller picks for this one intent ("tell user 42 about order 1042 shipping"). If the caller times out and sends the same request again, the same key tells us it's a repeat. (Idempotency keys, end to end covers the full protocol.)

POST /v1/notifications
Idempotency-Key: order-1042-shipped
Content-Type: application/json

{ "user_id": 42, "template": "order_shipped", "priority": "transactional",
  "params": { "order_id": 1042, "eta": "Tuesday" } }

HTTP/1.1 202 Accepted

{ "notification_id": "ntf_7f3a", "status": "accepted" }

202 Accepted means "stored, will be sent", never "delivered". The caller can look up ntf_7f3a later to see what happened.

Step through one notification

What each box does, and why it's separate

The API validates the request, checks the idempotency key, loads the user's preferencesNotification preferencesWhat each user has allowed: which channels (push yes, SMS no), which kinds (order updates yes, promotions no), and when. An opt-out is a “no” that the system must honour every time. and device tokensDevice tokenAn opaque string that APNs (Apple) or FCM (Google) gives an app on one phone. Your server stores it and sends it with each push so the provider knows which device to deliver to. It changes or dies when the app is reinstalled or uninstalled. (both read constantly, so they sit in a CacheCacheA small, fast store of answers you already computed or fetched, so the next request for the same thing skips the slow work. The hard part is knowing when a cached answer is out of date. in front of the database; see cache-aside), and writes one message per allowed channel. Rendering the templateTemplate (notifications)A stored message with blanks, like “Hi {{name}}, your order {{id}} shipped”. Callers send the template name and the values; the notification service fills them in per channel and per language. into final text can happen here or in the channel worker. Rendering late, in the worker, keeps the API fast.

Durability before the 202. The API must not answer until the message is stored somewhere that survives a crash. If it writes a database row and then publishes to the broker, a crash in between loses the message. Two standard fixes: write the row and an outbox entry in one transaction, and let a relay publish it (the Outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does., see Queues §exactly-once); or make the broker the first write and wait for its confirmation before answering.

One queue per channel is a BulkheadBulkheadSeparate pools (or limits) for different kinds of work, so one slow kind can't use up all the capacity. Named after the walls that keep a ship from sinking when one compartment floods.: if the email provider slows to a crawl, the email queue grows, but push and SMS workers keep going. Each channel also has its own worker pool sized for its own provider.

Priority lanesPriority laneA separate queue (with its own workers) for one class of urgency, e.g. “transactional” vs “bulk”. Urgent messages never wait behind a backlog of less urgent ones, because they're not in the same line. split each channel again: push.transactional and push.bulk, with their own workers. A password reset never stands in line behind the blast, because it's in a different line. Bulk workers are deliberately limited so that bulk traffic can never use up the whole provider allowance.

Blasts take a different entrance

A 10-million-user campaign isn't 10 million API calls. It's one request ("send weekend_sale to segment s_881, start 09:00"). A fan-out job reads the user list in pages, checks each user's preferences, and writes messages into the bulk lanes at a controlled pace. Fan-outFan-outDelivering one message to many recipients — one chat message pushed to all 500 people in the room. just means one thing turning into many copies. Doing it in a background job means a blast can't flood the API, and it can be paused halfway.

The tables, roughly

Retries without a storm

Which errors to retry

Providers answer with HTTP status codes, or don't answer at all. Sort every outcome into one of three buckets:

A timeout is the tricky one: it's an Unknown outcomeUnknown outcomeThe third possible result of a call over the network: you sent the request and got no answer (timeout, crash, lost reply). The work may or may not have happened, and you can't tell which.. The provider may have sent the push and the reply got lost. That's why retries and deduplication always come as a pair. (Success, failure, and "I don't know" explains why you can't avoid this case.)

// Outcome is what the provider told us (or didn't).
type Outcome struct {
	Status       int  // HTTP status from the provider; 0 = timeout / no answer
	InvalidToken bool // set by parsing the reply: APNs 410 Unregistered, FCM 404 UNREGISTERED
}

// decide: retry later, drop for good, or drop and delete the token.
func decide(o Outcome) string {
	switch {
	case o.InvalidToken:
		return "drop + delete token" // ← retrying can never help
	case o.Status == 0, o.Status == 429, o.Status >= 500:
		return "retry" // timeout, "slow down", provider trouble: may work later
	case o.Status >= 400:
		return "drop" // our request is wrong (bad address, bad payload)
	default:
		return "done"
	}
}

// ceiling is the longest we may wait before retry number `attempt` (1, 2, 3…).
func ceiling(attempt int) time.Duration {
	base, maxWait := time.Second, 5*time.Minute
	return min(base<<(attempt-1), maxWait) // 1s, 2s, 4s, 8s… never above 5 min
}

// backoff picks the actual wait: anywhere from 0 up to the ceiling ("full jitter").
func backoff(attempt int) time.Duration {
	return rand.N(ceiling(attempt)) // math/rand/v2
}

Running both with a few outcomes and the first ten retries (this is the JS output; Go prints the last four as 1m4s, 2m8s, 4m16s, 5m0s, otherwise it's identical):

status 200 → done
status   0 → retry
status 503 → retry
status 429 → retry
status 400 → drop
status 410 → drop + delete token
retry  1: wait up to 1s
retry  2: wait up to 2s
retry  3: wait up to 4s
retry  4: wait up to 8s
retry  5: wait up to 16s
retry  6: wait up to 32s
retry  7: wait up to 64s
retry  8: wait up to 128s
retry  9: wait up to 256s
retry 10: wait up to 300s

Why the waits grow, and why they're random

Exponential backoff (doubling the wait each time) gives a struggling provider room to recover. Jitter (picking a random wait up to the ceiling) stops everyone retrying at the same instant. Without it, 10,000 messages that failed together at 09:00:00 all come back at 09:00:01, fail together again, come back at 09:00:03… Each wave is a Thundering herdThundering herdMany clients hitting a server at the same moment — e.g. all retrying together right after an outage — which knocks it over again. that knocks the provider over again just as it recovers.

Where the waiting happens

A worker shouldn't sleep for 4 minutes holding a message: it would sit idle, and a restart would lose track of the wait. Instead, on a retryable failure the worker re-queues the message with a delay (attempts + 1, not_before = now + backoff) and acksAcknowledgement (ack)A “got it, done” message from a receiver back to the sender (or to a message broker). If no ack arrives in time, the sender assumes the message was lost and sends it again. the original (tells the broker it's done with that copy). Brokers offer this in different ways: SQS has per-message delays (up to 15 minutes), RabbitMQ setups use a "retry" queue whose messages expire back into the main one, and with Kafka people use separate retry topics (e.g. push.retry.1m, push.retry.10m).

Budgets and the dead-letter queue

• Cap the attempts. After, say, 8 attempts (7 retries, so at most about 2 minutes of waiting in total: 1 + 2 + 4 + … + 64 s) or a deadline, stop. A transactional message that's an hour late is often useless anyway: a login code expired long ago.
• A Retry budgetRetry budgetA cap on how much extra load retries may add (say, 10% of normal traffic), so retries can't multiply an outage.. If more than, say, 10% of sends this minute are retries, stop adding new ones. A provider outage then doesn't triple our own traffic.
• The Dead-letter queueDead-letter queueWhere a job goes after it has failed too many times, so it stops retrying forever and a human can look at it. (DLQ). A message that ran out of attempts, or that crashes the worker every time (a Poison messagePoison messageA message that fails every time it's processed (bad data, a bug for that one input). Retrying it can never succeed and, without a limit, blocks or burns the consumer.), goes to a separate queue with an alert, so a human can inspect it and re-send ("redrive") after a fix. Poison messages and dead-letter queues covers the mechanics.

No duplicates: dedupe at every hop

Where duplicates come from

1. The caller retries the API. Their request timed out, so they send it again. → The API's Idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once. check catches it (§05).
2. The queue delivers twice. Brokers promise At-least-once deliveryAt-least-once deliveryA delivery promise where a message is never lost but may arrive more than once (for example, re-sent after a crash). Receivers must therefore handle duplicates safely. delivery: if a worker takes a message and doesn't ack it in time (crash, restart, slow), the broker hands it to another worker. → The worker must dedupe. Exactly-once: what's really on offer explains why no broker setting removes this.
3. We retry a timeout that actually worked. The provider sent it; only the reply was lost. → Same worker check, plus the tricks at the end of this section.
4. A blast job restarts halfway. It re-reads the user list from the start. → Give each message a deterministic ID, such as campaign_881:user_42, so the re-run produces the same IDs and the worker skips the ones already sent.

What exactly is "one" message?

Before any check, decide what the dedupe key names. One notification can become several deliveries: ntf_7f3a goes out by push to Mia's iPhone, by push to her iPad, and by email. Each of those must be sent exactly once, independently. So the worker's key is notification + channel (+ device for push):

ntf_7f3a:push:iphone-8c2   ntf_7f3a:push:ipad-41d   ntf_7f3a:email

If the key were just ntf_7f3a, the push worker would mark it "sent", and the email worker would then find "sent" and skip the email: a lost notification, caused by the dedupe check itself.

The worker check: claim, send, mark

The simple version, "if I've seen this key, skip; otherwise send and remember it", has a gap: when do you remember it?

• Remember after sending: a crash between "provider accepted" and "remember" means the redelivered copy is sent again. A duplicate.
• Remember before sending: a crash after "remember" but before the send means the redelivered copy is skipped. The message is lost, and a lost password reset is worse than a duplicate.

The usual answer is a claim with a LeaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”.: write "sending" with a short expiry (say 30 seconds) before calling the provider, and "sent" with a long expiry after. Then a redelivered copy can find three things, and each needs a different action:

type entry struct {
	state string    // "sending" or "sent"
	until time.Time // when this entry expires
}

// Dedupe remembers which deliveries were sent. A key is one delivery:
// notification + channel (+ device for push), e.g. "ntf_1:push:iphone-a".
type Dedupe struct {
	mu   sync.Mutex
	seen map[string]entry
}

// Begin tries to claim key for `lease`. It returns
//
//	"claimed" → it's yours: send, then call Done
//	"sending" → another worker holds a live claim: requeue, try later
//	"sent"    → already sent: skip it and ack
func (d *Dedupe) Begin(key string, now time.Time, lease time.Duration) string {
	d.mu.Lock()
	defer d.mu.Unlock()
	if e, ok := d.seen[key]; ok && now.Before(e.until) {
		return e.state // ← "sending" or "sent": don't send
	}
	d.seen[key] = entry{"sending", now.Add(lease)} // new, or an expired claim
	return "claimed"
}

// Done marks key as sent and remembers it for the dedupe window.
func (d *Dedupe) Done(key string, now time.Time, window time.Duration) {
	d.mu.Lock()
	defer d.mu.Unlock()
	d.seen[key] = entry{"sent", now.Add(window)}
}

func main() {
	d := &Dedupe{seen: map[string]entry{}}
	t0 := time.Date(2026, 10, 31, 9, 0, 0, 0, time.UTC)
	lease, window := 30*time.Second, 24*time.Hour
	show := func(key, when, got string) { fmt.Printf("%-20s %-15s → %s\n", key, when, got) }

	// ntf_1 goes out by push and by email: two deliveries, two keys.
	show("ntf_1:push:iphone-a", "first delivery", d.Begin("ntf_1:push:iphone-a", t0, lease))
	show("ntf_1:email", "first delivery", d.Begin("ntf_1:email", t0, lease))
	d.Done("ntf_1:push:iphone-a", t0.Add(time.Second), window)
	show("ntf_1:push:iphone-a", "redelivered", d.Begin("ntf_1:push:iphone-a", t0.Add(time.Minute), lease))

	// ntf_2: the worker claims it and crashes before calling Done.
	show("ntf_2:sms", "first delivery", d.Begin("ntf_2:sms", t0, lease))
	show("ntf_2:sms", "after 10s", d.Begin("ntf_2:sms", t0.Add(10*time.Second), lease))
	show("ntf_2:sms", "after 31s", d.Begin("ntf_2:sms", t0.Add(31*time.Second), lease))
}

Both print the same thing:

ntf_1:push:iphone-a  first delivery  → claimed
ntf_1:email          first delivery  → claimed
ntf_1:push:iphone-a  redelivered     → sent
ntf_2:sms            first delivery  → claimed
ntf_2:sms            after 10s       → sending
ntf_2:sms            after 31s       → claimed

The email of ntf_1 is claimed even though its push was already sent: different keys. The redelivered push is skipped. And ntf_2's crashed claim blocks other workers for 30 seconds, then becomes claimable again.

In production the map is shared by all workers, typically in RedisRedisA popular in-memory data server: a fast key–value store and cache that also offers counters, sorted sets, pub/sub channels and streams (an append-only log for messages).. SET key sending NX PX 30000 means "set this key only if it doesn't exist (NX), expiring in 30,000 ms (PX)". It answers OK (claimed) or nothing (someone has it). To tell "sending" from "sent" when it fails, follow it with GET key, or do both in one small Lua script so no other worker can sneak in between. After the send, SET key sent EX 86400 keeps it for a day: the dedupe windowDedupe windowHow long a worker remembers that a delivery key (notification + channel, + device for push) was already sent, e.g. 24 hours. A duplicate arriving inside the window is skipped; after it, the memory is gone to save space.. §03 estimated that memory at about 23 GB.

Three timeouts that must line up

The lease only works if it outlasts the work it protects. Picture worker A, alive but slow: its provider call hangs for 40 seconds. At 30 s the lease expires, worker B claims the key and sends, and then A's call finishes too. Both sent, and nobody crashed. Two rules prevent it:

1. Lease > the longest a send can take. Put a hard deadline on the whole send (rendering plus the provider call): context.WithTimeout in Go, AbortSignal.timeout() in JS. With a 5-second provider timeout (§09), a 30-second lease leaves plenty of room. A worker that misses its deadline gives up rather than finishing late.
2. Broker redelivery timeout > the lease. The broker's Visibility timeoutVisibility timeout (SQS)When an Amazon SQS consumer receives a message, the message is hidden from other consumers for this long. If the consumer doesn't delete it in time (because it crashed or was slow), the message reappears and is delivered again. (how long it hides a taken message before handing it to someone else) should be longer than the lease, say 60 s. Otherwise the broker redelivers while A's claim is still live, and B just requeues it again and again.

The gap nobody can close

Look at ntf_2 again. If the first worker crashed after the provider accepted the message but before Done, the claim at 31 s sends it a second time. No check on our side can tell "crashed before the send" from "crashed after", because the provider's answer was lost with the worker. Two ways to soften it:

• Pass your key downstream. Some email and SMS provider APIs accept an idempotency key; if yours does, send the delivery key and let them drop the repeat. (A "client reference" or tag field usually doesn't dedupe; it only labels the message so you can match it up later.)
• Replace instead of repeat, for push. Both push services let you label a notification so that a newer one with the same label replaces the older one on screen instead of adding a second banner. Use the delivery key as the label and a duplicate push mostly disappears:
  • APNs (Apple Push Notification service): the apns-collapse-id header, at most 64 bytes. Apple documents it as merging notifications with the same ID into one for the user.
  • FCM (Firebase Cloud Messaging), on Android: the android.notification.tag field. A notification with the same tag replaces the one already shown.
  • Not FCM's collapse_key. Despite the name, it does something else: while a device is offline, FCM may replace a pending, not yet delivered message with a newer one carrying the same key. It doesn't touch a banner already on screen, notification messages ignore it, and FCM keeps at most 4 different collapse keys per device. Giving every message its own collapse_key can make FCM discard pending messages for an offline phone.

A safety net above the IDs

ID checks only work if the caller reuses the same key. A buggy upstream service that generates a new key on every retry slips right past them. Many teams add a coarse content check: "the same template to the same user within 10 minutes? Hold it." It catches bugs the ID checks can't.

Don't spam: caps, quiet hours, provider limits

Per-user: frequency caps

A frequency capFrequency capA per-user limit on how many notifications of a kind may be sent in a period, e.g. at most 3 marketing pushes a day. Extra ones are dropped or merged, so users aren't spammed into turning everything off. says "at most N of this kind per period", for example 3 marketing pushes per user per day. It's the single best defence against users turning notifications off entirely, which loses you the channel for good.

The check runs in the worker (or the fan-out job) just before sending. It's a counter per user per day, usually in Redis: increment cap:42:2026-10-31:marketing, and if the result is above 3, skip the message. The date in the key is the user's local date (a "day" is their day, not the server's), and the counter is only incremented when a message is actually sent, so a message held for quiet hours counts toward the day it goes out. That's a fixed-window counter, and here its simplicity is a feature: "per calendar day" is exactly what product people mean. Transactional messages are exempt. You never cap password resets.

What happens to a message over the cap? For marketing, drop it (and record why, so the dashboard can show it). For something like "someone liked your photo", merge it instead: "Mia and 4 others liked your photo".

Per-user: quiet hours and time zones

Quiet hoursQuiet hoursA time window in the user's own time zone (say 22:00 to 08:00) when non-urgent notifications are held back and sent the next morning instead., say 22:00 to 08:00, are in the user's Time zoneTime zoneThe offset from UTC that a place uses for its clocks (Hanoi is UTC+7, New York is UTC−5 or −4). A computer's “local time” depends on the time zone it is set to, which often differs between laptops and CI servers., not the server's. A blast queued at 16:30 UTCUTCCoordinated Universal Time: the world's reference time, with no time zones and no daylight saving. Servers and databases should store and log times in UTC and only convert to local time for display. lands at 23:30 in Hanoi, 16:30 in London and 12:30 in New York. So each user's message is checked on its own: send now, or hold it until 08:00 their time. ("UTC hygiene" in Clocks: time in real systems covers storing UTC and converting only for display.)

// Quiet hours: no marketing between 22:00 and 08:00 in the user's own time zone.
const quietStart, quietEnd = 22, 8

// sendAt returns when a marketing message may go out: now, or the next 08:00 local.
func sendAt(now time.Time, zone string) (time.Time, error) {
	loc, err := time.LoadLocation(zone) // e.g. "Asia/Ho_Chi_Minh"
	if err != nil {
		return time.Time{}, err
	}
	local := now.In(loc) // same instant, shown on the user's wall clock
	h := local.Hour()
	if h >= quietEnd && h < quietStart {
		return now, nil // daytime: send now
	}
	day := local
	if h >= quietStart {
		day = local.AddDate(0, 0, 1) // late evening → 08:00 tomorrow
	}
	return time.Date(day.Year(), day.Month(), day.Day(), quietEnd, 0, 0, 0, loc), nil
}

func main() {
	now := time.Date(2026, 10, 31, 16, 30, 0, 0, time.UTC) // one moment, three users
	for _, zone := range []string{"Asia/Ho_Chi_Minh", "Europe/London", "America/New_York"} {
		loc, _ := time.LoadLocation(zone)
		at, _ := sendAt(now, zone)
		fmt.Printf("%-17s local %s → send at %s UTC\n", zone,
			now.In(loc).Format("15:04"), at.UTC().Format("Jan 2 15:04"))
	}
}

Asia/Ho_Chi_Minh  local 23:30 → send at Nov 1 01:00 UTC
Europe/London     local 16:30 → send at Oct 31 16:30 UTC
America/New_York  local 12:30 → send at Oct 31 16:30 UTC

Asia/Ho_Chi_Minh  local 23:30 → send at 2026-11-01T01:00:00.000Z
Europe/London     local 16:30 → send at 2026-10-31T16:30:00.000Z
America/New_York  local 12:30 → send at 2026-10-31T16:30:00.000Z

The Hanoi user's message is held until 08:00 their time, which is 01:00 UTC. The JS version counts minutes on the wall clock, so on a night when the clocks change (Daylight saving timeDaylight saving time (DST)Moving clocks forward one hour in spring and back in autumn, as many countries do (but not Vietnam). On those days a calendar day is 23 or 25 hours long, so “+24 hours” isn't always “tomorrow”.) it can be an hour off; the Go version asks for "08:00 local" directly and doesn't have that problem.

Held messages go back into the bulk lane with send_after set (the same delayed re-queue as retries in §06). Watch out for the knock-on effect: if a whole country's held messages all become due at exactly 08:00:00, you've created a mini-blast. Spreading them over the first 30 minutes (jitter again) smooths it out.

Try the checks together. Pick a user, a time and a message type:

Per-provider: stay under their limit

Every provider limits how fast you may send (§03 used an illustrative 5,000 pushes a second). Go over it and you get 429 errors, which turn into retries, which make things worse. Rather than discover the limit by hitting it, each channel's workers pace themselves with a Token bucketToken bucketA rate-limit algorithm: a bucket refills with tokens at a steady rate (say 10 per second) up to a maximum. Each request takes one token; no token, no request. The maximum allows short bursts.: a bucket that refills at the provider's rate, and each send takes one token. No token, wait. Token bucket and leaky bucket explains the two knobs (rate and burst).

Two details matter here:

• The limit is shared by all workers. Twenty workers each limiting themselves to 5,000/s would send 100,000/s. Either give each worker a slice (5,000 ÷ 20 = 250/s each) or use one shared bucket in Redis, as in Many servers, one limit.
• Split the allowance between lanes. For example, bulk gets at most 70% of the provider's rate, so even a full blast leaves room for transactional sends.

When the provider fails

Sent is not delivered

"Sent" and "delivered"Sent vs delivered“Sent” means the provider (APNs, an email service, an SMS gateway) accepted the message from you. “Delivered” means it reached the phone or inbox, which you only learn later, if at all, from a status callback or from the app itself. are different facts, learned at different times:

• Sent: the provider answered our call with success. We know this right away.
• Delivered: the email reached the inbox, the SMS reached the phone. We learn it later, from a status callbackStatus callback (delivery receipt)A webhook the provider sends back to you later saying what happened to a message: delivered, bounced, failed, or the user marked it as spam. Email and SMS providers typically offer these.: a WebhookWebhookAn HTTP request one service sends to a URL another registered, to say something happened (order.paid). It's a reversed API call. When you receive webhooks (from a payment provider, say), expect them late, twice or out of order, and verify their signature. the provider sends to an endpoint of ours, saying "message abc was delivered" (or bounced, failed, marked as spam).
• Push is special. APNs (Apple Push Notification service) and FCM (Firebase Cloud Messaging) accept the message, but neither calls a webhook of yours per message saying "it was shown". Teams that need that number have the app itself report back when a notification is received or opened. FCM can also export, optionally and with a delay, per-message "delivered to the app on the device" data to BigQuery (Google's data warehouse) for Android, plus aggregate delivery reports. That's useful for dashboards, not for updating one notification's status in real time. (As of October 2026; check the Firebase docs.)

So each notification moves through states, and different events move it. Click the events to drive one message through its life:

The status-callback endpoint is a busy little service of its own (from §03's peak numbers, email 900/s + SMS 90/s ≈ 1,000 callbacks a second, and more if a provider sends several events per message: delivered, opened, clicked). It must check the callback is really from the provider (providers sign them, often with an HMACHMACA way to compute a MAC (a tamper-proof tag) from a message and a shared secret using a hash function such as SHA-256. Same message + same secret always gives the same tag; change one byte of either and the tag changes completely.), find our notification by the provider's message ID that we saved when sending, and update its status. Callbacks can arrive late, twice, or out of order ("delivered" before "sent" was recorded), so updates must only move a status forward, never back.

What the callbacks tell you to do

• Hard bounceBounce (email)An email that couldn't be delivered. A hard bounce (the address doesn't exist) is permanent: stop sending to it. A soft bounce (mailbox full, server busy) may work later. (address doesn't exist) or spam complaint: add the address to a suppression list and never send to it again. Email providers watch your bounce and complaint rates, and a high rate gets your mail sent to spam folders, for everyone.
• Dead device token (from the send response, §06): delete it.
• Undeliverable SMS number: mark the number invalid and ask the user to update it.

Timeouts and circuit breakers

Every provider call gets a short TimeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever., say 5 seconds for push. Without one, a provider that hangs instead of failing holds every worker hostage, and the queues grow while the workers wait.

When a provider is clearly down (say, more than half of the last 100 calls failed), retrying every message just burns the retry budget. A Circuit breakerCircuit breakerA wrapper around calls to another service that stops calling it for a while after too many failures (“opens”), failing fast instead of piling up waiting requests, then tries again carefully. watches the failure rate and, past a threshold, opens: workers stop calling that provider and fail fast instead. After a pause it lets a few test calls through ("half-open"); if they succeed, it closes again. While the breaker is open, messages wait in the queue, or go to the backup provider.

Failover to a second provider

For channels where a loss really hurts, keep a second provider under contract: a second SMS gateway or email service. When the main one's breaker opens, failoverProvider failoverSwitching a channel to a second, standby provider (another SMS gateway or email service) when the main one is down or failing, and switching back when it recovers. routes new sends to the backup. Some caveats:

• Push has no real second provider. In practice, push to an iPhone goes through APNs: even FCM's iOS SDK delivers through APNs under the hood. An Android phone with Google services receives through FCM. So if APNs is down, push to iPhones waits. Your fallback is another channel: send the login code by SMS or email instead. (As of October 2026. Rules like the EU's Digital Markets Act keep changing what platforms must allow, so check the current state.)
• The backup must be ready. Templates, sender names or numbers, and approved message formats all have to exist there already, and the backup needs some live traffic (say 5%) every day, or you discover it's broken during the outage.
• Failover can duplicate. A message that timed out at provider A may have gone out, and now B sends it too. For a login code, that's an acceptable price.

The provider that says 200 and goes silent

The nastiest failure: the provider accepts everything, returns 200, and then delivers nothing, or delivers hours late. Our dashboards show green. Two defences:

• A reconciliation jobReconciliation job (notifications)A periodic job that finds notifications stuck in “sent” with no final status after, say, an hour, and asks the provider about them or marks them unknown so they can be re-sent or reported. that finds messages stuck in sent with no callback after, say, an hour, and either asks the provider's status API or marks them unknown and alerts. (The general idea is ReconciliationReconciliation (data)A periodic job that compares your records with another system's (e.g. your orders vs the payment provider's charges) and flags or fixes any differences. The safety net for whatever retries and keys missed.: compare your records with theirs and fix the differences.)
• A delivery-rate alarm: if the share of email or SMS reaching delivered within 10 minutes drops from 98% to 60%, page someone, and consider failing over even though no call is failing.

Failure drills: break it on purpose

The simulator

Every dot is a notification: transactional or bulk. Turn on trouble and watch the queues, the retries, the dead-letter queue and the counters. Try these four drills in order:

1. Start a blast with priority lanes off. Watch how long transactional messages wait (the "oldest tx waiting" number). Then turn lanes on and start another blast.
2. Kill provider A. A few sends fail and retry, then the circuit breaker opens and messages pile up in the queue (failover is off). Turn failover on.
3. Crash workers mid-send with dedupe off, then on. Watch the duplicates counter. With dedupe on it drops sharply but not to zero, and the log shows why: each remaining duplicate came from a crash before "sent" was written, the gap from §07.
4. Inject a poison message and watch it bounce through retries into the DLQ.

The failure modes, one by one

A blast delays password resets. Covered in §03 and §05: with one shared line, the reset waits behind 10 million messages, up to half an hour. Separate priority lanes with reserved workers and a reserved slice of the provider's rate fix it. Monitor: the age of the oldest message in the transactional lane.

A poison message. One message makes the worker crash every time (a template that fails to render for one user's name, say). Without an attempt cap, it's redelivered forever and keeps killing workers. With a cap, it lands in the Dead-letter queueDead-letter queueWhere a job goes after it has failed too many times, so it stops retrying forever and a human can look at it. after N tries and the lane keeps flowing. Monitor: DLQ size, with an alert on any growth.

A worker crashes mid-send. The broker redelivers the message to another worker. The dedupe claim (§07) skips it if it was marked sent, waits if the lease is live, and resends if the lease expired, which is the rare duplicate we accepted. Monitor: how often a worker takes over an expired claim (each one is a possible duplicate), plus user reports.

A provider says 200 and never delivers. No error anywhere; only the delivered rate falls. The reconciliation job and a delivery-rate alarm catch it (§09).

The provider is down. The breaker opens, messages wait or fail over. Transactional SMS and email move to the backup provider; push falls back to another channel for urgent messages and waits for the rest.

What users see when each component dies

What to monitor, and staff-level trade-offs

The dashboard

The broker's Consumer lagConsumer lagHow far a subscriber of a message broker is behind the newest message — e.g. 5,000 messages or 3 minutes. Growing lag means the subscriber is too slow. is the same idea as oldest-message age, measured in messages instead of seconds. Seconds are easier to set an alert on, because the right number of messages changes with traffic. Observability covers how to record these.

Trade-offs a staff engineer raises

Build or buy. Hosted notification platforms bundle templates, preferences, channels and provider failover. For a small team, buying is often right. Building pays off at large volume, with unusual routing rules, or when notifications are core to the product. Either way, the concepts on this page are what you'll configure or debug.

Cost, not just speed. SMS is paid per message, so a duplicate or a retry storm on SMS costs real money, and one bad loop can produce a large bill overnight. Put a hard daily spend cap on paid channels, and alert when spend runs ahead of the usual curve.

Channel fallback order. "Push, and if no device token, email; SMS only for security codes" is a product decision with cost and annoyance trade-offs. Make it data (a per-template policy), not code scattered across workers.

Multi-region. Running the pipeline in two regions survives a region outage, but now the dedupe store and idempotency keys must be shared or partitioned by user, or the same message can be sent from both regions. Most teams pin each user to a home region and fail over the whole pipeline, rather than sending from both at once.

Batching for throughput. Some provider APIs accept many recipients per call. Batching cuts per-call overhead a lot for blasts, but one bad recipient can fail the batch, and per-message retry and status get more complicated. Use it for bulk; keep transactional one at a time.

Ordering is usually unnecessary. "Order shipped" arriving before "order confirmed" looks odd but harms no one, and strict ordering across retries is expensive. (Queues §ordering covers what it would take.) If one user's messages must stay in order, key messages by user ID so they share a partition (one ordered slice of a queue or Kafka topic; messages with the same key always land in the same one).

Check yourself

Decide before you click. Every one is covered above: Q1 in §03 and §05, Q2 and Q3 in §06, Q4 in §07, Q5 in §08, Q6 in §09, Q7 in §09 and §10.

Cheat sheet

Numbers from §03 (approximate): 50M DAU → ~330M notifications a day, ~11,000/s at peak. A 10M blast at an illustrative 5,000/s takes ~33 minutes, so it must queue in its own lane.