Idempotency & Retries

The 30-second version

You tap Pay €50. The spinner turns for ten seconds, then: "Something went wrong. Please try again." Do you tap again? Maybe the payment never went through. Or maybe it did, and only the "success" message got lost on the way back to your phone. You can't tell, and neither can the app.

Watch both endings. First as most code is written, then with one small addition:

The fix in the second run is an Idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once.: a random ID the app creates once per purchase and sends with every attempt. The server writes down which keys it has finished. When a key comes back, the server doesn't do the work again; it just sends the saved answer.

What this topic covers

1. Why retries are unavoidable: a request has three outcomes, not two (§02).
2. What makes an operation safe to repeat, and how to design operations that are safe by nature (§03).
3. Why Exactly-onceExactly-onceThe (mostly impossible over a network) promise that a message is processed one time, no more, no less. Real systems get the effect with at-least-once delivery plus idempotent processing. delivery is impossible, and what real systems do instead (§04).
4. Idempotency keys end to end, with a working Postgres implementation in Go and JavaScript (§05–§07).
5. The same idea for message consumers that receive duplicates (§08).
6. Retrying well: which errors to retry, how long to wait, and how retries can take down a whole system (§09).
7. Running it in production (§10), then pitfalls, a quiz and a cheat sheet.

Success, failure, and “I don't know”

A program talking to a server over the network is in exactly that position. The trip has several legs (the request travels, the server works, the reply travels back), and any of them can fail without telling you which. Try each failure:

Look at the left box: it never changes. In every case, all the phone knows is "no answer within 10 seconds". When part of a system fails while the rest keeps going, and the caller can't see which part, that's called a Partial failurePartial failureWhen one part of a distributed system fails while the rest keeps working — e.g. the server processed your order but the reply was lost. The caller can't tell "never happened" from "happened, answer lost".. It's the defining problem of any Distributed systemDistributed systemA program that runs on several machines that cooperate over a network. Any machine or link can fail or slow down independently, which is what makes them hard.: a program spread over several machines that talk over a network.

Three outcomes, not two

That last row is the whole topic. "It may still be happening" matters: in the just slow case, the first request is still running on the server when you retry. So your retry and the original can run at the same time. Keep that in mind; it comes back in §05.

"Just check first, then retry"?

A natural idea: before retrying, ask the server "did my payment go through?". Two problems:

1. Which payment? If you paid €50 to the same shop yesterday, "a €50 payment" doesn't identify this one. You need a unique name for this purchase, sent with the original request. Hold that thought: that unique name is the idempotency key from §01.
2. The check races the original. In the just slow case, your check runs while the first request is still in progress. The check says "no payment yet", you retry, and then the original finishes too. Two charges.

So checking first doesn't remove the problem. The retry itself has to be safe.

You will get duplicates, even if you never write a retry

It's tempting to think "my code doesn't retry, so I don't get duplicates". Here's who retries on your behalf:

• People. They tap again, refresh, open a second tab.
• Apps and SDKs. Many HTTP client libraries retry failed requests automatically.
• Proxies and load balancersLoad balancerA component that spreads incoming requests or connections across several servers, so no single one is overloaded and a dead one can be skipped.. They may resend a request to another server if the first one fails (Load Balancing §09).
• Go's own HTTP client. It quietly resends some requests on a fresh connection when a reused one turns out to be dead (HTTP §08).
• Message queues. They redeliver any message whose consumer didn't confirm it in time (§08).
• Schedulers. A job that crashed halfway gets restarted from the top.

What “idempotent” really means

Everyday examples:

• "Turn the light on" is idempotent: pressing it again changes nothing. "Toggle the light" is not: the second press undoes the first.
• An elevator call button: pressing it five times still brings one elevator.
• Rounding a number: round(round(2.7)) = round(2.7) = 3.

In software, the "world" is your data plus anything else you change: rows, files, emails sent, money moved. Those changes are called side effectsSide effectAnything an operation changes in the outside world besides returning a value: a row written, a card charged, an email sent. Side effects are what make repeating a request dangerous.. Try a few operations:

SET balance = 50

Two things idempotency does not mean

It doesn't mean the same reply. DELETE /cart/items/3 answers 204 the first time and 404 Not Found the second. That's fine: the cart is in the same state either way. What we care about is the effect on the data. (A good idempotency-key server goes further and does return the same reply every time; that's §05.)

It doesn't mean "no side effects at all." Writing a log line or bumping a metric on every request is fine. Nobody is harmed by two log lines. What matters is the effects someone would notice: money, messages, stock counts.

Three ways to make an operation idempotent by design

Before reaching for idempotency keys, see if you can make the operation safe by nature. It's simpler and has nothing to clean up later.

1. Say the result, not the change. "Set the quantity to 3" can be repeated; "add 1 to the quantity" can't. Many APIs can be reshaped this way: PUT /cart/items/7 {"qty": 3} instead of POST /cart/items/7/increment.

2. Let the caller pick the ID. If the client generates the order's ID (a random UUIDUUID (v4 / v7)A 128-bit ID like 3f2b…-…. Version 4 is fully random; version 7 starts with a timestamp, so new IDs are roughly in increasing order.) and sends it along, the server can tell a repeat from a new order. With a UNIQUE constraintUNIQUE constraintA rule on a database column: the database refuses to store a second row with the same value there — even if two inserts arrive at the same instant. on the ID, a repeated insert is simply skipped. An ID that comes from the caller or from the real world, rather than from the database, is a Natural keyNatural keyAn ID that comes from the real world or from the caller (an order number, “user 7's subscription for October”) instead of being invented by the database. Because a retry carries the same value, the database can recognise it..

-- Every run adds another €50. A retry = €100.
UPDATE wallets
SET balance_cents = balance_cents + 5000
WHERE id = 7;

-- The top-up has its own ID, chosen by the caller.
-- A retry inserts nothing: "INSERT 0 0".
INSERT INTO topups (id, wallet_id, cents)
VALUES ('topup-8f2c41', 7, 5000)
ON CONFLICT (id) DO NOTHING;
-- The balance is then SUM(cents) over topups,
-- or is updated only when the insert really happened.

ON CONFLICT (id) DO NOTHING is PostgreSQL for "if a row with this ID already exists, skip this insert instead of failing". Postgres reports INSERT 0 1 when a row was inserted and INSERT 0 0 when it was skipped. The cousin ON CONFLICT … DO UPDATE is an UpsertUpsertInsert a row, or update it if a row with the same key already exists, in one statement. In Postgres: INSERT … ON CONFLICT … DO UPDATE..

3. Only act if the data is still in the state you expect. A Conditional writeConditional writeA write that only happens if the data is still in the state you expect, e.g. UPDATE … WHERE status = 'paid' or WHERE version = 6. A repeat finds the condition false and changes nothing. carries its own precondition:

UPDATE orders SET status = 'shipped'
WHERE id = 81 AND status = 'paid';   -- 1st run: UPDATE 1.  Repeat: UPDATE 0.

The WHERE status = 'paid' makes it a one-way step: once the order is shipped, the condition is false and the repeat does nothing. The same trick with a version number (WHERE version = 6) is Optimistic lockingOptimistic lockingDon't lock while reading. When writing, check that nobody changed the row since you read it (usually with a version number); if someone did, re-read and retry.. (Going further: HTTP has the same idea. An If-Match header carrying an ETagETagA response header holding a version label for the content (often a hash), e.g. ETag: "v42". The client sends it back in If-None-Match to ask “has it changed since this version?”. means "only do this if the resource is still at this version".)

HTTP methods, briefly

HTTP's spec labels GET, HEAD, PUT, DELETE and OPTIONS idempotent, and POST (and usually PATCH) not. HTTP §08 covers the details. Two things worth adding:

• The label is a promise your handler must keep, not something HTTP enforces. A PUT handler that appends to a list, or sends an email every time, breaks the promise, and every proxy that retries PUTs will punish you for it.
• POST is where the money is: creating orders, charging cards, sending messages. You can't make "create a new thing" idempotent by rephrasing it. That's what idempotency keys are for.

At-most-once, at-least-once, and the exactly-once myth

Any system that sends messages (an app calling an API, a service publishing events, a queue handing out jobs) makes one of three promises:

• At-most-once deliveryAt-most-once deliveryEach message is delivered once or not at all: it is never processed twice, but it may be lost (e.g. the network drops it, or the receiver crashes after acknowledging but before handling it) and nobody retries.: send once, never resend. Nothing is ever duplicated, but a lost message is gone for good. Fine for a "user is typing…" indicator; terrible for payments.
• At-least-once deliveryAt-least-once deliveryA delivery promise where a message is never lost but may arrive more than once (for example, re-sent after a crash). Receivers must therefore handle duplicates safely.: keep resending until the receiver confirms. Nothing is lost, but the receiver may get the same message several times.
• Exactly-onceExactly-onceThe (mostly impossible over a network) promise that a message is processed one time, no more, no less. Real systems get the effect with at-least-once delivery plus idempotent processing.: every message arrives once, no more, no less. Everyone wants it. Over an unreliable network, nobody can promise it.

Here's how each behaves when the confirmation gets lost:

Why exactly-once delivery can't exist

The receiver confirms with an acknowledgement (ack)Acknowledgement (ack)A “got it, done” message from a receiver back to the sender (or to a message broker). If no ack arrives in time, the sender assumes the message was lost and sends it again.. But the ack is itself a message, and it can be lost too. So the sender can't tell "message lost" from "ack lost". Could more messages fix that? A famous thought experiment says no:

This is the Two Generals problemTwo Generals problemA thought experiment, proven in the 1970s: two sides that can only talk over a channel that may lose messages can never both be certain they agree. Whoever sent the last message can't know it arrived.. It doesn't depend on how clever the protocol is: whoever sends the last message in any exchange can't know it arrived. The only way to be sure is to never lose messages, and real networks lose them.

So a sender has to pick: resend when unsure (at-least-once: risk duplicates) or don't (at-most-once: risk loss). For anything that matters, you pick at-least-once and deal with duplicates on the receiving side.

"Exactly-once" = at-least-once + dedup

When a product says "exactly-once", read it as "exactly-once effect" (some say "effectively once"). The recipe is always the same:

1. The sender gives every message a unique ID and resends until it gets an ack.
2. The receiver remembers which IDs it has already handled, and skips repeats. That's DeduplicationDeduplicationRecognising that a message or request was already handled (usually by an ID you remember) and skipping it the second time..
3. Step 2 has to be atomic with the work itself: "do the work" and "remember the ID" must succeed or fail together. If they can come apart, a crash in between either loses the work or repeats it (§06 and §08 show how).

"Doesn't TCP already handle this?"

TCPTCPTransmission Control Protocol: delivers a stream of bytes between two programs reliably and in order, resending anything lost. HTTP/1.1, HTTP/2 and most databases run on it. does resend lost packets and drop duplicate ones, but only inside one connection. When a connection dies (Wi-Fi to 4G, a server restart) and your app opens a new one and sends the request again, TCP sees brand-new data. It has no idea it's the same payment.

This is a general principle, the end-to-end argument: lower layers can make duplicates rarer, but only the endpoint that performs the effect knows what "the same request" means. So that endpoint is where deduplication has to live.

Idempotency keys, end to end

§03's tricks work when you can reshape the operation. For "create a payment", you can't: every legitimate payment is new. So the client adds a name to the request. It's usually sent as an HTTP header:

POST /v1/payments HTTP/1.1
Host: api.example.com
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
Content-Type: application/json

{"amount_cents": 5000, "currency": "eur"}

That header is the Idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once.. The value is a random UUIDUUID (v4 / v7)A 128-bit ID like 3f2b…-…. Version 4 is fully random; version 7 starts with a timestamp, so new IDs are roughly in increasing order., so no two intents ever share one by accident. Payment APIs such as Stripe's made this design popular, and many other APIs copy it.

What the server does with it

The server keeps a table: one row per key, holding a Request fingerprintRequest fingerprintA hash of the important parts of a request (method, path, body). Stored next to the idempotency key so the server can spot the same key being reused for a different request, which is a client bug (usually answered with 422). of the request, a status, and the saved response. Each incoming request falls into one of five cases:

The third row is a replayed responseReplayed responseThe saved response from the first time a key was processed, sent back unchanged to every later request with that key. The client can't tell a replay from the original, which is the point.: the client can't tell it from the original, which is what you want. The fourth row handles §02's just slow case, where the retry arrives while the original is still running. The fifth catches a client bug: the same key reused for a different payment.

Try all five. Each request takes about a second on this server, so you have time to double-tap:

Rules for the client

• Create the key when the intent begins, not per attempt. For a checkout, that's when the payment form opens or the order is created, and every retry of that payment reuses it. A new key per attempt is the same as no key at all.
• Survive restarts. If the app crashes and restarts, it should retry with the same key. So save the key alongside the pending action (in the app's storage, or in your database row for outgoing calls).
• Use random keys. A UUID v4, or any long random string. Don't build the key from the request's contents ("user 7, €5"): two real €5 coffees on the same day would collide.
• A new intent gets a new key, even if it looks identical to the last one.

Rules for the server

Scope keys to the caller. Make the table's key (account_id, key), not just key. Two customers who happen to send the same string must not see each other's results. Without the scope, someone who guesses or steals a key could read another user's saved response.

Fingerprint the request. Store a hash of the parts that matter (method, path, body). A Hash functionHash functionA function that turns any input (a string, a file) into a fixed-size number or fingerprint. The same input always gives the same output, and small changes give very different outputs — used to spread keys across buckets or servers, and (with cryptographic hashes like SHA-256) to detect tampering. such as SHA-256 turns any input into a short, fixed-size value that changes completely if one byte changes. If a later request has the same key but a different fingerprint, answer 422. Never silently replay the old answer: the client asked for something else.

Decide which outcomes to save. Save final results: success, and business answers like "card declined", which would be the same on a retry. Don't save outcomes that didn't really happen: a request that failed validation before claiming the key, a 409 for an in-flight duplicate, or a crash. Those should be retryable. (Stripe, for example, documents that it saves the first result once the endpoint has started executing, even a 500, but saves nothing for requests that fail validation or collide with a concurrent one.)

Keep keys long enough, then delete them. A key only has to outlive every retry the client could make, so keep it for longer than your clients' total retry time, with plenty of margin. Stripe documents removing keys after they're at least 24 hours old. A key that comes back after it has been deleted is treated as new, so don't make the window too short.

Building it: Postgres, Go and JavaScript

This section builds the server from §05 for the common case: the work is writes to your own database (create a charge row, move a balance). If the work calls another company's API, you need §07's extra machinery. Start here either way.

Two tables. One holds keys, one is the effect we're protecting:

CREATE TABLE idempotency_keys (
  account_id    bigint      NOT NULL,
  key           text        NOT NULL,
  request_hash  text        NOT NULL,       -- fingerprint of the request body
  response_code int,                        -- filled in when the work is done
  response_body text,                       -- text, not jsonb (see below)
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)             -- one row per key, per account
);

CREATE TABLE charges (
  id           bigserial PRIMARY KEY,
  account_id   bigint NOT NULL,
  amount_cents int    NOT NULL
);

PRIMARY KEY (account_id, key) makes the pair unique: it acts as a UNIQUE constraintUNIQUE constraintA rule on a database column: the database refuses to store a second row with the same value there — even if two inserts arrive at the same instant. over both columns together, and it's the scoping rule from §05 written as a constraint. The database will refuse a second row for the same account and key, even if both inserts arrive in the same microsecond.

The handler's core

The flow, in plain words: start a Database transactionDatabase transactionA group of database changes that happen all together or not at all (COMMIT keeps them, ROLLBACK undoes them). How much other users can see or interfere with the half-done state depends on the isolation level.; try to insert the key; if the insert was skipped, it's a repeat, so read and return the saved response; otherwise do the work, save the response and commit. The Go version uses database/sql (*sql.DB, BeginTx, QueryRowContext), introduced in Transactions §04.

type PayRequest struct {
	AmountCents int `json:"amount_cents"`
}

type Response struct {
	Code int
	Body []byte
}

// pay charges the account at most once per (accountID, key).
// Repeats get the first response back, byte for byte.
func pay(ctx context.Context, db *sql.DB, accountID int64, key string, req PayRequest) (Response, error) {
	reqJSON, err := json.Marshal(req)
	if err != nil {
		return Response{}, err
	}
	sum := sha256.Sum256(reqJSON)
	hash := hex.EncodeToString(sum[:]) // fingerprint of the request body

	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return Response{}, err
	}
	defer tx.Rollback() // does nothing if we committed

	// 1. Claim the key. A concurrent duplicate waits here until we commit.
	var claimed bool
	err = tx.QueryRowContext(ctx, `
		INSERT INTO idempotency_keys (account_id, key, request_hash)
		VALUES ($1, $2, $3)
		ON CONFLICT (account_id, key) DO NOTHING
		RETURNING true`, accountID, key, hash).Scan(&claimed)

	if errors.Is(err, sql.ErrNoRows) { // 2. Seen before: replay.  ← the point
		var storedHash string
		var r Response
		err = tx.QueryRowContext(ctx, `
			SELECT request_hash, response_code, response_body
			FROM idempotency_keys WHERE account_id = $1 AND key = $2`,
			accountID, key).Scan(&storedHash, &r.Code, &r.Body)
		if err != nil {
			return Response{}, err
		}
		if storedHash != hash {
			return Response{Code: 422, Body: []byte(`{"error":"key reused with a different request"}`)}, nil
		}
		return r, nil
	}
	if err != nil {
		return Response{}, err
	}

	// 3. First time: do the work and save the response, in the SAME transaction.
	var chargeID int64
	err = tx.QueryRowContext(ctx,
		`INSERT INTO charges (account_id, amount_cents) VALUES ($1, $2) RETURNING id`,
		accountID, req.AmountCents).Scan(&chargeID)
	if err != nil {
		return Response{}, err
	}
	body := fmt.Appendf(nil, `{"charge_id":%d,"amount_cents":%d}`, chargeID, req.AmountCents)
	_, err = tx.ExecContext(ctx, `
		UPDATE idempotency_keys SET response_code = 201, response_body = $3
		WHERE account_id = $1 AND key = $2`, accountID, key, body)
	if err != nil {
		return Response{}, err
	}
	if err := tx.Commit(); err != nil { // charge + saved response, or neither
		return Response{}, err
	}
	return Response{Code: 201, Body: body}, nil
}

It works: the real run

We ran both versions against PostgreSQL 16: one payment, a retry, a retry with a changed amount, then five requests with the same new key at the same moment (goroutines in Go, Promise.all in JS). The Go output:

first try:             201 {"charge_id":1,"amount_cents":5000}
retry, same key:       201 {"charge_id":1,"amount_cents":5000}
same key, new amount:  422 {"error":"key reused with a different request"}
concurrent #5:         201 {"charge_id":2,"amount_cents":700}
concurrent #1:         201 {"charge_id":2,"amount_cents":700}
concurrent #4:         201 {"charge_id":2,"amount_cents":700}
concurrent #3:         201 {"charge_id":2,"amount_cents":700}
concurrent #2:         201 {"charge_id":2,"amount_cents":700}
charges in the table: 2

Eight requests (seven 201s and one 422), two charges. The JS version prints the same lines (its concurrent ones come out in order #1–#5).

Why the concurrent case just works

There's no lock in our code, yet five simultaneous requests produced one charge. The unique index did it:

1. Request A inserts the key row. A hasn't committed yet, so the row is invisible to everyone else, but the index already knows about it.
2. Request B tries to insert the same key. Postgres can't decide yet whether B conflicts (A might still roll back), so B's insert waits for A's transaction to finish.
3. A commits. Now B's insert definitely conflicts, so DO NOTHING skips it and returns no rows.
4. B runs its SELECT and sees A's saved response. B replays it.

If A had rolled back instead (it crashed, or the charge failed), B's insert would go through and B would do the work itself. Nobody is left stuck behind a dead request.

Wiring it to HTTP

The handler around pay reads the header, validates before claiming the key (so a bad request isn't saved), and takes the account from the login, not the body:

func payHandler(db *sql.DB) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		key := r.Header.Get("Idempotency-Key")
		if key == "" || len(key) > 255 {
			http.Error(w, "an Idempotency-Key header is required", http.StatusBadRequest)
			return
		}
		var req PayRequest
		if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.AmountCents <= 0 {
			http.Error(w, "invalid body", http.StatusBadRequest) // nothing claimed: retryable after a fix
			return
		}
		accountID := accountFrom(r) // from the login session or API token, never from the body
		resp, err := pay(r.Context(), db, accountID, key, req)
		if err != nil {
			// The transaction rolled back, so nothing was charged or saved: a retry will redo it.
			http.Error(w, "internal error", http.StatusInternalServerError)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(resp.Code)
		w.Write(resp.Body)
	}
}

In Go, passing r.Context() means that if the client hangs up mid-request, the context is cancelled, the transaction rolls back, and nothing is saved, so the client's retry starts fresh. That's correct: an unsaved key is always safe to process again.

Side effects you can't roll back: recovery points

§06 put everything in one transaction. That only works while every step is a write to your own database. Real payment flows look more like this: save the request, call a payment provider over the network (another company's API, outside your database), save the result, queue a receipt email, reply.

If the server crashes right after the provider charged the card but before your database recorded it, a ROLLBACK erases your record of the charge, and the card stays charged. The next retry has no idea and charges again.

Phases and recovery points

The fix, popularised by Stripe's engineers, is to treat the request as a small state machineFinite state machine (FSM)A model with a fixed list of states (pending, paid, shipped…) and a fixed list of allowed moves between them. The thing is always in exactly one state, and only listed moves are legal. (a fixed list of states, with only certain moves allowed between them). Each row in the keys table remembers a Recovery pointRecovery pointA saved marker of how far a multi-step request got (started → charged → done). A retry reads it and continues from there instead of redoing finished steps.: started → charged → done. A retry reads the point and continues from there. It doesn't start over.

Each phase is either one local transaction, or one external call that is itself idempotent. Never two external calls in one phase, because a crash between them leaves you unsure which one happened.

Because the key row is now committed early (it has to be, so a retry can find it), we also need a way to say "someone is working on this key right now". That's a LeaseLeaseA permission that expires on its own after a set time unless it is renewed: “you are the leader for the next 10 seconds”. If the holder crashes or is cut off, the lease runs out and someone else can take over. Caches use the same idea for “the right to fill this key”.: a locked_until timestamp. If the process holding it crashes, the lease simply expires and the next retry takes over. Each lease also gets a random Lease tokenLease tokenA random value written when a process takes a lease and checked on every write it makes later (“…AND lease_id = mine”). If the lease expired and someone else took over, the token no longer matches, so the old holder's late writes change nothing. (lease_id), so a process can later check that the lease is still its own. You'll see why that matters below.

CREATE TABLE payment_keys (
  account_id         bigint NOT NULL,
  key                text   NOT NULL,
  request_hash       text   NOT NULL,
  recovery_point     text   NOT NULL DEFAULT 'started',  -- started → charged → done
  locked_until       timestamptz,                         -- when the lease expires
  lease_id           text,                                -- who holds the lease
  provider_charge_id text,
  response_code      int,
  response_body      text,
  PRIMARY KEY (account_id, key)
);

Pick what goes wrong (a crash, or a worker that is merely slow) and step through the original request and the client's retry:

1func pay(key string) Response {
2    row := claimOrRead(key)
3    if row.Hash != hash { return 422 }
4    if row.Point == "done" {
5        return row.Saved // replay
6    }
7    lease := takeLease(key) // or 409
8    if row.Point == "started" {
9        id := provider.Charge(amt,
10            IdemKey(key)) // OUR key
11        fenced(lease, "charged", id)
12    }
13    // here Point == "charged"
14    tx := db.Begin()
15    fenced(tx, lease, "done", resp)
16    tx.Insert(outbox, "receipt")
17    tx.Commit()
18    return resp
19}

The real thing in Go

Here is the full flow. The fake provider in our test program behaves like Stripe: it remembers idempotency keys and returns the existing charge for a repeat.

var (
	errInFlight  = errors.New("409: a request with this key is still in progress")
	errLostLease = errors.New("our lease expired and another worker took over; stopping")
)

type execer interface { // *sql.DB and *sql.Tx both have this method
	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
}

// fenced runs a write that must only apply while we still hold the lease.
// Every such query ends in "AND lease_id = <ours>"; 0 rows means we lost it.
func fenced(ctx context.Context, db execer, query string, args ...any) error {
	res, err := db.ExecContext(ctx, query, args...)
	if err != nil {
		return err
	}
	if n, _ := res.RowsAffected(); n == 0 {
		return errLostLease
	}
	return nil
}

func payExternal(ctx context.Context, db *sql.DB, prov *Provider, acct int64, key, hash string, cents int) (int, string, error) {
	// Phase 0: claim the key if it's new, then read it.
	_, err := db.ExecContext(ctx, `INSERT INTO payment_keys (account_id, key, request_hash)
		VALUES ($1, $2, $3) ON CONFLICT DO NOTHING`, acct, key, hash)
	if err != nil {
		return 0, "", err
	}
	var point, storedHash string
	var code sql.NullInt64
	var body sql.NullString
	err = db.QueryRowContext(ctx, `SELECT recovery_point, request_hash, response_code, response_body
		FROM payment_keys WHERE account_id = $1 AND key = $2`, acct, key).
		Scan(&point, &storedHash, &code, &body)
	if err != nil {
		return 0, "", err
	}
	if storedHash != hash { // checked BEFORE the lease, so a bad request can't block a good one
		return 422, `{"error":"key reused with a different request"}`, nil
	}
	if point == "done" { // finished: replay, no lease needed
		return int(code.Int64), body.String, nil
	}

	// Take the lease, with a fresh token that only we know.
	lease := rand.Text() // crypto/rand, Go 1.24+
	var chargeID sql.NullString
	err = db.QueryRowContext(ctx, `UPDATE payment_keys
		SET lease_id = $3, locked_until = now() + interval '30 seconds'
		WHERE account_id = $1 AND key = $2 AND recovery_point <> 'done'
		  AND (locked_until IS NULL OR locked_until < now())  -- free, or the holder's lease ran out
		RETURNING recovery_point, provider_charge_id`, acct, key, lease).Scan(&point, &chargeID)
	if errors.Is(err, sql.ErrNoRows) {
		return 0, "", errInFlight // someone holds a live lease (or just finished: a retry will replay)
	}
	if err != nil {
		return 0, "", err
	}
	// We hold the lease now. Finish our writes even if the client hangs up (that cancels
	// r.Context()); otherwise a lease release could silently fail and block retries for 30 s.
	ctx = context.WithoutCancel(ctx)

	for {
		switch point {
		case "started": // Phase 1: the external call. Pass OUR key downstream.  ← the point
			pctx, cancel := context.WithTimeout(ctx, 10*time.Second) // well under the 30 s lease
			id, err := prov.Charge(pctx, fmt.Sprintf("%d:%s", acct, key), cents)
			cancel()
			if errors.Is(err, ErrCardDeclined) { // a final business answer: save it like a success
				resp := `{"error":"card declined"}`
				err = fenced(ctx, db, `UPDATE payment_keys SET recovery_point = 'done',
					response_code = 402, response_body = $4, locked_until = NULL, lease_id = NULL
					WHERE account_id = $1 AND key = $2 AND lease_id = $3`, acct, key, lease, resp)
				if err != nil {
					return 0, "", err
				}
				return 402, resp, nil
			}
			if err != nil { // transient (timeout, 5xx): free the lease so a retry can go at once
				_, rerr := db.ExecContext(ctx, `UPDATE payment_keys SET locked_until = NULL
					WHERE account_id = $1 AND key = $2 AND lease_id = $3`, acct, key, lease)
				return 0, "", errors.Join(err, rerr) // report both if the release failed too
			}
			err = fenced(ctx, db, `UPDATE payment_keys SET recovery_point = 'charged', provider_charge_id = $4
				WHERE account_id = $1 AND key = $2 AND lease_id = $3`, acct, key, lease, id)
			if err != nil {
				return 0, "", err // errLostLease: someone else owns this key now, so touch nothing
			}
			point, chargeID = "charged", sql.NullString{String: id, Valid: true}

		case "charged": // Phase 2: local side effects + response, in ONE transaction.
			resp := fmt.Sprintf(`{"charge":"%s","amount_cents":%d}`, chargeID.String, cents)
			tx, err := db.BeginTx(ctx, nil)
			if err != nil {
				return 0, "", err
			}
			err = fenced(ctx, tx, `UPDATE payment_keys SET recovery_point = 'done',
				response_code = 201, response_body = $4, locked_until = NULL, lease_id = NULL
				WHERE account_id = $1 AND key = $2 AND lease_id = $3`, acct, key, lease, resp)
			if err == nil {
				_, err = tx.ExecContext(ctx, `INSERT INTO outbox (topic, payload)
					VALUES ('receipt-email', $1)`, resp)
			}
			if err == nil {
				err = tx.Commit()
			}
			if err != nil {
				tx.Rollback()
				return 0, "", err
			}
			return 201, resp, nil
		}
	}
}

Three decisions in that code are easy to get wrong:

• Check the body before taking the lease. If the 422 check came after the lease, a wrong request would grab a 30-second lease and leave without releasing it, and the client's real retry would get 409 until it ran out.
• A decline is an answer; a timeout is not. "Card declined" is final, so it is saved as done with a 402, and every retry replays it (§05). A timeout or 5xx from the provider says nothing about what happened, so we free the lease and return an error. The next retry runs phase 1 again, and the provider's own dedup keeps it to one charge.
• Don't let a hang-up stop you halfway. Once we hold the lease, the code switches to context.WithoutCancel(ctx). If the client disconnects, Go cancels r.Context(), and a lease release using it would fail with context canceled, so every retry would get 409 for 30 s. (We checked: without WithoutCancel, the "client hangs up" run below gets exactly that 409.) The provider call is still capped by its own 10 s timeout.
• Every phase write is fenced. It only applies AND lease_id = <ours>. Why is the next callout.

We ran it with Fault injectionFault injectionDeliberately breaking things in a test or staging environment (dropping replies, killing a process mid-request, adding delay) to check that the system handles real failures correctly.: a switch that makes the process "crash" (return an error before finishing) at a chosen line, or stall in the middle of a phase. After each crash, the test expires the lease (as 30 real seconds would) and retries:

== no crash
  try 1: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  try 2: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  try 3: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  provider charges: 1, receipt emails queued: 1, final point: done
== crash after the provider charged
  try 1: 0  💥 crashed after the provider charged
  try 2: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  try 3: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  provider charges: 1, receipt emails queued: 1, final point: done
== crash before the final commit
  try 1: 0  💥 crashed before the final commit
  try 2: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  try 3: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  provider charges: 1, receipt emails queued: 1, final point: done
== P1 is slow: its lease expires mid-charge
  P2 (retry, takes over): 201 {"charge":"ch_1","amount_cents":5000} <nil>
  P1 (wakes up late): 0  our lease expired and another worker took over; stopping
  provider charges: 1, receipt emails queued: 1, final point: done
== wrong body on a key nobody holds, then the real retry
  try 1: 0  provider timeout
  wrong body: 422 {"error":"key reused with a different request"} <nil>
  real retry: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  provider charges: 1, receipt emails queued: 1, final point: done
== card declined
  try 1: 402 {"error":"card declined"} <nil>
  try 2: 402 {"error":"card declined"} <nil>
  provider charges: 0, receipt emails queued: 0, final point: done
== client hangs up during a provider timeout
  try 1: 0  provider timeout
  retry right away: 201 {"charge":"ch_1","amount_cents":5000} <nil>
  provider charges: 1, receipt emails queued: 1, final point: done
== concurrent duplicate
  try: 0  409: a request with this key is still in progress

One charge and at most one email in every scenario. Our first drafts had two bugs that tests like these catch. First, replays of a done key also took the lease and never released it, so the third try got a 409; that's why done rows now replay before any lease is taken. Second, and worse, is the slow-worker case in the next callout. Test your crash paths; they're the code that never runs on a good day.

Side effects outside your database

Pass your key downstream. If the provider accepts idempotency keys (Stripe, Adyen, PayPal and many others do), send one derived from yours, like "42:k1" above. Then a repeated phase 1 costs nothing. This is the single most important line in the function.

Emails, push notifications, webhooks: use the outbox. Don't send them inline. Insert a row into an outbox table in the same transaction as the state change, and let a separate worker send them (Behavioral patterns §10 and Transactions §10 cover the Outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does. pattern). The worker itself delivers at-least-once, so a crash after sending but before marking the row sent can send the email twice. Pass the outbox row's ID to the email provider if it supports dedup; otherwise accept the rare duplicate. A duplicate email is annoying; a duplicate charge is a support ticket.

When a provider has no idempotency support, you have weaker options, best first:

1. Let the provider check for you. Many APIs let you attach your own reference (an order ID) and reject a second object with the same reference, or let you create objects with an ID you choose. That's idempotency under a different name.
2. Look before you retry. Before redoing phase 1, search the provider for an object carrying your reference. It narrows the window but doesn't close it: the first attempt may still be in flight (§02).
3. Reconcile. Run a periodic ReconciliationReconciliation (data)A periodic job that compares your records with another system's (e.g. your orders vs the payment provider's charges) and flags or fixes any differences. The safety net for whatever retries and keys missed. job that compares your records with the provider's (most offer a daily report) and flags mismatches, such as a charge with no matching order, for refund or review.
4. Escalate unknowns to a person. For high-value, rare operations (a wire transfer), "outcome unknown, a human will check" is a perfectly good final state.

Idempotent consumers: dedup for queues and events

So far the duplicates came from clients retrying HTTP requests. The same thing happens, constantly, inside systems built on a Message brokerMessage brokerA server that accepts messages from producers and hands them to consumers later — RabbitMQ, Kafka, SQS, NATS. It decouples who sends from who handles, and buffers bursts.: a server (Kafka, RabbitMQ, Amazon SQS…) that stores messages from producers and hands them to consumers. Queues & Streams covers how brokers work. Here we only need one fact: a broker resends a message until the consumer acknowledges it.

Where duplicates come from

• The consumer crashed after doing the work, before its ackAcknowledgement (ack)A “got it, done” message from a receiver back to the sender (or to a message broker). If no ack arrives in time, the sender assumes the message was lost and sends it again.. The broker never heard "done", so it redelivers. This is §02's lost reply, in a different costume.
• Kafka consumers restart from their last committed offsetOffset (Kafka)A message's position number in a Kafka partition (0, 1, 2…). A consumer “commits” the offset it has finished; after a crash it restarts from the last committed offset, so anything after it is delivered again.. Offsets are usually committed every few seconds, so a crash, a deploy or a rebalanceRebalance (Kafka)Kafka moving partitions between the consumers of a group, e.g. when one joins, leaves or crashes. Messages whose offsets weren't committed yet are delivered again to the new owner. (Kafka moving partitionsPartition (Kafka)One ordered slice of a topic. Messages with the same key always land in the same partition, so they stay in order relative to each other., the topic's sub-queues, between consumers) replays everything processed since the last commit.
• SQS's Visibility timeoutVisibility timeout (SQS)When an Amazon SQS consumer receives a message, the message is hidden from other consumers for this long. If the consumer doesn't delete it in time (because it crashed or was slow), the message reappears and is delivered again. ran out. If processing takes longer than the timeout, the message reappears and a second consumer starts on it while the first is still working. That's a concurrent duplicate, like §05's double-tap.
• The producer retried its publish. Its send timed out, but the broker had stored the message. Now there are two copies, each with its own broker-assigned ID.

That last point decides which ID to deduplicate on: one the producer puts in the message (an event ID, or the ID of the business action), the same in every copy. The broker's own delivery ID or offset differs between the two copies of a duplicate publish.

The dedup table

Give every consumer a table of message IDs it has finished. The key is (consumer, message_id), so two different consumers of the same message each process it once:

CREATE TABLE processed_messages (
  consumer     text NOT NULL,
  message_id   text NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, message_id)
);

type Message struct {
	ID       string // set by the PRODUCER; the same on every redelivery
	WalletID int64
	Cents    int64
}

// handle applies a "top up" message at most once, however many times it's delivered.
func handle(ctx context.Context, db *sql.DB, msg Message) error {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer tx.Rollback()

	res, err := tx.ExecContext(ctx, `
		INSERT INTO processed_messages (consumer, message_id) VALUES ('wallet-topups', $1)
		ON CONFLICT DO NOTHING`, msg.ID)
	if err != nil {
		return err
	}
	if n, err := res.RowsAffected(); err != nil {
		return err
	} else if n == 0 {
		return nil // duplicate: already applied. Ack it and move on.  ← the point
	}

	_, err = tx.ExecContext(ctx,
		`UPDATE wallets SET balance_cents = balance_cents + $1 WHERE id = $2`, msg.Cents, msg.WalletID)
	// (Production code would also check RowsAffected() == 1: a missing wallet is a bug to report, not a duplicate.)
	if err != nil {
		return err
	}
	return tx.Commit() // the effect and the "seen it" mark commit together
}
// The caller acks the message only when handle returns nil.

We fed five deliveries to each version: evt-1, evt-2, evt-1 again (its ack was "lost"), evt-2 again, and evt-3. Both printed:

5 deliveries, 3 distinct messages → balance: 1000

That's 500 + 300 + 200. Without the dedup table, it would have been 1800.

Ack after commit, never before

The order of the last two steps decides your delivery promise:

The third row is the sneaky one. A dedup mark in Redis and an effect in Postgres can't commit together: whichever is written first, a crash in between leaves them out of step. Keep the mark in the same database, and the same transaction, as the effect.

When you don't need a dedup table

The §03 tricks work for consumers too, and they need no extra table:

• Upsert by a natural key. "Set user 7's email to x" or "insert order o-81 if it isn't there" are already idempotent.
• Versions. If each event carries a version number that goes up with every change to the same thing, apply it only if it's newer than what you have. That also handles events arriving out of order, which plain dedup doesn't:

-- product 1 is at version 4. Events arrive: v6, then v5 (late), then v6 again (duplicate).
UPDATE products SET price_cents = 1500, version = 6 WHERE id = 1 AND version < 6;  -- UPDATE 1
UPDATE products SET price_cents = 1200, version = 5 WHERE id = 1 AND version < 5;  -- UPDATE 0
UPDATE products SET price_cents = 1500, version = 6 WHERE id = 1 AND version < 6;  -- UPDATE 0
-- final: price 1500, version 6 ✓

This works when each event carries the full new state ("price is now 1500"). For events that describe a change ("add 500 to the balance"), skipping an older version would lose money: use the dedup table.

Outbox in, inbox out

Put both halves together and you get effectively-once processing across services. The producer writes its event to an Outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does. table in the same transaction as its own change, and a relay publishes it, at-least-once, with an event ID. The consumer records that ID in its dedup table, which in this role is called the inboxInbox / processed-messages tableA table where a consumer records the ID of every message it has already handled, in the same transaction as the work. A redelivered message finds its ID there and is skipped., in the same transaction as its effect. Duplicates are expected at every hop and harmless at the end. Sagas build multi-step business flows out of exactly this.

Retrying well: what, when, and how much

Idempotency makes retries safe. This section makes them smart. The basic backoff loop is already in Async / Await §11; here we add what changes when the request has side effects and a key.

What to retry

A Transient errorTransient errorA failure that's likely to go away by itself if you wait a bit: a timeout, a dropped connection, 503 “overloaded”, 429 “slow down”. Worth retrying. The opposite of a permanent error like 400 Bad Request. is one that may go away by itself. Those are worth retrying. A permanent error will come back exactly the same, so retrying it just wastes time and load.

gRPC §09 has the same table for gRPC status codes (UNAVAILABLE yes, INVALID_ARGUMENT no).

A retry loop for requests with side effects

import { setTimeout as sleep } from "node:timers/promises";

// 409 is here because OUR server (§06, §07) only uses it for "this key is still in progress".
// If yours also uses 409 for business conflicts, check the error code in the body instead.
const RETRYABLE = new Set([408, 409, 429, 500, 502, 503, 504]);

// Retry-After is either seconds ("2") or an HTTP date. Returns ms, or null if absent.
function retryAfterMs(res) {
  const v = res?.headers.get("retry-after");
  if (!v) return null;
  const secs = Number(v);
  return Number.isFinite(secs) ? secs * 1000 : Math.max(0, Date.parse(v) - Date.now());
}

// Pass in the key that was created when the intent began; every attempt reuses it.
async function postWithRetry(url, body, { key = crypto.randomUUID(), deadlineMs = 10_000, maxAttempts = 5 } = {}) {
  const deadline = AbortSignal.timeout(deadlineMs); // ONE time budget for every attempt
  const deadlineAt = Date.now() + deadlineMs;

  for (let attempt = 1; ; attempt++) {
    let res = null, error = null;
    try {
      res = await fetch(url, {
        method: "POST",
        headers: { "Content-Type": "application/json", "Idempotency-Key": key }, // same key every attempt ← the point
        body: JSON.stringify(body),
        signal: AbortSignal.any([deadline, AbortSignal.timeout(2000)]), // per-try cap too
      });
    } catch (err) {
      if (deadline.aborted) throw err; // out of total time: stop
      error = err;                     // network error or per-try timeout: outcome UNKNOWN
    }
    if (res && !RETRYABLE.has(res.status)) return res; // success, or a final "no" (400, 422…)

    const backoff = Math.min(5000, 200 * 2 ** attempt);
    const wait = retryAfterMs(res) ?? Math.random() * backoff; // server's hint wins; else full jitter
    if (attempt === maxAttempts || wait > deadlineAt - Date.now()) { // no tries left, or no time to wait
      if (res) return res;
      throw error;
    }
    await res?.body?.cancel(); // throw away the failed reply's body
    await sleep(wait, undefined, { signal: deadline }); // rejects at once if the deadline passes
  }
}

We pointed both at a test server that fails on purpose: first 503, then 429 with Retry-After: 1, then it charges and drops the connection without replying, then (on the next attempt) replays the saved response. The JS log, with one console.log added before each wait:

server: /payments attempt 1 key=9dc86581…
client: attempt 1 → 503; waiting 301 ms
server: /payments attempt 2 key=9dc86581…
client: attempt 2 → 429; waiting 1000 ms
server: /payments attempt 3 key=9dc86581…
server: charged, now dropping the connection
client: attempt 3 → UND_ERR_SOCKET; waiting 653 ms
server: /payments attempt 4 key=9dc86581…
client: final 201 true {"charge_id":1}

Four attempts, one charge, and the last line shows the server's Idempotent-Replayed: true header. In Node, the dropped connection surfaces as TypeError: fetch failed with err.cause.code === "UND_ERR_SOCKET". Two more runs, in both languages: an endpoint that answers 409 (in progress) once and then 201 was retried after a short wait and succeeded, and a 429 with Retry-After: 60 against our 10-second deadline gave up at once instead of sleeping for a minute and then failing.

The Go run (also with a log line before each wait) had a surprise: the server again logged four attempts, but the Go client only logged two waits. Go's HTTP client had resent attempt 3 by itself on a fresh connection. Its transport treats a request carrying an Idempotency-Key header as safe to replay when a reused keep-alive connection dies before any reply, so our loop never saw that failure. When we renamed the header to X-Key, the failure came back to our code as EOF. One more reason the server must dedup: retries happen in layers you didn't write.

How long to wait

The server's hint wins. If a 429 or 503 carries Retry-After, wait that long. The server knows when it will have room; your guess doesn't.

Otherwise, exponential Exponential backoffExponential backoffWaiting longer after each failed retry, typically doubling: 1 s, 2 s, 4 s, 8 s… up to a cap. It gives a struggling server room to recover instead of hammering it. with full JitterJitterA small random amount added to a delay or timer — a retry wait, a reconnect, a connection's maximum age — so that many clients don't all act at the same instant and stampede a server.: pick a random wait between 0 and a cap that doubles each attempt. The randomness spreads clients out so they don't all come back in the same instant, which Async / Await §11 shows with a chart.

One deadline for everything. The user waits at most, say, 10 s in total. That's one budget shared by every attempt and every wait, not 10 s per attempt. Both versions above stop the moment it's spent, even in the middle of a wait, and don't start a wait that would outlast it. In Go, the DeadlineDeadlineA point in time ("by 12:00:01.500") after which a call is abandoned. Unlike a per-step timeout, one deadline is shared by every step and every service working on the same request. rides along in the ctx (and across services, too: gRPC §08).

Retry storms: when retries cause the outage

One retry is cheap insurance. Retries at every layer multiply. If the app makes 3 attempts, the gateway makes 3 per request it receives, and the service makes 3 per database call, then a dead database receives 3 × 3 × 3 = 27 queries for one tap, exactly when it can least afford them. That's a Retry stormRetry stormWhen a struggling service gets even more traffic because every caller retries its failed requests — often at several layers at once — so a small slowdown becomes a full outage.. Play with the numbers:

Notice the two sides. At a 5% failure rate, three attempts hide almost every failure from users for a few percent of extra load: that's what retries are for. At 100%, retries can't help anyone, and they bury the database under several times its normal load the moment it tries to come back.

The defences (Load Balancing §09 and Worker Pools §07 cover them from the server's side):

1. Retry at one layer, usually the one closest to the user or to the failure. Everyone else fails fast and passes the error up.
2. Cap retries with a Retry budgetRetry budgetA cap on how much extra load retries may add (say, 10% of normal traffic), so retries can't multiply an outage.. Retries may add at most ~10% to normal traffic. When the budget is spent, stop retrying and return the error. A tiny version:

// Every first attempt earns `ratio` tokens; every retry spends 1.
class RetryBudget {
  #tokens;
  constructor(ratio = 0.1, most = 10) {
    this.ratio = ratio;
    this.most = most;
    this.#tokens = most; // a small starting credit, also the cap
  }
  onRequest() { this.#tokens = Math.min(this.most, this.#tokens + this.ratio); } // per first attempt
  tryRetry() {                                                                    // before each retry
    if (this.#tokens < 1) return false;
    this.#tokens -= 1;
    return true;
  }
}

In a simulated full outage (1,000 requests, every attempt failing, each wanting 2 retries), both versions allowed 109 retries instead of 2,000: about 10% extra load, as designed. #tokens in the JS version is a private field: only code inside the class can see it. gRPC has the same idea built in as retryThrottling (gRPC §09).

3. Back off and respect Retry-After, as above, so the retries that do happen are spread out.
4. Stop calling a service that keeps failing. A circuit breaker counts recent failures, and after too many it skips the call entirely (failing instantly) for a cool-down period, then lets a few test requests through. It's the retry budget's bigger sibling.

Staff-level: the contract, the cost, the regions, the proof

1 · Make it a contract

Callers can only use keys well if you tell them exactly how they work. Document, per endpoint:

• Which endpoints accept a key, and whether it's required. For anything that moves money or sends messages, require it: 400 without one.
• The header name and format. Idempotency-Key, a string up to 255 characters, UUID recommended. An IETF draft ("The Idempotency-Key HTTP Header Field", from the HTTP APIs working group) standardises this header; it was still a draft when this was written, but it's close to what payment APIs already do.
• How long keys are kept (e.g. 24 hours) and what happens after: the same key is then treated as new.
• The replies. 409 for "still in progress, retry later", 422 for "key reused with a different request" (both match the draft), and a header marking replays. Stripe uses Idempotent-Replayed: true.
• What is saved. Whether a 500 or a validation error is saved and replayed, or retryable with the same key (§05).

Put retries and keys in your SDK. Most callers won't implement §09 correctly. If you ship a client library, have it generate the key once per call and retry with backoff by default. Stripe's official libraries, for example, add a key automatically when they retry. The correct path should be the lazy one.

Webhooks you send are the mirror image. You'll redeliver them (at-least-once), so put a stable event ID in every one and tell receivers to deduplicate on it (§08).

2 · Budget for the key table

A worked example, with round numbers. Say an API handles 2,000 keyed writes per second, keeps keys 24 hours, and each row (key, hash, status, a small saved response, index entry) takes about 1 KB:

• 2,000/s × 86,400 s = about 173 million rows per day.
• × 1 KB ≈ 170 GB of keys at any moment, plus churn: 173 million inserts and 173 million deletes every day.

That's a real table. Ways to keep it manageable:

• Drop whole days instead of deleting rows. PartitionTable partitioningPostgres storing one big table as several sub-tables, e.g. one per day. Queries still see one table, but an old day can be removed by dropping its sub-table: one quick command instead of a slow DELETE. the table by creation day (Postgres stores each day in its own sub-table) and drop partitions older than the window. Deleting millions of rows one by one leaves dead space the database must clean up later; dropping a partition is instant.
• Store less. Keep the status code and the created resource's ID, not a large response body, if your API can rebuild the response from those. (You lose "byte for byte"; decide if callers need it.)
• Keep the window honest. It must cover the longest retry any client could make, including retries after an app restart. Shrinking it to save disk quietly turns late retries back into duplicates.

3 · Multiple regions

Suppose you run in two regions (say Europe and the US), each with its own database, replicating to each other asynchronously. A client's first attempt lands in Europe and times out; the retry is routed to the US (DNS failover, a load balancer, a moving phone). The US table hasn't received Europe's key row yet, because of replication lag. Both regions process the payment.

Options, roughly in order of how often they're used:

1. Give every account a Home regionHome regionIn a system running in several geographic regions, the one region that owns a given piece of data (e.g. all of customer 42's keys). Requests for it are routed there so two regions never process the same key at once.. Route all of an account's writes, and therefore all its keys, to that one region. A retry that lands elsewhere is forwarded, not processed.
2. Use a strongly consistent, multi-region store for keys (Spanner, CockroachDB and similar). Every claim then pays a cross-region round trip, often 50–150 ms. CAP explains why that trade-off can't be avoided.
3. Accept rare duplicates and reconcile. For low-value effects, detect doubles afterwards and refund them automatically.

4 · Prove it: break things on purpose

Idempotency code only matters on bad days, so ordinary tests never exercise it. Fault injectionFault injectionDeliberately breaking things in a test or staging environment (dropping replies, killing a process mid-request, adding delay) to check that the system handles real failures correctly. manufactures the bad day. The most useful fault is the one from §01: do the work, then lose the reply. Wrap your handler in staging (never production) like this:

// dropReplies runs the real handler, then throws its reply away for a share of
// requests. The client sees a dropped connection: exactly a lost response.
func dropReplies(next http.Handler, share float64) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if rand.Float64() >= share {
			next.ServeHTTP(w, r)
			return
		}
		next.ServeHTTP(httptest.NewRecorder(), r) // side effects happen; the reply goes nowhere
		panic(http.ErrAbortHandler)               // abort: the server closes the connection
	})
}

With share set to 1, a plain POST to the wrapped Go server failed on the client with EOF, and the JS one with fetch failed (cause.code: "UND_ERR_SOCKET"), while the handler's counter showed the work was done. That's the failure your clients must survive. Then add:

• A duplicating client in integration tests: send every mutating request twice, sometimes at the same moment, and assert that the effect happened once.
• Crash points like §07's: kill the process between phases and check that a retry finishes the job.
• Invariant checks that run after the test, or nightly against production: "at most one charge per key", "every charge has an order". These are cheap SQL queries and catch what the other tests missed.

5 · Watch it

Also log the idempotency key on every line a request writes, the same way you'd log a Request IDRequest IDA unique ID given to each incoming request and printed in every log line that request causes, so you can find all of one request's logs among thousands.. When a customer says "I was charged twice", the key is how you find both attempts in seconds.

Pitfalls gallery

-- two duplicates can BOTH see "no row"…
SELECT 1 FROM idempotency_keys
WHERE account_id = $1 AND key = $2;
-- (none) → do the work…
INSERT INTO idempotency_keys …;

-- claim first; the UNIQUE index decides
INSERT INTO idempotency_keys …
ON CONFLICT DO NOTHING
RETURNING true;
-- a row back = you own it

Check yourself

Predict the answer before you click. Every one is covered above: Q1 in §03, Q2 in §05, Q3 in §06, Q4 in §09, Q5 and Q6 in §08.

Cheat sheet