Design a Payment System

The 30-second version

We'll design the payments side of a marketplaceMarketplaceA platform where many independent sellers sell to buyers (think Etsy or an app store). The buyer pays the platform, and the platform pays each seller their share minus a fee.: buyers pay the platform, the platform keeps a fee, and sellers get the rest. Our system never talks to banks or card networks directly. It asks a payment service providerPayment service provider (PSP)A company such as Stripe, Adyen or PayPal that talks to card networks and banks for you. Your system asks the PSP to charge a card and learns the result from its API response or a webhook. (PSP), such as Stripe or Adyen, to charge the card, and the PSP tells us what happened.

That sounds like one API call. It isn't, because any network call can fail halfway. The PSP may have charged the card even though our request timed out. Its notifications may arrive late, twice or in the wrong order. Our own server may crash between "the PSP said yes" and "we wrote that down". Each of those, handled carelessly, means a customer charged twice or a seller never paid.

Here is the whole design we'll build, one piece per section. Click a box to see what it does and where it's explained:

The four ideas that do the work

The fourth idea is the safety net under the other three: reconciliation (§09). Every day, compare what we think happened with what the PSP and the bank say happened, and chase every difference.

Requirements: what must never go wrong

As in System Design Method, we start by pinning down what the system must do (functional requirements) and how well it must do it (non-functional requirements).

Functional: what it does

1. Checkout. A buyer pays for an order with a card. The order can contain items from one seller (we keep it to one seller per order to stay simple).
2. Record every money movement: the charge, the platform's fee, the seller's share, the PSP's processing fee.
3. Pay sellers. Once an order is delivered, the seller's share becomes payable. A daily payoutPayoutSending a seller the money the platform owes them, by bank transfer, usually on a schedule (daily, weekly) and only for orders that are final. sends each seller what they're owed.
4. Refunds. Full or partial, for any captured payment, never more than was paid.
5. Show status: the buyer sees "paid" or "failed"; the seller sees what they're owed; finance can see every balance and its history.

Out of scope for this page: currencies other than one (we'll say where a second one changes things), subscriptions, and the fraud-scoring models themselves.

Non-functional: how well

The last row is a real trade-off. A news feed that shows a slightly stale post is fine; a payment system that "probably" charged someone is not. When the two pull in different directions, correctness wins.

What we deliberately don't do: handle card numbers

Our servers never see a card number. The PSP's payment form (a small piece of the PSP's page embedded in our checkout) collects the number straight from the buyer's browser and hands us a token that stands for it. That's tokenizationTokenization (cards)The PSP's payment form collects the card number directly from the buyer's browser and gives your server a token (like pm_1Nq…) that stands for the card. You store and send the token; the real card number never reaches your servers..

{ "order_id": "ord_81", "payment_method": "pm_1NqZ8x...", "amount_cents": 10000, "currency": "EUR" }

Why go to that trouble? Anyone who stores, processes or transmits card numbers must comply with PCI DSSPCI DSSThe security standard set by the card industry for anyone who stores, processes or transmits card numbers. The less card data touches your systems, the fewer of its requirements apply to you., a long list of security requirements. If card numbers never touch your systems, far fewer of those requirements apply to you, though not zero: you still have to protect the checkout page that loads the PSP's form.

Rough numbers: modest load, data kept for years

A back-of-the-envelopeBack-of-the-envelope estimateA rough calculation with round numbers ("100 million users × 10 requests a day ≈ 12,000 QPS") done in a minute to see whether a design needs one server or a thousand. Being within 10× is the goal; precision comes later. estimate uses round numbers to find out roughly how big the problem is. (System Design Method teaches the method; here we just apply it.) All numbers below are approximate and made up for the example.

Worked example: a large marketplace

Assume 1 million orders a day. That's a big marketplace. Remember the shortcut: a day has 86,400 seconds, which we round to 100,000 (10⁵).

What the numbers tell us:

• 100 payments a second is not a lot. A single PostgreSQL server on decent hardware can usually take a few thousand small writes a second (measure yours). We don't need to split the database across machines on day one.
• The PSP is the slow part. A charge call usually takes hundreds of milliseconds, sometimes seconds, and it's another company's system. Our design has to tolerate it being slow or briefly down, not outrun it.
• The ledger only grows. Financial records must be kept for years. How long depends on the country and the kind of record, often somewhere between 5 and 10 years, so check with whoever handles legal matters for you. At about 1 TB a year, we'll want to split the table by month (§11) long before it's a problem.

Your turn

Change the inputs. Try a small shop (10,000 orders a day) or a giant one, and see that even 100× more orders stays within reach of a few database servers. Storage is the number that keeps climbing.

The simplest design that works

The pieces

• Payments service. Our API. It takes POST /payments from the checkout page, talks to the PSP, and serves status pages.
• PostgreSQL. One database holding the payments table, the ledger, the idempotency keys and an outbox (§10). Putting them together means one ACID transaction can update a payment, write its ledger entry and queue its "order paid" message: all of them happen, or none do.
• The PSP. Charges cards, then later sends the money to our bank account.
• Webhook receiver. The PSP notifies us of events (charge succeeded, refund done, money settled, dispute opened) by calling our URL. That's a webhookWebhookAn HTTP request one service sends to a URL another registered, to say something happened (order.paid). It's a reversed API call. When you receive webhooks (from a payment provider, say), expect them late, twice or out of order, and verify their signature..
• Background jobs. A recovery job that resolves payments stuck in "unknown" (§10), a payout job that pays sellers daily, and a reconciliation job that checks our books against the PSP's and the bank's (§09).

The happy path, step by step

1. The buyer opens the checkout page. The page creates an idempotency key for this checkout (§05).
2. The PSP's embedded form turns the card into a token. The page sends POST /payments with the order, the amount, the token and the key.
3. Transaction 1: our service inserts a payments row with status created and the key, and commits. The database refuses a second row with the same key, so a repeated request finds this row instead of creating another.
4. Transaction 2: it updates the row to unknown (and updated_at to now) and commits. This says "we're about to call the PSP; from now on, it may have charged".
5. Outside any transaction, it asks the PSP to charge the card, passing a key of its own: the payment ID pay_81.
6. Transaction 3: the PSP answers "succeeded". Our service locks the row, moves it to captured through the state machine (§06), writes the ledger entry, and queues an "order paid" message. All three, or none.
7. A little later the PSP sends a webhook saying the same thing. We notice we already know, and do nothing.
8. Days later, the PSP pays the money into our bank account (settlementSettlementThe PSP actually paying the money it collected into your bank account, usually in one batch per day, a few days after the charges, with its own fees already taken off.), the order is delivered, and the seller's share goes out with the next payout.

Why two commits before the call? Because the two states mean different things after a crash. created means "the PSP was never asked", so it's safe to try again from scratch. unknown means "the PSP may have acted", so we must ask it before deciding anything. And why is the PSP call outside any transaction? A transaction held open during a slow network call keeps its locks the whole time, and a crash would roll back the only record that a charge might exist.

This order is the most important idea on the page, so try breaking it:

Card payments can happen in two steps: authorize (put a hold on the buyer's money) and later capture (actually take it). That's authorize / captureAuthorize / captureCard payments in two steps. Authorize puts a hold on the money (it can be voided for free); capture actually takes it. Holds expire if never captured, usually after some days.. Our design authorizes and captures in one call at checkout, because marketplace deliveries can take longer than a hold lasts. The state machine in §06 still has both states, for flows that split them.

Money is an integer

Every amount in this design is a whole number of the currency's smallest unit: 10000 cents, never 100.00 euros. That's called minor unitsMinor units (money)Money counted in the currency's smallest unit, as a whole number: 4850 cents instead of 48.50 dollars. Integers add up exactly; floating-point numbers don't (0.1 + 0.2 = 0.30000000000000004).. Floating-point numbers can't store most decimal fractions exactly, and a ledger has to add up to the cent. Numbers in Computers: Money shows the bugs floats cause and how to round when you split amounts.

Two more rules follow from that:

• Store the currency next to every amount. 10000 means €100.00 in euros but ¥10,000 in yen, which has no minor unit at all. An amount without a currency is a bug waiting to happen.
• Use 64-bit integers. In PostgreSQL that's bigint; in Go, int64. JavaScript's normal number is only exact up to about 9 × 10¹⁵, which is plenty for one payment but not for summing a big ledger, so the JS code on this page uses BigInt (written 10000n) for sums.

Idempotency, from the click to the PSP

Quick reminder from Idempotency & Retries: an operation is idempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly. if doing it twice has the same effect as doing it once. "Charge €100" isn't. You make it safe with an idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once.: a random ID the client sends with the request, so the server can spot a repeat. §05 there covers the protocol in full; here we apply it to all three hops of a checkout.

Hop 1: browser → our API

The checkout page creates the key when the checkout starts (a random UUIDUUID (v4 / v7)A 128-bit ID like 3f2b…-…. Version 4 is fully random; version 7 starts with a timestamp, so new IDs are roughly in increasing order.), keeps it, and sends it on every attempt to pay for this order. A double click, a page that retries after a network error, or a user who taps "Pay" again: all send the same key.

Our API stores the key on the payment row with a UNIQUE constraintUNIQUE constraintA rule on a database column: the database refuses to store a second row with the same value there — even if two inserts arrive at the same instant., so the database itself refuses a second payment with the same key:

CREATE TABLE payments (
  id              text PRIMARY KEY,           -- "pay_81", also our key at the PSP (hop 2)
  idempotency_key text NOT NULL UNIQUE,       -- ← a second INSERT with this key fails
  order_id        text NOT NULL,
  amount_cents    bigint NOT NULL CHECK (amount_cents > 0),
  currency        text NOT NULL,              -- "EUR"
  status          text NOT NULL DEFAULT 'created',
  refunded_cents  bigint NOT NULL DEFAULT 0   -- refunds so far (§08)
                  CHECK (refunded_cents BETWEEN 0 AND amount_cents),
  psp_charge_id   text UNIQUE,                -- filled in when the PSP answers
  updated_at      timestamptz NOT NULL DEFAULT now()
);

When a request arrives, the API tries INSERT … ON CONFLICT (idempotency_key) DO NOTHING. If the row already existed, this is a repeat: return the payment's current status instead of starting a new charge. Idempotency §06 shows exactly how to do this in PostgreSQL, including two copies arriving at the same instant.

Hop 2: our API → the PSP

Our API calling the PSP has the same problem one hop further in: our request can time out after the PSP already charged the card. So we pass a key to the PSP too. PSPs such as Stripe and Adyen accept one on payment requests and return the original result for a repeat.

Use a key derived from our payment, such as the payment ID pay_81. It must be the same on every attempt for this payment, including attempts made by the recovery job hours later after a crash. Never generate a new one per attempt; that's the same as having no key.

Hop 3: what if the PSP doesn't answer?

The PSP call times out. Did the card get charged? We don't know. Idempotency §02 explains why no amount of cleverness can tell "never arrived" from "done, but the reply was lost".

So our design makes "we don't know" an honest, stored state: unknown. It's step 4 of the order in §04: after inserting the payment as created and before calling the PSP, we update it to unknown in its own committed transaction. From that moment until the PSP's answer is saved, we truly don't know, and if our server crashes during the call, the row still says so. Then we find out, by whichever of these comes first:

• The PSP's webhook arrives and says what happened (§06).
• The recovery job finds payments that have been unknown for more than a couple of minutes and asks the PSP again, with the same key. If the PSP had charged, it replays that charge; if it never got the request, it does it now. Either way: one charge (§10).

Meanwhile, the buyer sees "Payment processing, we'll email you", not "Payment failed". Telling them it failed would invite them to pay again by another route, which is exactly the double charge we're avoiding.

Try to break it

Pick a failure and step through it. Then switch the protections off to see what each one prevents.

The payment state machine

A state machineFinite state machine (FSM)A model with a fixed list of states (pending, paid, shipped…) and a fixed list of allowed moves between them. The thing is always in exactly one state, and only listed moves are legal. is a fixed list of states plus the moves allowed between them. It's the right shape for a payment because two systems (ours and the PSP's) report on the same payment at different times, and we need a rule for every combination.

The states

The main path, created → unknown → authorized → captured → settled, only goes forward. failed and voided can only happen before money is taken; refunded and disputed only after. What happens after a dispute (won or lost) is its own small process, and out of scope here.

Webhooks arrive late, twice and out of order

The PSP sends a webhook for each event. Three things about webhooks catch everyone out:

• Twice. The PSP retries a webhook until we answer 200. If we processed it but our reply got lost, it comes again.
• Late or early. The webhook can arrive before our own API call has even returned, or minutes after.
• Out of order. "Captured" can arrive before "authorized".

Four rules handle all of it:

1. Dedupe by the PSP's event ID. Every webhook has a unique ID. Save it in a processed_events table in the same transaction that acts on it; if the insert finds it's already there, skip. (Idempotency §08 covers this pattern for any message consumer.)
2. Forward only. An event for a state we've already passed is stale: record it, but don't move back. An event that skips ahead ("captured" while we're still unknown) is fine: it means the steps in between happened.
3. Money moving against our records is an alert, not a log line. If the PSP says "captured" for a payment we marked failed (we told the buyer it failed!), don't just ignore it. Store an alert in the same transaction and have a person decide: capture it properly, or refund it. This can happen if a lookup said "not found" too early (§10).
4. Reject what's impossible but harmless. "Voided" for a captured payment moves no money; it's a bug somewhere. Log it and change nothing.

Refund events are the exception to "event = state". A refund webhook says "€30 of this payment went back", which isn't a state. It's matched to our own refunds row (we created it, and already reserved the amount in refunded_cents, when we asked for the refund) and marks that row succeeded. The payment becomes refunded only when its succeeded refunds add up to the full amount (§08, step 3). A "settled" event can still arrive after a full refund (the refund happened before the day's batch closed); that's normal and just noted.

Feed the machine some events, in any order. Try a few partial refunds too, and "failed" followed by "captured":

In code

Two small tables back the handler, next to payments from §05:

CREATE TABLE payment_events (                -- every transition, forever
  payment_id  text NOT NULL REFERENCES payments (id),
  from_status text NOT NULL,
  to_status   text NOT NULL,
  source      text NOT NULL,                 -- "api", "webhook evt_4", "recovery job"
  created_at  timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE processed_events (              -- PSP webhook IDs we've handled
  event_id    text PRIMARY KEY,
  received_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE payment_alerts (                -- money moved against our records
  payment_id  text NOT NULL REFERENCES payments (id),
  event_id    text NOT NULL,
  detail      text NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);

The decision itself is a small pure function, next. The handler around it does its work in one transaction: dedupe the event, lock the payment row with SELECT … FOR UPDATE (a row lockRow lock (SELECT … FOR UPDATE)Pessimistic locking (“lock first, then read and write”) in a database: reading a row with FOR UPDATE locks it until your transaction ends, so other transactions that want to lock or change it wait., so two webhooks for the same payment can't interleave), then save the new status together with a history row, and an alert if rule 3 fired.

Every writer of status goes through the same next under the same lock: the webhook handler, our API when the PSP's reply comes back, and the recovery job. None of them does a bare UPDATE payments SET status = ….

The code uses a PostgreSQL driver in each language: pgx in Go and pg (node-postgres) in JavaScript. (New to Go? The primer in Async / Await §02 covers the basics.)

// Position on the happy path. Events may move a payment forward, never back.
var rank = map[string]int{"created": 0, "unknown": 1, "authorized": 2, "captured": 3, "settled": 4}

// Moves off the happy path, and the states they are allowed from.
// ("refunded" is set by the refund code once its succeeded refunds add up to amount_cents, §08.)
var sideExits = map[string][]string{
	"failed":   {"created", "unknown", "authorized"},
	"voided":   {"authorized"},
	"refunded": {"captured", "settled"},
	"disputed": {"captured", "settled"},
}

// next decides what an event does to a payment in state cur.
// alert is true when money moved on a payment we had marked "no money moved".
func next(cur, to string) (state, verdict string, alert bool) {
	noMoney := cur == "failed" || cur == "voided"
	moneyMoved := to == "captured" || to == "settled"
	r, toMain := rank[to]
	cr, curMain := rank[cur]
	switch {
	case cur == to:
		return cur, "already there", false
	case noMoney && moneyMoved:
		return cur, "MONEY MOVED on a " + cur + " payment: alert", true // ← a human decides
	case toMain && curMain && r > cr:
		return to, "moved", false // forward, possibly skipping a step we never heard about
	case toMain && curMain:
		return cur, "stale, ignored", false // an old event arriving late
	case slices.Contains(sideExits[to], cur):
		return to, "moved", false
	case cur == "refunded" && to == "settled":
		return cur, "noted: refunded before the batch closed", false
	default:
		return cur, "not allowed from " + cur, false
	}
}

// handleEvent applies one PSP webhook event exactly once, in one transaction.
func handleEvent(ctx context.Context, db *pgx.Conn, eventID, paymentID, to string) (string, error) {
	tx, err := db.Begin(ctx)
	if err != nil {
		return "", err
	}
	defer tx.Rollback(ctx)

	// 1. Dedupe by the PSP's event id: a replayed webhook inserts nothing.
	tag, err := tx.Exec(ctx, `INSERT INTO processed_events (event_id) VALUES ($1)
		ON CONFLICT DO NOTHING`, eventID)
	if err != nil {
		return "", err
	}
	if tag.RowsAffected() == 0 {
		return "duplicate event, skipped", nil
	}
	// 2. Lock the payment row so two events for it can't interleave.
	var cur string
	err = tx.QueryRow(ctx, `SELECT status FROM payments WHERE id = $1 FOR UPDATE`, paymentID).Scan(&cur)
	if err != nil {
		return "", err
	}
	// 3. Decide, then store the new state AND the history row together.
	newState, verdict, alert := next(cur, to)
	if newState != cur {
		if _, err := tx.Exec(ctx, `UPDATE payments SET status = $2, updated_at = now() WHERE id = $1`,
			paymentID, newState); err != nil {
			return "", err
		}
		if _, err := tx.Exec(ctx, `INSERT INTO payment_events (payment_id, from_status, to_status, source)
			VALUES ($1, $2, $3, $4)`, paymentID, cur, newState, "webhook "+eventID); err != nil {
			return "", err
		}
	}
	if alert { // stored in the same transaction, so the event can't be "processed" without it
		if _, err := tx.Exec(ctx, `INSERT INTO payment_alerts (payment_id, event_id, detail)
			VALUES ($1, $2, $3)`, paymentID, eventID, verdict); err != nil {
			return "", err
		}
	}
	return fmt.Sprintf("%s → %s (%s)", cur, newState, verdict), tx.Commit(ctx)
}

We fed both versions the same six webhooks against PostgreSQL 16, in the order they "arrived". The last one is for pay_82, a payment we had already marked failed. Real output (identical for Go and JS):

evt_2 pay_81 captured    created → captured (moved)
evt_1 pay_81 authorized  captured → captured (stale, ignored)
evt_2 pay_81 captured    duplicate event, skipped
evt_3 pay_81 voided      captured → captured (not allowed from captured)
evt_4 pay_81 settled     captured → settled (moved)
evt_9 pay_82 captured    failed → failed (MONEY MOVED on a failed payment: alert)

The history table holds exactly the two real moves of pay_81, created → captured from webhook evt_2 and captured → settled from webhook evt_4, and payment_alerts holds one row for pay_82 for a person to resolve.

The ledger: double entry, append-only

The payments table tells us what happened to each charge. It doesn't tell us whose money it is now. Of the €100 a buyer just paid, how much do we owe the seller? How much is our fee? How much went back in refunds? For that we need a ledgerLedgerAn append-only record of money movements. Entries are never edited or deleted; a mistake is fixed by adding a new correcting entry, so the history always explains the current balance.: an append-only record of money movements.

Accounts and entries

A ledger accountLedger accountA named bucket in the ledger, such as seller:alice or platform:fees. Its balance is never stored by hand: it is the sum of every line ever posted to it. is a named bucket. Ours:

A journal entryJournal entryOne money movement in a ledger, made of two or more lines (account + amount) that add up to zero. It is written all at once and never changed afterwards. moves money between accounts. It has two or more lines, each with an account and an amount: + money into the account, − money out of it. The one rule, which is the heart of double-entryDouble-entry bookkeepingEvery money movement is recorded as two or more lines that cancel out: debits on some accounts and equal credits on others (a card payment can be split between a seller and a fee account). If all entries don't sum to zero, something is wrong — a built-in error check that's centuries old. bookkeeping:

The lines of every entry add up to zero.

One checkout, start to finish

Bob buys a €100.00 lamp from Alice. Our fee is 10%. Amounts are in cents.

Now derive the balances by adding up each account's lines: ext:customers −10000, platform:holding 0, seller:alice 0 (she's been paid), platform:fees +800 (our real profit: €10 fee minus €2 for the PSP), ext:psp_fees +200, ext:seller_banks +9000. Add those six up: 0. Because every entry sums to zero, all balances together always do too. If they ever don't, a bug has written a broken entry, and you know immediately.

A refund

A week later Bob returns the lamp before Alice was paid out (so imagine entry 4 hasn't happened). Refunding him is a new entry that moves the money back, in the same proportions it was split:

We didn't delete entries 1 and 2. We added entry 5, which undoes their effect, and the history shows all three. That's the second rule: never edit or delete an entry. A mistake is fixed by posting a correcting entry, so the history always explains the current balance.

What if Alice had already been paid out? Then entry 5 drives seller:alice to −9000: Alice owes us €90. That's a real situation, and the ledger shows it plainly. The payout job then takes it out of her next payouts. (Whether the PSP gives back its €2 fee on a refund depends on the provider; if not, that's a cost for the platform.)

Play with it

Post entries in any order. Watch the journal grow at the bottom, the balances being recomputed from it, and the total stay at zero. Then try the two buttons marked ✗.

Posting entries safely

Two tables

CREATE TABLE journal_entries (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  external_id text NOT NULL UNIQUE,            -- "pay_81:capture": posting twice is a no-op
  description text NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE ledger_lines (
  entry_id     bigint NOT NULL REFERENCES journal_entries (id),
  account      text   NOT NULL,                -- "seller:alice", "platform:fees", …
  amount_cents bigint NOT NULL CHECK (amount_cents <> 0)  -- + in, − out; never a float
);
CREATE INDEX ON ledger_lines (account);

Two bits of SQL syntax: GENERATED ALWAYS AS IDENTITY makes PostgreSQL number the entries itself (1, 2, 3…), and REFERENCES journal_entries (id) is a foreign key: the database refuses a line whose entry_id doesn't exist.

The important column is external_id. It names why the entry exists: pay_81:capture, pay_81:split, refund_7. Its UNIQUE constraintUNIQUE constraintA rule on a database column: the database refuses to store a second row with the same value there — even if two inserts arrive at the same instant. makes posting idempotent: if the code that posts "pay_81 was captured" runs twice (our API and a webhook both saw the capture, or a retry after a crash), the second attempt inserts nothing.

Posting one entry

The function checks the zero-sum rule, then writes the entry and all its lines in one transaction, so a crash halfway can never leave half an entry behind.

// Line is one line of a journal entry.
type Line struct {
	Account string
	Cents   int64 // + money into the account, − money out of it
}

var ErrUnbalanced = errors.New("journal entry does not sum to zero")

// post writes one journal entry and all its lines in ONE transaction.
// It returns false, and changes nothing, if externalID was posted before.
func post(ctx context.Context, db *pgx.Conn, externalID, desc string, lines []Line) (bool, error) {
	var sum int64
	for _, l := range lines {
		sum += l.Cents
	}
	if len(lines) < 2 || sum != 0 { // ← the double-entry rule, checked before we touch the DB
		return false, ErrUnbalanced
	}

	tx, err := db.Begin(ctx)
	if err != nil {
		return false, err
	}
	defer tx.Rollback(ctx) // does nothing if Commit already ran

	var entryID int64
	err = tx.QueryRow(ctx, `
		INSERT INTO journal_entries (external_id, description) VALUES ($1, $2)
		ON CONFLICT (external_id) DO NOTHING
		RETURNING id`, externalID, desc).Scan(&entryID)
	if errors.Is(err, pgx.ErrNoRows) {
		return false, nil // already posted: this is a retry, nothing to do
	}
	if err != nil {
		return false, err
	}
	for _, l := range lines {
		_, err := tx.Exec(ctx, `INSERT INTO ledger_lines (entry_id, account, amount_cents)
			VALUES ($1, $2, $3)`, entryID, l.Account, l.Cents)
		if err != nil {
			return false, err // the deferred Rollback undoes the entry too
		}
	}
	return true, tx.Commit(ctx) // all lines appear together, or none do
}

Running it

The test program posts the capture and the split of the €100 lamp, then retries the capture as if a webhook arrived after our API had already posted it, then tries an entry with a typo (−5000 against +500), and finally computes every balance:

SELECT account, SUM(amount_cents) FROM ledger_lines GROUP BY account ORDER BY account;

Real output from both versions against PostgreSQL 16 (Go shown; JS prints the same apart from omitting err=<nil>):

post pay_81:capture  posted=true  err=<nil>
post pay_81:split    posted=true  err=<nil>
post pay_81:capture  posted=false err=<nil>
unbalanced: journal entry does not sum to zero
  ext:customers      -10000
  platform:fees        1000
  platform:holding        0
  seller:alice         9000
  TOTAL                   0

The retried capture changed nothing, the typo never reached the database, and the total is zero.

The zero-sum check lives in our code, so a different code path (a one-off script, a future bug) could still skip it. The backstop is the ledger-total alert in §12: one query over all lines that must always return 0. (A database-level check is possible too, with a constraint trigger that runs at COMMIT, at the cost of more moving parts.)

Never UPDATE a balance in place

The tempting design is a balances table and UPDATE balances SET cents = cents + 9000 WHERE account = 'seller:alice'. Don't make that your source of truth:

• It erases the history. After the update, nobody can say why Alice's balance is what it is. "Every cent explainable" is gone.
• A bug is invisible. If one code path forgets the matching update on the other account, money silently appears or disappears. With entries, the zero-sum check catches it.
• It's a race magnet. Two transactions doing read-modify-write on the same row is the classic lost update.

Computing SUM over millions of lines on every page view gets slow, so real systems do cache balances (§11). But the cache is derived from the lines, rebuilt from them when in doubt, and never edited by hand.

Refunds: the same care as charges

A refund moves real money, so it gets everything the charge got: a limit, a key, an unknown state, and a ledger entry only once the PSP confirms. Each refund has its own row, and its ID is also its idempotency key at the PSP:

CREATE TABLE refunds (
  id           text PRIMARY KEY,                  -- "refund_7": also our idempotency key at the PSP
  payment_id   text NOT NULL REFERENCES payments (id),
  amount_cents bigint NOT NULL CHECK (amount_cents > 0),
  status       text NOT NULL DEFAULT 'unknown',   -- unknown → succeeded | failed
  updated_at   timestamptz NOT NULL DEFAULT now()
);

Step 1: reserve and record, in one statement. "Refund €80, then €30 more" on a €100 payment must fail the second time, even if both arrive at the same instant. So the limit check is part of the write itself, a conditional update on payments.refunded_cents, and the refund row is inserted only if it succeeded:

WITH reserved AS (
  UPDATE payments
  SET    refunded_cents = refunded_cents + 3000, updated_at = now()
  WHERE  id = 'pay_81' AND status IN ('captured', 'settled')
    AND  refunded_cents + 3000 <= amount_cents    -- ← never more than was captured
  RETURNING id
)
INSERT INTO refunds (id, payment_id, amount_cents)
SELECT 'refund_7', id, 3000 FROM reserved;        -- INSERT 0 0 = refused

WITH reserved AS (…) names the result of the UPDATE so the INSERT can use it; one statement is always one transaction. Because the UPDATE locks the payment row, a second refund running at the same moment waits, then re-checks the condition against the new total. A partial refund leaves the status at captured, so later refunds still pass the check.

Step 2: call the PSP outside any transaction, with idempotency key refund_7. A timeout leaves the refund unknown and the amount reserved; the recovery job from §10 re-sends stuck refunds with the same key, exactly as it does for payments.

Step 3: record the PSP's answer. Note that refunded_cents already counts the money when we asked, so it includes refunds the PSP hasn't confirmed yet. That's right for the limit check, but it can't decide whether "all of it went back". Only succeeded refunds can. On success, one transaction locks the payment, marks the refund succeeded, posts its ledger entry (external ID refund_7), and moves the payment to refunded only if the confirmed refunds now add up to the whole amount:

BEGIN;
SELECT 1 FROM payments WHERE id = 'pay_81' FOR UPDATE;    -- one refund outcome at a time per payment
UPDATE refunds SET status = 'succeeded', updated_at = now()
WHERE  id = 'refund_7' AND status = 'unknown';             -- 0 rows = already recorded: stop here
-- … post the ledger entry "refund_7" here, in this same transaction (post() above)
UPDATE payments SET status = 'refunded', updated_at = now()
WHERE  id = 'pay_81'
  AND  status <> 'refunded'                                -- move it once
  AND  amount_cents = (SELECT coalesce(SUM(amount_cents), 0) FROM refunds
                       WHERE payment_id = 'pay_81' AND status = 'succeeded');  -- confirmed only
COMMIT;

When that last UPDATE changes a row, write the matching payment_events history row in the same transaction, as every other status change does (§06); in the app this goes through the same next() helper.

On a final "failed" from the PSP (not a timeout), mark the refund failed and give the reservation back, in one statement, and only if the refund was still unknown. A duplicate "failed" notice then matches no row and releases nothing:

WITH failed AS (
  UPDATE refunds SET status = 'failed', updated_at = now()
  WHERE  id = 'refund_9' AND status = 'unknown'        -- a repeat notice matches 0 rows…
  RETURNING payment_id, amount_cents
)
UPDATE payments p SET refunded_cents = p.refunded_cents - failed.amount_cents, updated_at = now()
FROM   failed WHERE p.id = failed.payment_id;          -- …so nothing is released twice

No ledger entry was posted for a failed refund, so there's nothing to reverse.

We ran the awkward case against PostgreSQL 16 on a captured €100 payment: reserve €80 (refund_A) and €20 (refund_C), confirm refund_A, then fail refund_C twice, then refund the last €20 (refund_D) and confirm it. Real output, condensed: the BEGIN/COMMIT lines are dropped, each SELECT result is written on one line, and the ← notes are ours.

-- reserve €80 (refund_A) and €20 (refund_C)
INSERT 0 1
INSERT 0 1
-- the PSP confirms refund_A
UPDATE 1
UPDATE 0                       ← not "refunded": refund_C is still pending
status = captured, refunded_cents = 10000
-- the PSP says refund_C failed, then sends the same notice again
UPDATE 1
UPDATE 0                       ← the duplicate releases nothing
status = captured, refunded_cents = 8000
-- the last €20 can still be refunded (refund_D), and is confirmed
INSERT 0 1
UPDATE 1
UPDATE 1                       ← confirmed refunds now total €100
status = refunded, refunded_cents = 10000

Reconciliation: trust, but check daily

Everything so far relies on our code being right and our messages getting through. Some day a webhook will be lost, a bug will skip a ledger entry, or the PSP will make a mistake of its own. Reconciliation doesn't prevent any of that. It finds it, within a day, while it's still easy to fix.

Three sources, three questions

The job runs in two steps. First, ledger vs PSP report, line by line, matched on the PSP's charge ID (which we stored on the payment) and compared on amount and type. Then, PSP report vs bank: the report's net total for the batch should equal one deposit on the bank statement.

What can differ

Most mismatches are timing, and a good job carries them forward automatically. A category that never resolves itself, such as "missing in our ledger", should page a person.

Run the matcher

Here is one day's data, with a few problems planted (including a bank shortfall in step 2). Run the matcher, then click any row that isn't green to see what it means:

Why daily?

• The inputs are daily. PSPs usually publish one settlement report per payout batch, and banks one statement per day.
• Small windows are easy to fix. A mismatch found the next morning has a handful of suspects. One found at month-end has thousands.
• Money has deadlines. Disputes, refunds and seller payouts all run on clocks measured in days. A missing payment found late may be past the point where you can still do something about it.

Partial failure: charged, but we didn't write it down

A partial failurePartial failureWhen one part of a distributed system fails while the rest keeps working — e.g. the server processed your order but the reply was lost. The caller can't tell "never happened" from "happened, answer lost". is when one part of a distributed system fails while the rest keeps going. In a checkout, two parts can disagree: the PSP (which moved real money) and our database (which records it). Here are the two ways they can disagree, and the design choices that make each one recoverable.

Case 1: the PSP charged, but our database doesn't know

Our service called the PSP, the PSP charged the card, and then our process crashed, or the reply was lost, or the database write of captured failed. The money moved; our records don't say so.

What saves us is the order of operations from §04 (try the timeline there again with this case in mind):

1. Commit the intent first. Insert the payment as created, then update it to unknown, committing each, before calling the PSP. Idempotency §07 teaches this general technique as "recovery points".
2. Call the PSP outside any transaction. Never hold a database transaction open across a network call that can take seconds. It holds locks the whole time, and if the call times out, a ROLLBACK would erase our only record that the charge might have happened.
3. Save the result in a new transaction: status (through next, §06), ledger entry and outgoing messages together.

If the crash lands between steps 2 and 3, the row says unknown. The webhook or the recovery job (below) will move it on. Without step 1, there would be no row at all, and only reconciliation would notice, a day later.

Case 2: our database says captured, but the PSP says it failed

This one should never be caused by our happy path, because we only write captured after the PSP says so. It happens through bugs (code that writes captured before the call), through a capture that the PSP accepts and then fails later (it sends a webhook saying so), or through someone editing data by hand. The defences: the state machine refuses impossible moves (§06), the webhook carries the PSP's later verdict, and reconciliation flags "missing at the PSP" (§09). The fix is a correcting state change and ledger entry, never a silent edit.

Telling other services: the outbox

When a payment is captured, other services need to know: the order service ships the item, an email goes to the buyer. If we commit the payment and then publish an "order paid" message to a message brokerMessage brokerA server that accepts messages from producers and hands them to consumers later — RabbitMQ, Kafka, SQS, NATS. It decouples who sends from who handles, and buffers bursts., a crash between the two loses the message. Publish first and crash before commit, and the order ships for a payment we have no record of.

The fix is the outbox patternOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does.: write the message into an outbox table in the same transaction as the status change and ledger entry. A separate relay process reads the table and publishes. It may publish a message twice after a crash, so receivers dedupe by message ID. Sagas: Reliable messaging builds this in detail, and a whole checkout across several services is a saga.

The recovery job

Every minute, it finds payments (and refunds, §08) that have been unknown for more than 2 minutes (shorter, and the original call may still be running) and asks the PSP again, with the original key. The PSP either replays what it already did, or does it now: one charge either way.

If the job runs on several machines at once, two of them must not grab the same payment. Each run claims its batch like this:

WITH stuck AS (
  SELECT id FROM payments
  WHERE  status = 'unknown' AND updated_at < now() - interval '2 minutes'
  ORDER  BY updated_at
  LIMIT  50
  FOR UPDATE SKIP LOCKED              -- rows another instance has claimed are skipped, not waited for
)
UPDATE payments p SET updated_at = now()   -- bump: no instance picks it again for 2 minutes
FROM   stuck WHERE p.id = stuck.id
RETURNING p.id, p.amount_cents;

Stuck refunds are claimed by the same query run over the refunds table. FOR UPDATE SKIP LOCKED locks the rows it picks and silently skips rows another transaction already holds, and bumping updated_at keeps the next run from picking the same payment straight away. (Against PostgreSQL 16, with one payment stuck for 5 minutes, it returned exactly that payment.) Then, for each claimed payment:

// resolveUnknown re-sends each stuck payment with its ORIGINAL key.
// The PSP either replays what it already did, or does it now: one charge either way.
func resolveUnknown(ctx context.Context, payments []*Payment, psp *FakePSP, now time.Time) {
	for _, p := range payments {
		if p.Status != "unknown" || now.Sub(p.UpdatedAt) < 2*time.Minute {
			continue // only payments stuck for a while (the original call may still be running)
		}
		p.UpdatedAt = now // real code: claimed with FOR UPDATE SKIP LOCKED + updated_at = now()
		cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
		chargeID, err := psp.Charge(cctx, p.ID, p.Cents) // ← same key as the first attempt
		cancel()
		switch {
		case err == nil:
			p.Status = "captured" // real code: next() + ledger entry, in one transaction
			fmt.Printf("%s: captured as %s\n", p.ID, chargeID)
		case errors.Is(err, ErrDeclined):
			p.Status = "failed"
			fmt.Printf("%s: failed (%v)\n", p.ID, err)
		default:
			fmt.Printf("%s: still unknown (%v), will retry next run\n", p.ID, err)
		}
	}
}

The full test programs (not shown) add a Payment type and a fake PSP that behaves like a real one: it remembers keys and returns the first result for a repeat. We gave it four stuck payments: pay_81, which the PSP had charged (our reply was lost); pay_82, which never reached the PSP; pay_83, whose card is declined; and pay_84, stuck for only a few seconds. Real output, identical for Go and JS:

pay_81: captured as ch_1
pay_82: captured as ch_2
pay_83: failed (card declined)
charges at the PSP: 2

pay_81 got its existing charge back instead of a second one, pay_82 was charged now, and pay_84 was left alone because its first call might still be running. Two payments, two charges.

Kill a component

Pick a part of the system and break it. See what buyers notice, what's at risk, and how the design recovers:

What breaks at 10× and 100×

At 100 payments a second at peak, the design from §04 is comfortable. Slide the traffic up and watch where it starts to hurt:

10×: about 1,000 payments a second at peak

Hot accounts. Recall that balances are sums of lines (§08). To show balances quickly, you may cache them in a balances table updated in the same transaction as each entry. But nearly every payment touches platform:holding and platform:fees. If each transaction must lock and update that one row, all payments queue behind each other: a hot accountHot accountA ledger account that nearly every transaction touches, such as the platform's fee account. If you keep its balance in one row that every transaction must lock and update, that row becomes a queue all payments wait in.. Fixes, simplest first:

• Don't cache hot accounts' balances in the write path. Compute them from lines, plus a nightly snapshot ("balance at midnight + lines since").
• Split the account into, say, 16 sub-accounts (platform:fees:0 … platform:fees:15), pick one at random per entry, and add them up when reading. 16 rows to lock instead of one.
• Update cached balances asynchronously, in batches, from the lines. The cache is then a little behind, so never use it to decide whether a payout or refund is allowed.

Reports on the primary. Finance dashboards and exports scan millions of lines. Move them to a read replica. A replica lags behind the primary by up to a few seconds (replication lag), which is fine for a chart and wrong for "can we refund this?". Money decisions always read the primary.

Webhook bursts. After a PSP incident, it may resend thousands of webhooks at once. Have the receiver just check the signature, store the raw event, answer 200, and let workers process the events from a queue at a steady pace.

One PSP is a single point of failureSingle point of failure (SPOF)One component that, if it breaks, takes the whole system down with it — because everything depends on it and nothing can take its place.. Add a second PSP and route payments between them (by country, card type, or health). Each payment stays with the PSP that charged it for its whole life: refunds and disputes must go back to the same one.

100×: about 10,000 payments a second at peak

Now the ledger writes and storage outgrow one primary. Two steps:

• Partition by time. Table partitioningTable partitioningPostgres storing one big table as several sub-tables, e.g. one per day. Queries still see one table, but an old day can be removed by dropping its sub-table: one quick command instead of a slow DELETE. stores ledger_lines as one sub-table per month. Old months are read-only and can be moved to cheaper disks, while staying queryable for audits.
• Shard by seller. Split the data across several databases by seller (see Sharding). The catch: every entry touches a platform account too, and an entry spread over two shards can't be one transaction (cross-shard writes). The usual trick is to give each shard its own copies of the platform accounts (platform:fees@shard3), so every entry stays inside one shard, and add the copies up when reporting.

Failure modes, fraud basics and what to monitor

Failure modes and their defences

Fraud, at a high level

Fraud prevention is its own field, and PSPs offer a lot of it built in. A system designer should know the main shapes:

• Stolen cards. A real cardholder disputes the charge weeks later, and the money is pulled back (a chargebackChargeback (dispute)The cardholder asks their bank to reverse a charge (“I never got it”, “I didn't make this”). The money is pulled back from you while it's investigated; you can send evidence and may win it back.), often with a fee on top. Payouts held until delivery (our holding account) give you time to notice before the money is gone.
• Card testingCard testingFraud where someone with a list of stolen card numbers makes many tiny purchases to find out which cards still work. The tell-tale sign is a burst of small, mostly declined payments.. A burst of small payments, most declined, often from few IP addresses or accounts. Defend with a rate limitRate limitA cap on how many requests are allowed per unit of time — e.g. 100 per second. It says nothing about how many may be running at once. on payment attempts per account, per card and per IP address (Rate Limiting shows how to pick the keys), plus a CAPTCHA after repeated declines.
• Refund abuse and seller collusion. A seller "sells" to themselves with stolen cards and gets paid out before the disputes arrive. Defences: payout delays for new sellers, and limits on how much a new seller can be paid out before their history builds up.
• Card checks done by the PSP: the card's security code, address checks and 3-D Secure (the bank's own confirmation step, such as an app prompt) shift some fraud liability away from you. Turn them on; don't build them.

What to monitor

A handful of numbers tells you whether the money is right. (For how to set targets and alerts, see Observability: SLOs and alerting.)

The ledger total is the cheapest and most powerful check on the list: one query, one number, and it should never be anything but zero.

Check yourself

Predict the answer before you click. Each one is covered above: Q1 in §05, Q2 in §05 and §10, Q3 in §06, Q4 in §07, Q5 in §08, Q6 in §09.

Cheat sheet