Sagas & Distributed Transactions

The 30-second version

A trip, four services

You're building a travel site. Booking a trip means four things must happen:

1. Flights reserves seat 12A.
2. Hotels reserves a room.
3. Cars reserves a rental car.
4. Payments charges the customer's card $840.

In a small app, all four tables live in one database and you'd wrap the four changes in one BEGIN … COMMIT. But at your company each of these is a separate microserviceMicroserviceA small, separately deployed program that owns one part of a bigger system and talks to the others over the network (HTTP, gRPC, a message queue). owned by a different team, and each one has its own database. Some aren't even yours: the payment is a call to a card processor's API.

Now the hotel says "sorry, no rooms" after the flight is already reserved. What happens to the seat? Try it, first with no plan, then with a saga:

With no plan, you end up with a customer who holds a seat on a plane but has no hotel, or worse, has been charged for a trip that doesn't exist. Nothing crashed; every service did its own job correctly. The problem is that nobody owned the whole thing.

Why one transaction can't span them

BEGIN and COMMIT are a conversation between your code and one database server, over one connection. When you write COMMIT, that server makes its own changes permanent. It has never heard of the Hotels database and can't make its changes permanent too.

So the question this whole page answers is: how do you keep several independent systems consistent when no one transaction covers them? Making a program run on several machines that cooperate over a network like this is called a distributed systemDistributed systemA program that runs on several machines that cooperate over a network. Any machine or link can fail or slow down independently, which is what makes them hard., and this is one of its classic problems.

The two answers

Here's the map of what's coming:

Two-phase commit: everyone agrees, or nobody does

First, why the network makes this hard

Inside one program, calling a function has two outcomes: it returns, or it throws. A call to another machine over the network has three:

1. Yes: the reply says it worked.
2. No: the reply says it failed (room not available, card declined).
3. Don't know: no reply came back. Maybe the request was lost. Maybe it arrived, the work was done, and the reply was lost. Maybe the other machine is just slow.

That third case, the unknown outcomeUnknown outcomeThe third possible result of a call over the network: you sent the request and got no answer (timeout, crash, lost reply). The work may or may not have happened, and you can't tell which., is the root of everything on this page. After a timeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever., you genuinely cannot tell "it didn't happen" from "it happened and I didn't hear".

The protocol: a wedding ceremony

Two-phase commitTwo-phase commit (2PC)A protocol for committing one transaction across several databases: a coordinator first asks everyone to “prepare” (promise they can commit), then tells all of them to commit. Correct but slow, and stuck if the coordinator dies mid-way. (2PC) has one coordinatorCoordinator (2PC)The one process in two-phase commit that runs the show: it asks every participant to prepare, collects their votes, decides commit or abort, and tells everyone the decision. and several participantsParticipant (2PC)One of the databases (or services) taking part in a two-phase commit. It does its part of the work, votes yes or no, then waits for the coordinator's decision. (here, the Flights and Hotels databases):

• Phase 1, prepare. The coordinator asks every participant "can you commit?". Each one does its work, saves it durably to disk without committing, keeps its locksLock (mutex)A “do not disturb” sign for a piece of data: only the task holding the lock may touch it; others wait their turn. on the rows it changed, and votes. A "yes" vote is a promise: "whatever happens, even if I crash and restart, I will be able to commit this". This is the prepare phasePrepare (vote) phasePhase 1 of two-phase commit. Each participant does its work, saves it durably to disk without committing, keeps its locks, and answers “yes, I can commit” or “no”. A yes is a promise it can't take back..
• Decision. If every vote is yes, the coordinator decides COMMIT; if any vote is no (or doesn't arrive in time), it decides ABORT. It writes the decision to its own disk first, so it can't forget it.
• Phase 2, commit or abort. The coordinator tells every participant the decision. Each one commits (or rolls back), releases its locks, and acknowledges.

Pick a scenario and step through it. Watch the 🔒 icons: they're rows that nobody else can change.

Why it's correct, and why it can get stuck

It's correct because of the promise. Once a participant voted yes, it can always commit, so the coordinator's decision can always be carried out. And because the decision is on the coordinator's disk, a restarted coordinator tells everyone the same answer it would have given before. Nobody ever ends up committed while someone else aborted.

It gets stuck for the same reason. A participant that voted yes has given up the right to decide alone. If the coordinator dies before sending the decision, the participant is an in-doubtIn-doubt transactionA participant that voted yes in two-phase commit but hasn't heard the decision. It may not commit or abort on its own, so it keeps its locks and waits, possibly for hours. transaction: it can't commit (maybe someone else voted no) and can't abort (maybe the coordinator already told others to commit). So it waits, with its locks held, until the coordinator comes back. A protocol where one dead machine can freeze the others like this is called a blocking protocolBlocking protocolA protocol where one failed machine can force the others to stop and wait until it comes back. Two-phase commit is blocking: if the coordinator dies at the wrong moment, participants are stuck..

§03 runs this for real in PostgreSQL, and explains why most teams don't use 2PC between their own services.

2PC for real: Postgres, XA, and why services avoid it

Postgres as a participant

A prepared transactionPrepared transaction (Postgres)A transaction that ran PREPARE TRANSACTION 'id': its changes are saved on disk and its locks held, but it is neither committed nor rolled back until someone runs COMMIT PREPARED or ROLLBACK PREPARED. It survives restarts. is Postgres's "I do". We ran this on our own PostgreSQL 16 cluster, with two databases standing in for two services: hotel (a rooms table) and flights (a seats table). On a fresh install, it's switched off:

hotel=# BEGIN;
hotel=# UPDATE rooms SET free = free - 1 WHERE id = 7;
hotel=# PREPARE TRANSACTION 'trip-42-hotel';
ERROR:  prepared transactions are disabled
HINT:  Set max_prepared_transactions to a nonzero value.

The default is max_prepared_transactions = 0 on purpose: a forgotten prepared transaction holds locks forever, and most apps never need one. We set it to 10 in postgresql.conf (it needs a server restart) and tried again:

hotel=# BEGIN;
hotel=# UPDATE rooms SET free = free - 1 WHERE id = 7;
hotel=# PREPARE TRANSACTION 'trip-42-hotel';
PREPARE TRANSACTION
hotel=# SELECT gid, database FROM pg_prepared_xacts;
      gid       | database
----------------+----------
 trip-42-hotel  | hotel

The promise survives a crash

The whole point of a "yes" vote is that it survives a crash. We prepared a second transaction in flights, then killed the server without a clean shutdown (pg_ctl stop -m immediate) and started it again. Its log said:

LOG:  recovering prepared transaction 735 from shared memory
LOG:  recovering prepared transaction 738 from shared memory

Both were still listed in pg_prepared_xacts, still holding their locks. Then we played "other customer" and tried to change the room from a fresh session, with a 2-second lock timeoutlock_timeoutA PostgreSQL setting: if a statement has waited this long for a lock, give up with an error (SQLSTATE 55P03) instead of waiting forever. The transaction is then aborted and must be rolled back. so it wouldn't wait forever:

hotel=# SET lock_timeout = '2s';
hotel=# UPDATE rooms SET free = 5 WHERE id = 7;
ERROR:  canceling statement due to lock timeout
CONTEXT:  while updating tuple (0,1) in relation "rooms"

That's the in-doubt problem from §02, live: nobody can touch room 7 until someone runs COMMIT PREPARED 'trip-42-hotel' or ROLLBACK PREPARED 'trip-42-hotel'. Postgres will never decide on its own.

A tiny coordinator in Go

The coordinator is your code. Here's the smallest version that works: phase 1 on both databases, decide, phase 2. We ran it against the same cluster; the rooms table has a rule (a CHECK (free >= 0) constraint) so a full hotel votes no.

// prepare is phase 1 on one participant: do the work, then PREPARE (= vote yes).
func prepare(ctx context.Context, db *sql.DB, gid, work string) error {
    conn, err := db.Conn(ctx) // BEGIN…PREPARE must run on ONE connection
    if err != nil {
        return err
    }
    defer conn.Close()
    for _, q := range []string{"BEGIN", work, "PREPARE TRANSACTION '" + gid + "'"} {
        if _, err := conn.ExecContext(ctx, q); err != nil {
            conn.ExecContext(ctx, "ROLLBACK") // error ignored on purpose: we're voting no anyway
            return err
        }
    }
    return nil
}

// finish is phase 2 on one participant.
func finish(ctx context.Context, db *sql.DB, decision, gid string) error {
    _, err := db.ExecContext(ctx, decision+" '"+gid+"'")
    var pgErr *pgconn.PgError
    if decision == "ROLLBACK PREPARED" && errors.As(err, &pgErr) && pgErr.Code == "42704" {
        return nil // "does not exist": it never prepared, so there's nothing to roll back
    }
    return err
}

func bookTrip(ctx context.Context, flights, hotels *sql.DB, trip string) error {
    // Phase 1: ask every participant to prepare.
    errF := prepare(ctx, flights, trip+"-flight", `UPDATE seats SET booked_by = 'ana' WHERE id = '12A'`)
    errH := prepare(ctx, hotels, trip+"-hotel", `UPDATE rooms SET free = free - 1 WHERE id = 7`)

    decision := "COMMIT PREPARED"
    if errF != nil || errH != nil {
        decision = "ROLLBACK PREPARED"
    }
    // ← a real coordinator writes the decision to its own disk HERE, before phase 2

    // Phase 2: tell EVERY participant, even one whose vote we never received.
    if err := finish(ctx, flights, decision, trip+"-flight"); err != nil {
        return err // must be retried until it succeeds
    }
    return finish(ctx, hotels, decision, trip+"-hotel")
}

Booking twice with only one free room (plus a fmt.Printf of the votes, not shown) printed:

trip-1: COMMIT PREPARED (flight vote: <nil>, hotel vote: <nil>)
trip-2: ROLLBACK PREPARED (flight vote: <nil>, hotel vote: ERROR: new row for relation "rooms"
        violates check constraint "free_not_negative" (SQLSTATE 23514))

The second trip's seat change had already been prepared, and was rolled back with the hotel's. All or nothing, across two databases.

Why ABORT goes to everyone, even the ones that said no

Look at phase 2: it sends the decision to every participant, not only to those that voted yes. That's the unknown outcomeUnknown outcomeThe third possible result of a call over the network: you sent the request and got no answer (timeout, crash, lost reply). The work may or may not have happened, and you can't tell which. from §02 again. When prepare returns an error, the coordinator can't tell "Hotels said no" from "Hotels prepared, and the reply got lost":

We tested exactly that: we made prepare pretend the Hotels reply was lost after PREPARE TRANSACTION had succeeded. The coordinator decided ROLLBACK PREPARED, sent it to both, and afterwards pg_prepared_xacts was empty and room 7 still had its 1 free room. Postgres answers ROLLBACK PREPARED for an unknown name with ERROR: prepared transaction with identifier "…" does not exist, error code (SQLSTATE) 42704; finish treats that code as "nothing to undo".

XA: the standard version

XAXAA 1991 standard (from X/Open) for two-phase commit between a transaction manager and several databases or message queues. Java's JTA and many databases implement it. is the 1991 standard interface for this. A transaction manager (the coordinator, for example in a Java app server using JTA, the Java Transaction API) drives several resource managers (databases, some message brokers) through the same prepare/commit dance. MySQL speaks it with XA START, XA PREPARE, XA COMMIT. It's the same protocol with the same blocking weakness, just standardized.

Where 2PC works well

2PC is alive and well inside distributed databases. CockroachDB, Google Spanner, YugabyteDB and Citus all run 2PC (or an optimized variant of it) between their own shardsShardOne piece of a split-up dataset, together with the database server that stores it. “Tenant 42 is on shard 3” means its rows live on that server. (the slices a big database is split into, one per machine) for transactions that touch more than one shard. It works there because:

• One team controls every participant: same software, same version, same data center, low latency.
• The coordinator's decision is replicated with a consensusConsensusGetting several nodes to agree on one value (or one ordered log of values) even if some of them crash or messages are lost. Raft and Paxos are consensus algorithms. protocol (several machines agree on each decision; see Consensus), so if the coordinator machine dies, another one already knows the decision and finishes the job. The "coordinator dies and everyone waits" problem mostly goes away. (That's how CockroachDB, Spanner and YugabyteDB do it. Citus's coordinator is a single Postgres server that records its decisions in its own tables and recovers from them after a restart or a failover to its standby.)

Why it rarely crosses microservice boundaries

Between your own services, 2PC runs into four walls:

1. Locks across the network. Every participant holds locks from "prepare" until "commit", which is at least two network round trips, plus however long the slowest participant takes. A popular hotel row locked for 50 ms per booking caps that room type at about 20 bookings a second, before anything goes wrong.
2. Availability multiplies. A 2PC transaction needs every participant up at the same moment. If each of four services is up 99.9% of the time, all four are up together only about 0.999⁴ ≈ 99.6% of the time (approximately; it assumes failures are independent). That's roughly 35 hours a year when you can't book trips, instead of about 9. The availabilityAvailabilityThe fraction of time (or requests) a system answers successfully — 99.9% means about 43 minutes of failure a month. of the whole is worse than any part.
3. Coupling. Every service must support the same 2PC protocol and expose "prepare" to callers. Teams can no longer change their database freely; they're bound together at the transaction level.
4. The outside world doesn't play. A card processor's HTTP API, an airline's booking system, an email provider: none of them will hold a "prepared" charge and wait for your decision.

So across services, most teams give up on "everyone commits at the same instant" and use the other answer. That's next.

Sagas: do each step, know how to undo it

The mechanism

A sagaSagaA long business process split into steps, each with its own transaction, plus an “undo” step (a compensation) for each one. If step 3 fails, the compensations for steps 2 and 1 run instead of a rollback. is:

• a list of steps T1, T2, … Tn, where each step is a local transactionLocal transactionAn ordinary BEGIN…COMMIT transaction inside one database. It is ACID, but it only covers that one database. in one service (reserve the flight in the Flights database, and commit);
• for each step, a compensating actionCompensating actionA step that semantically undoes an earlier step that already committed — refund a payment, release a reserved seat — used when you can't roll back across services. C1, C2, … that undoes its effect (C1 = release the seat).

If every step succeeds, the saga is done. If step k fails, the saga runs C(k−1), …, C2, C1, newest first, and ends "compensated". Why newest first? Think socks and shoes: you put socks on first and shoes second, so to undo, the shoes come off first. Later steps may build on earlier ones, and unwinding in reverse means each undo runs while the things it depends on still exist.

Pick which step fails and run it. Turn on "release seat fails once" to see what happens when an undo itself hits an error:

Notice what didn't happen: the step that failed has no compensation run for it. Its own local transaction failed, so it rolled back inside its own database, and there's nothing to undo. Only the steps that committed get compensated. (When the failure was a "don't know", that changes; §05 covers it.)

Undo is not rollback

A database ROLLBACK erases a transaction as if it never happened: nobody else ever saw it. A compensation is different. Step 1 committed. Other people may already have seen the seat as taken. The compensation is a new business action that cancels the effect: a semantic undoSemantic undoUndoing the effect of a committed step with a new, opposite business action (a refund, a cancellation) instead of erasing it. The history keeps both actions..

That last row matters most. A compensation is just another network call, so it can time out or fail like any other. It must be retried until it succeeds (with a pause between attempts that grows each time, called backoffExponential backoffWaiting longer after each failed retry, typically doubling: 1 s, 2 s, 4 s, 8 s… up to a cap. It gives a struggling server room to recover instead of hammering it.), which is what "release seat fails once" showed.

Where the name comes from

The word comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem, "Sagas", which proposed it for long-lived transactions inside one database: instead of one transaction holding locks for hours, split it into short ones with compensations. Microservices adopted the idea because it needs nothing from the databases except ordinary local transactions.

What you give up

Compared with one big transaction, a saga gives up two things:

• Instant all-or-nothing. For a while, some steps are done and others aren't. The system ends up consistent eventually, once the saga finishes or finishes compensating. That's eventual consistencyEventual consistencyA promise that if writes stop, every copy or derived view of the data (replicas, caches, read models, other services) will end up agreeing — but for a while, different readers may see different (stale) versions.. The term usually describes copies of data catching up with each other; here it's applied to the several services of one business process catching up with each other.
• Isolation. Other requests can see the in-between state ("seat taken" for a trip that's about to be cancelled). §07 is all about the bugs this causes and how to prevent them.

In return, no service ever waits for another while holding locks, and any service can be down for a while without freezing the others. The saga simply waits and continues.

Designing steps: compensatable, pivot, retriable

Rule 1: every step and every undo must be safe to repeat

The code driving the saga (we'll write it in §09) does a step, then records "step done". It can crash between those two things. When it restarts, it sees "step not done" and runs the step again. Messages between services also tend to arrive twice (§08). So every step and every compensation must be idempotentIdempotentDoing it twice has the same effect as doing it once. “Set balance to 50” is idempotent; “add 50 to balance” is not — so only the first is safe to retry blindly.: running it twice has the same effect as running it once.

The usual trick is an idempotency keyIdempotency keyA unique ID the client sends with a request (e.g. an Idempotency-Key header) so the server can recognise a repeat and do the work only once.: a unique ID sent with every request, so the receiver can recognise a repeat. In a saga, the saga's own ID (plus the step name) makes a perfect key:

-- Flights service. One row per saga: the saga ID is the key.
CREATE TABLE flight_holds (
  saga_id text PRIMARY KEY,             -- ON CONFLICT (saga_id) below needs this
  seat    text NOT NULL,
  status  text NOT NULL DEFAULT 'held'  -- held | released
);
-- Only one saga may HOLD a given seat (released rows don't count).
CREATE UNIQUE INDEX one_holder_per_seat ON flight_holds (seat) WHERE status = 'held';

-- "reserve seat" for saga trip-42. Running it twice reserves one seat.
INSERT INTO flight_holds (saga_id, seat) VALUES ('trip-42', '12A')
ON CONFLICT (saga_id) DO NOTHING;

ON CONFLICT (saga_id) DO NOTHING means "if a row with this saga_id already exists, skip silently instead of failing" (a form of upsertUpsertInsert a row, or update it if a row with the same key already exists, in one statement. In Postgres: INSERT … ON CONFLICT … DO UPDATE. that does nothing on conflict). It only works because saga_id is the primary key; without a unique constraint on that column, Postgres refuses with ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification. The partial unique index (WHERE status = 'held') stops two different sagas holding seat 12A at once. The Idempotency topic goes deep on keys, storing responses, and retries.

The matching "release seat" is shown under Rule 3, because it has one more job to do.

Rule 2: a compensation must eventually succeed

If "refund the card" fails, you can't compensate the compensation. You retry it, with growing pauses, for as long as it takes. If it keeps failing for hours (the card processor is down, or says "card closed"), the saga parks itself in a "needs attention" state and alerts a person. Design every compensation so that, given time, it can succeed: "issue a refund" can; "un-send an email" can't (§11).

Rule 3: handle "don't know" and "undo before do"

Two awkward cases come from the unknown outcomeUnknown outcomeThe third possible result of a call over the network: you sent the request and got no answer (timeout, crash, lost reply). The work may or may not have happened, and you can't tell which. of §02:

• The step timed out. Did the hotel reserve the room or not? You have two safe choices. Either retry the same step with the same key until you get a definite yes or no (the key makes the retry harmless), or treat it as failed and compensate it too, which means the compensation must cope with "there's nothing to undo" and just succeed. Never just assume "it failed" and skip its undo.
• The undo arrives before the do. The "reserve" message is delayed in the network; the saga times out and sends "release"; then the old "reserve" finally arrives. If Flights naively reserves now, the seat is held forever for a trip that was cancelled. The fix: when a release finds nothing to release, record "trip-42: released" anyway, a marker called a tombstoneTombstone (in a saga)A record that says “this saga's step was cancelled” even though the step never happened, left by a compensation that found nothing to undo. A late-arriving “do” sees it and does nothing., so the late "reserve" sees it and does nothing.

The release written as an upsert does exactly that. If the row exists, it flips it to released; if it doesn't, it creates it, already released:

-- "release seat" for trip-42: safe to run twice, and safe to run BEFORE the reserve.
INSERT INTO flight_holds (saga_id, seat, status) VALUES ('trip-42', '12A', 'released')
ON CONFLICT (saga_id) DO UPDATE SET status = 'released';

Now the late reserve from Rule 1 hits the existing trip-42 row, and its ON CONFLICT (saga_id) DO NOTHING refuses it automatically. We ran the release-then-late-reserve sequence on Postgres 16: the release printed INSERT 0 1, the late reserve INSERT 0 0, and the row stayed released. Step through both versions:

A plain UPDATE flight_holds SET status = 'released' WHERE saga_id = 'trip-42' looks just as idempotent, but when nothing is there it changes 0 rows and leaves no trace, so the late reserve succeeds.

Rule 4: order steps around the pivot

Not all steps are alike. Sorting them into three kinds tells you how to order the saga:

• A compensatable stepCompensatable stepA saga step that has a compensating action, so it can be undone later if a following step fails (reserve a seat → release the seat). can be undone later: reserve a seat (release it), place a hold on the card (void the hold).
• The pivot stepPivot stepThe point of no return in a saga: the step that can't be undone (or that you choose not to undo). Before it, failures are compensated; after it, the saga only moves forward. is the point of no return: the step you can't, or won't, undo. Once it succeeds, the saga only moves forward.
• A retriable stepRetriable stepA saga step after the pivot that is guaranteed to succeed eventually if you keep retrying it (send the confirmation email, update a search index). It never needs a compensation. comes after the pivot and is guaranteed to succeed if you keep retrying: send the confirmation email, add the trip to the customer's "My trips" page.

The ordering rule: compensatable steps first, then the pivot, then retriable steps. Before the pivot, any failure can be undone. After the pivot, nothing can fail permanently, so nothing ever needs undoing.

Who's in charge? Orchestration vs choreography

The same saga, two ways

In orchestrationOrchestrationRunning a saga from one central coordinator (the orchestrator) that tells each service what to do next and decides when to compensate. Like a conductor with an orchestra., one orchestratorOrchestratorThe service or component that drives an orchestrated saga: it stores the saga's state, sends each command, and runs compensations on failure. (for example a "Trip service", or a workflow engine, §10) stores the saga's state and sends commands to each service: "reserve a seat for trip-42". Each service replies, and the orchestrator decides what's next, including when to compensate.

In choreographyChoreographyRunning a saga with no central coordinator: each service listens for events from the others and reacts by doing its step and publishing its own event. Like dancers who each know their cue., there's no central brain. Each service publishes an event ("FlightReserved for trip-42") to a message brokerMessage brokerA server that accepts messages from producers and hands them to consumers later — RabbitMQ, Kafka, SQS, NATS. It decouples who sends from who handles, and buffers bursts., and whoever cares reacts. The Hotels service listens for FlightReserved and reserves a room; Flights listens for HotelFailed and releases the seat.

Switch between the styles and run each scenario. Watch who talks to whom:

Trade-offs

The missing I: others see the in-between

Inside one database, the "I" in ACIDACIDThe four guarantees of a database transaction: Atomic (all or nothing), Consistent (rules hold), Isolated (concurrent transactions don't see each other's half-done work), Durable (once committed, it survives a crash). (isolation) hides your half-done work from everyone else until COMMIT (see Transactions §05 for what goes wrong without it). A saga commits after every step, so there's no such hiding. People sometimes say sagas are "ACD": atomic in the end (thanks to compensations), consistent in the end, durable, but not isolated.

Watch it happen

Trip-42 reserves the last room at a hotel. Then its payment is declined, and the room is released. Meanwhile other requests read the room. Step through it, then switch the hotel to use a "pending" status and run it again:

Three anomalies to watch for

These are the same problems from the Transactions topic, now spread across services:

• Dirty read. Another request reads a change that's later compensated. Customer B is told "sold out" because of a room that trip-42 holds and then releases; a report counts revenue from a charge that's about to be refunded. This is the saga version of a dirty readDirty read (saga version)Reading a change that a saga step has already committed, but that a compensation undoes later. Unlike a dirty read inside one database, the data really was committed, so the database can't prevent it.. Inside one database, a dirty read means seeing a change before it commits, and Postgres never allows it. Here the change was committed; it's undone later by a compensation, so no database setting can prevent it.
• Lost update. Two sagas change the same thing and one overwrites the other. Saga A reads balance = 100, saga B reads 100 and writes 70, then A writes 130: B's change is gone. (Same shape as the lost updateLost updateTwo writers each read the same old value and write back their change; the second write silently erases the first. in Race conditions §05.)
• Changing reads. A saga reads a price in step 1 and relies on it in step 4, but another saga changed it in between. Inside one transaction this is called a non-repeatable readNon-repeatable readInside one transaction you read the same row twice and get two different values, because another transaction committed a change in between..

Countermeasures

None of these come for free; you design them into the data. The four most useful:

1. Semantic lock: make "in progress" visible. Instead of marking the room booked in step 2, mark it pending with the saga's ID. That status is a semantic lockSemantic lockA status field in your own data (“PENDING”, “held by trip-42”) that tells other requests “a saga is still working on this, don't treat it as final”. It is a lock made of business data, not a database lock.: a lock made of business data. Other code treats pending as "not final": the search page can say "1 room, held, check back in a few minutes", and another saga can't take it. The pivot (or a later step) flips it to booked; the compensation flips it back to free.

-- step "reserve hotel" for trip-42: hold, don't book
UPDATE rooms SET status = 'pending', held_by = 'trip-42' WHERE id = 7 AND status = 'free';
-- UPDATE 1

-- trip-43 wants the same room at the same time
UPDATE rooms SET status = 'pending', held_by = 'trip-43' WHERE id = 7 AND status = 'free';
-- UPDATE 0   ← "0 rows changed" = someone else holds it; trip-43's step fails cleanly

-- after trip-42's pivot: confirm (only the holder can)
UPDATE rooms SET status = 'booked' WHERE id = 7 AND held_by = 'trip-42' AND status = 'pending';
-- UPDATE 1

We ran these on Postgres 16; the comments show the results. The WHERE … AND status = 'free' makes the check and the change one atomic statement, so two sagas can't both see free. Add a deadline too (held_until) so a pending hold left behind by a crashed saga is eventually released by a cleanup job.

2. Commutative updates: make the order not matter. Write balance = balance + 30 instead of reading the balance and writing balance = 130. Adding and subtracting give the same result in any order, so two sagas can interleave freely and a compensation (balance − 30) undoes exactly its own change. These are commutative updatesCommutative operationsOperations whose result doesn't depend on order: +50 then +20 gives the same total as +20 then +50. Commutative updates can be applied on different replicas and merged later without conflict..

3. Version check: detect that someone else changed it. Store a version number on the row and only write if it's unchanged since you read it (this is optimistic lockingOptimistic lockingDon't lock while reading. When writing, check that nobody changed the row since you read it (usually with a version number); if someone did, re-read and retry.). Say a refund compensation read balance = 80, version = 1 and wants to subtract 30, so it writes 50:

UPDATE accounts SET balance = 50, version = version + 1 WHERE id = 1 AND version = 1;
-- UPDATE 1 the first time; UPDATE 0 if someone else got there first → re-read and decide again

For "changing reads", the same idea applies at saga level: before relying on a value read earlier, re-read it in the step that uses it (re-check the price at the capture step) and fail the saga if it changed.

4. Announce only after the pivot. If the confirmation email or the "you're booked!" page is a retriable step after the pivot, nobody is ever told "booked" about a trip that's then compensated. The semantic lock plus good step order (§05) prevents most visible dirty reads.

Reliable messaging underneath: outbox and inbox

The problem every step has

Every saga step ends with two actions: commit a change in its own database ("seat held"), and tell someone ("FlightReserved", or a reply to the orchestrator). Those are two different systems, so:

• Commit, then crash before sending: the seat is held, but the saga never hears about it and is stuck forever.
• Send, then the commit fails: the saga moves on believing the seat is held, but it isn't.

Doing both separately like this is sometimes called a dual write, and it can't be made safe by careful ordering. Behavioral patterns §10 has an interactive crash demo of both failures.

The outbox: one transaction, two rows

The outboxOutbox patternWrite an outgoing event into an outbox table in the same database transaction as the change itself; a separate process reads that table and publishes the events. Either both happen or neither does. fixes it with a transaction you already have. In the same local transaction as the business change, insert the message into an outbox table. Either both rows are committed or neither is:

CREATE TABLE outbox (
  id         bigserial PRIMARY KEY,                     -- order of writing
  message_id uuid NOT NULL DEFAULT gen_random_uuid(),  -- the message's ID, fixed NOW
  topic      text NOT NULL,
  payload    jsonb NOT NULL,
  sent_at    timestamptz                               -- NULL = not published yet
);

BEGIN;
INSERT INTO flight_holds (saga_id, seat) VALUES ('trip-42', '12A');
INSERT INTO outbox (topic, payload)
  VALUES ('flight.reserved', '{"saga_id": "trip-42", "seat": "12A"}');
COMMIT;

Note message_id: every message gets its ID when the outbox row is written, not when it's published. That will matter in a moment, when the same message is published twice.

Then a separate background process, the relayOutbox relayThe background process that reads unsent rows from an outbox table, publishes them to the message broker, and marks them sent. It may publish a row twice after a crash., reads unsent outbox rows, publishes them to the message brokerMessage brokerA server that accepts messages from producers and hands them to consumers later — RabbitMQ, Kafka, SQS, NATS. It decouples who sends from who handles, and buffers bursts., and marks them sent. Here's one in Go that we ran against Postgres 16:

// relayOnce publishes up to 100 unsent outbox rows. Safe to run on several machines.
func relayOnce(ctx context.Context, db *sql.DB, publish func(msgID, topic string, payload []byte) error) (int, error) {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return 0, err
    }
    defer tx.Rollback() // does nothing if Commit already succeeded

    rows, err := tx.QueryContext(ctx, `
        SELECT id, message_id, topic, payload FROM outbox
        WHERE sent_at IS NULL
        ORDER BY id
        LIMIT 100
        FOR UPDATE SKIP LOCKED`) // ← other relays skip the rows we're holding
    if err != nil {
        return 0, err
    }
    var batch []msg // msg is a small struct: id, msgID, topic, payload
    for rows.Next() {
        var m msg
        if err := rows.Scan(&m.id, &m.msgID, &m.topic, &m.payload); err != nil {
            return 0, err
        }
        batch = append(batch, m)
    }
    if err := rows.Err(); err != nil {
        return 0, err
    }

    for _, m := range batch {
        if err := publish(m.msgID, m.topic, m.payload); err != nil {
            return 0, err // rollback: rows stay unsent and are retried next round
        }
        // ← a crash right here means m is published again next time, with the SAME msgID
        if _, err := tx.ExecContext(ctx, `UPDATE outbox SET sent_at = now() WHERE id = $1`, m.id); err != nil {
            return 0, err
        }
    }
    return len(batch), tx.Commit()
}

In production you call relayOnce in a loop, every few hundred milliseconds. We ran it against Postgres 16 with one outbox row and a publish that hands the message straight to the receiver shown below; the results are under the next heading.

At least once, so the receiver dedupes

Look at the comment in the relay: if it crashes after publishing but before committing sent_at, it publishes that message again after restarting. Brokers redeliver too, when a consumer crashes before sending its ackAcknowledgement (ack)A “got it, done” message from a receiver back to the sender (or to a message broker). If no ack arrives in time, the sender assumes the message was lost and sends it again. (its "I've handled this, don't send it again"). So delivery is at-least-onceAt-least-once deliveryA delivery promise where a message is never lost but may arrive more than once (for example, re-sent after a crash). Receivers must therefore handle duplicates safely.: never lost, sometimes doubled. (The Queues topic covers delivery guarantees in depth.)

The receiver handles that with an inboxInbox / processed-messages tableA table where a consumer records the ID of every message it has already handled, in the same transaction as the work. A redelivered message finds its ID there and is skipped.: a table of message IDs it has already processed (message_id is its primary key). In the same transaction as the work, insert the ID first, and look at how many rows the insert added:

// handle runs in the Hotels service for every delivered message.
func handle(ctx context.Context, db *sql.DB, msgID, sagaID string) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    res, err := tx.ExecContext(ctx,
        `INSERT INTO inbox (message_id) VALUES ($1) ON CONFLICT DO NOTHING`, msgID)
    if err != nil {
        return err
    }
    if n, _ := res.RowsAffected(); n == 0 {
        return nil // seen this ID before: skip the work, just ack the message
    }
    if _, err := tx.ExecContext(ctx,
        `UPDATE rooms SET status = 'pending', held_by = $1 WHERE id = 7 AND status = 'free'`, sagaID); err != nil {
        return err
    }
    return tx.Commit() // the inbox row and the room change commit together
}

res.RowsAffected() returns how many rows the statement changed (and an error we ignore with _); if n, _ := …; n == 0 declares n and tests it in one line. We made the relay crash right after publish, then restarted it. With print statements added, the two runs showed:

publish 245a9184-3e7e-493f-b37b-749a2f7e8b26 flight.reserved {"seat": "12A", "saga_id": "trip-42"}
processed 245a9184-3e7e-493f-b37b-749a2f7e8b26
💥 crashed after publish, before marking sent

publish 245a9184-3e7e-493f-b37b-749a2f7e8b26 flight.reserved {"seat": "12A", "saga_id": "trip-42"}
duplicate 245a9184-3e7e-493f-b37b-749a2f7e8b26 → skip
sent 1 <nil>

The second publish carried the same ID, because the ID was fixed in the outbox row. If the relay generated a fresh ID each time it published, the duplicate would look new and slip past the inbox. Because the inbox row and the work commit together, a crash can't leave "work done but not recorded" or the other way round. This is deduplicationDeduplicationRecognising that a message or request was already handled (usually by an ID you remember) and skipping it the second time., and together with at-least-once delivery it gives you the effect of exactly-once processing.

Here's the whole path in one picture, including the crash that causes a duplicate:

Build it: an orchestrated saga in Go and JavaScript

The saga's status is a state machine

Each saga moves through a few statuses, and only some moves are legal. That's a finite state machineFinite state machine (FSM)A model with a fixed list of states (pending, paid, shipped…) and a fixed list of allowed moves between them. The thing is always in exactly one state, and only listed moves are legal., the same idea as the order statuses in Behavioral patterns §06:

We store it in one table, next to a step number that says which step to run (or undo) next:

CREATE TABLE sagas (
  id         text PRIMARY KEY,         -- e.g. 'trip-42', also the idempotency key
  status     text NOT NULL,            -- running | compensating | done | compensated
  step       int  NOT NULL,            -- which step to run (or undo) next
  updated_at timestamptz NOT NULL DEFAULT now()
);

The orchestrator

Each step is a name, a Do function and an optional Compensate function. Run loads the saga's row, does exactly one thing based on it, saves the new row, and loops:

// ErrRejected means "the service said no, for good" (hotel full, card declined).
// Any other error means "we don't know" → retry the same step later.
var ErrRejected = errors.New("rejected")

type Step struct {
    Name       string
    Do         func(ctx context.Context, sagaID string) error
    Compensate func(ctx context.Context, sagaID string) error // nil = nothing to undo
}

type Orchestrator struct {
    db    *sql.DB
    steps []Step
}

// Run moves one saga forward (or backward) from whatever state is saved.
func (o *Orchestrator) Run(ctx context.Context, id string) error {
    for {
        var status string
        var step int
        err := o.db.QueryRowContext(ctx,
            `SELECT status, step FROM sagas WHERE id = $1`, id).Scan(&status, &step)
        if err != nil {
            return err
        }

        switch {
        case status == "running" && step == len(o.steps):
            return o.save(ctx, id, "done", step)

        case status == "running":
            s := o.steps[step]
            err := s.Do(ctx, id)
            if errors.Is(err, ErrRejected) { // a definite "no": undo what's done
                log.Printf("%s: %s rejected → compensating", id, s.Name)
                if err := o.save(ctx, id, "compensating", step-1); err != nil {
                    return err
                }
                continue
            }
            if err != nil { // unknown outcome: stop, state unchanged, retry later
                return fmt.Errorf("%s: %w", s.Name, err)
            }
            log.Printf("%s: %s ✓", id, s.Name)
            if err := o.save(ctx, id, "running", step+1); err != nil {
                return err
            }

        case status == "compensating" && step < 0:
            return o.save(ctx, id, "compensated", 0)

        case status == "compensating":
            s := o.steps[step]
            if s.Compensate != nil {
                if err := s.Compensate(ctx, id); err != nil {
                    return fmt.Errorf("undo %s: %w", s.Name, err) // retry later
                }
                log.Printf("%s: undo %s ✓", id, s.Name)
            }
            if err := o.save(ctx, id, "compensating", step-1); err != nil {
                return err
            }

        default: // "done" or "compensated": nothing left to do
            return nil
        }
    }
}

func (o *Orchestrator) save(ctx context.Context, id, status string, step int) error {
    _, err := o.db.ExecContext(ctx,
        `UPDATE sagas SET status = $2, step = $3, updated_at = now() WHERE id = $1`,
        id, status, step)
    return err
}

Read the loop as the state machine: running runs the next step and moves forward; a rejection switches to compensating, starting at the step before the failed one; compensating undoes one step at a time going backwards; reaching the end in either direction writes a final status. An error that's neither success nor rejection (a timeout) returns without changing the row, so the next attempt retries the same step, which is safe because steps are idempotent (§05). "The next attempt" needs something to trigger it; that's the retry loop below.

The steps

Each "service" in our demo is a table in the same database, which is enough to see the mechanics (in real life each Do would be a call to another service, with the saga ID as its idempotency key). The tables are simplified versions of §05's (each has saga_id as its primary key, which the ON CONFLICT clauses need). Every step and compensation is idempotent, and the releases are tombstone upserts from §05:

{
    Name:       "reserve flight",
    Do:         exec(`INSERT INTO flight_holds (saga_id) VALUES ($1) ON CONFLICT DO NOTHING`),
    Compensate: exec(`INSERT INTO flight_holds (saga_id, status) VALUES ($1, 'released')
                      ON CONFLICT (saga_id) DO UPDATE SET status = 'released'`),
},
// reserve hotel: same shape, but returns ErrRejected when HOTEL_FULL=1 is set
{
    Name: "charge card",
    Do:   exec(`INSERT INTO charges (saga_id, cents) VALUES ($1, 84000) ON CONFLICT DO NOTHING`),
    // no Compensate: this is the pivot. Once it succeeds, we only go forward.
    // (Simplified: a real saga would authorize earlier and capture here, as in §05.)
},
{
    Name: "email confirmation", // retriable: never rejected
    Do:   exec(`INSERT INTO emails (saga_id) VALUES ($1) ON CONFLICT DO NOTHING`),
},

(exec(sql) is a small helper that returns a function running that SQL with the saga ID as $1.)

Starting, resuming and retrying

A saga's row is created by Start. It's idempotent too, so a client that retries "book trip-42" doesn't start two sagas:

// Start records a new saga. Calling it twice with the same id is harmless.
func (o *Orchestrator) Start(ctx context.Context, id string) error {
    _, err := o.db.ExecContext(ctx,
        `INSERT INTO sagas (id, status, step) VALUES ($1, 'running', 0)
         ON CONFLICT (id) DO NOTHING`, id)
    return err
}

ResumeAll runs SELECT id FROM sagas WHERE status IN ('running', 'compensating') and calls Run for each one. The program calls it once at startup (so every saga a crash interrupted gets finished) and then in a loop, which is what retries a step that timed out:

for {
    if err := o.ResumeAll(ctx); err != nil { // retries every unfinished saga
        log.Print(err)
    }
    time.Sleep(5 * time.Second)
}

A saga that keeps failing the same step will be retried forever by this loop, which is why you also need the "stuck saga" alert from §11. (A fancier version stores a next_attempt_at per saga and backs off.)

Running it

Against our Postgres 16 cluster, with Go 1.24 and pgx v5. A happy trip, a full hotel, and a crash right after the hotel step committed but before the saga's row was updated:

$ ./saga trip-1
trip-1: reserve flight ✓
trip-1: reserve hotel ✓
trip-1: charge card ✓
trip-1: email confirmation ✓

$ HOTEL_FULL=1 ./saga trip-2
trip-2: reserve flight ✓
trip-2: reserve hotel rejected → compensating
trip-2: undo reserve flight ✓

$ CRASH_AFTER_HOTEL=1 ./saga trip-3
trip-3: reserve flight ✓
💥 crash: hotel reserved, but the saga state was not saved yet

$ ./saga            # restart, no arguments: just resume
trip-3: resuming
trip-3: reserve hotel ✓
trip-3: charge card ✓
trip-3: email confirmation ✓

And a hotel call that times out once (we made the first call return an error that isn't ErrRejected), with the retry loop running every second:

$ HOTEL_TIMEOUT_ONCE=1 ./saga trip-4
trip-4: reserve flight ✓
trip-4: will retry later: reserve hotel: hotel API: timeout after 5s
trip-4: resuming
trip-4: reserve hotel ✓
trip-4: charge card ✓
trip-4: email confirmation ✓

(Log timestamps removed.) After the crash, the sagas row said running, step 1, so the restart ran "reserve hotel" a second time. Thanks to ON CONFLICT DO NOTHING, hotel_holds still had exactly one row for trip-3. The Node version printed the same lines for the first two runs. Step through the crash in slow motion:

1func main() {
2    o.ResumeAll(ctx)        // finish sagas a crash interrupted
3    o.Start(ctx, "trip-3")  // INSERT … ON CONFLICT DO NOTHING
4    o.Run(ctx, "trip-3")
5}
6 
7// inside Run, case status == "running":
8err := s.Do(ctx, id)        // the step's work commits
9// 💥 a crash here: work done, saga row not updated
10o.save(ctx, id, "running", step+1)

Workflow engines: Temporal, Step Functions

What you'd otherwise build yourself

The §09 orchestrator is about 100 lines, and it's missing: retries with backoff, per-step timeouts, "wait 24 hours for the hotel to confirm", "wait for a human to approve", locking so only one instance runs each saga, a UI to see where every saga is, and versioning. A workflow engineWorkflow engineA system that runs long, multi-step processes for you: it stores where each one is, retries failed steps, waits on timers, and resumes after crashes. Temporal, Cadence, AWS Step Functions. provides all of that. The main ones:

• Temporal (and Cadence, the Uber project it was forked from): you write workflows as normal Go, Java, TypeScript or Python code.
• AWS Step Functions: you describe the workflow as a JSON state machine (states, transitions, Retry and Catch rules) and AWS runs it.
• Others: Azure Durable Functions, Camunda (BPMN diagrams), Netflix Conductor, Restate.

Durable execution: how Temporal does it

The core idea is durable executionDurable executionRunning ordinary-looking code whose progress is saved after every step, so if the machine dies, the code resumes on another machine from where it was, with local variables intact.. You write a function; the engine makes it impossible to lose its progress. Temporal splits your code into two kinds:

• The workflow: the plan. Plain code that decides what to do next. It must not talk to the outside world directly.
• ActivitiesActivity (Temporal)In Temporal, a normal function that does real work with the outside world (call an API, write a database). Temporal retries it on failure and records its result.: the actual work (call the hotel API, charge the card). Ordinary functions that Temporal runs, times out and retries for you.

Every time the workflow starts an activity, and every time one finishes, Temporal appends a record to the workflow's event historyEvent history (Temporal)The append-only record Temporal keeps of everything a workflow did: which activities started, what they returned, which timers fired. Replaying it rebuilds the workflow's state after a crash.. If the machine running the workflow dies, another machine re-runs the workflow function from the top, but instead of calling the activities again, it feeds back the recorded results from the history. The code arrives at the same point, with the same local variables, and continues.

Watch a worker die halfway through and another one pick up the same workflow:

1func BookTrip(ctx workflow.Context, t Trip) error {
2    workflow.ExecuteActivity(ctx, ReserveFlight, t).Get(ctx, nil)
3    workflow.ExecuteActivity(ctx, ReserveHotel, t).Get(ctx, nil)
4    workflow.ExecuteActivity(ctx, ChargeCard, t).Get(ctx, nil)
5    return nil
6}

Here is the trip saga as a Temporal workflow in Go. We compiled it against the Temporal Go SDK v1.38 and ran it in Temporal's in-process test environment (no server), with the hotel activity returning a "Rejected" error:

// BookTrip is the workflow: plain Go code that Temporal makes durable.
func BookTrip(ctx workflow.Context, t Trip) (err error) {
    // Before the pivot: retry a few times, then give up and compensate.
    stepCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second, // each attempt may take at most 10 s
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:        time.Second,
            BackoffCoefficient:     2, // wait 1 s, 2 s, 4 s, … between attempts
            MaximumAttempts:        5,
            NonRetryableErrorTypes: []string{"Rejected"}, // a definite "no": don't retry
        },
    })
    // The pivot and the compensations must not give up: retry until a definite answer.
    mustCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:        time.Second,
            BackoffCoefficient:     2,
            MaximumInterval:        5 * time.Minute, // never wait more than 5 min between tries
            MaximumAttempts:        0,               // 0 = unlimited
            NonRetryableErrorTypes: []string{"Rejected"},
        },
    })

    var undo []any // compensating activities, in the order they were registered
    defer func() {
        if err == nil {
            return
        }
        // Run compensations newest-first, even if the workflow was cancelled.
        dctx, _ := workflow.NewDisconnectedContext(mustCtx)
        for i := len(undo) - 1; i >= 0; i-- {
            if cerr := workflow.ExecuteActivity(dctx, undo[i], t).Get(dctx, nil); cerr != nil {
                workflow.GetLogger(ctx).Error("compensation failed", "error", cerr) // + alert a human
            }
        }
    }()

    undo = append(undo, ReleaseFlight) // register first: the call may succeed, then time out
    if err = workflow.ExecuteActivity(stepCtx, ReserveFlight, t).Get(ctx, nil); err != nil {
        return err
    }
    undo = append(undo, ReleaseHotel)
    if err = workflow.ExecuteActivity(stepCtx, ReserveHotel, t).Get(ctx, nil); err != nil {
        return err
    }
    // Pivot: retried until the processor gives a definite yes (or a definite "Rejected").
    return workflow.ExecuteActivity(mustCtx, ChargeCard, t).Get(ctx, nil)
}

// The hotel activity says a definite no when it's full:
func ReserveHotel(ctx context.Context, t Trip) error {
    return temporal.NewApplicationError("no rooms left", "Rejected")
}

With a full hotel, the test environment recorded the activities in this order: ReserveFlight, ReserveHotel (one attempt only, because "Rejected" is non-retryable), ReleaseHotel, ReleaseFlight. In a second test, ChargeCard returned a "Timeout" error three times and then succeeded: the recorded order was ReserveFlight, ReserveHotel, ChargeCard ×4, and the workflow finished without compensating.

Why two retry policies? The steps before the pivot may give up after 5 attempts, because compensating them is safe. The pivot may not: if every attempt to charge timed out, the card might have been charged anyway (the unknown outcome), and giving up would release the flight and hotel of a customer who has paid. So the pivot retries until it gets a definite answer, with the card processor's idempotency key making repeats harmless. Compensations get the same "never give up" policy (Rule 2 in §05); if one keeps failing, the attempt count climbs, and that's what you alert on. Notice that ReleaseHotel ran even though the hotel never reserved anything: we registered each compensation before its activity, because the activity might succeed and then time out (the unknown outcome from §02). That's why "release" must succeed when there's nothing to release (§05).

The same workflow in JavaScript, with Temporal's TypeScript SDK (which also accepts plain JS). Not run; shown to compare the shape:

import { proxyActivities } from "@temporalio/workflow";

const { reserveFlight, releaseFlight, reserveHotel, releaseHotel, chargeCard } =
  proxyActivities({
    startToCloseTimeout: "10 seconds",
    retry: { maximumAttempts: 10, nonRetryableErrorTypes: ["Rejected"] },
  });

export async function bookTrip(trip) {
  const undo = [];
  try {
    undo.push(releaseFlight);
    await reserveFlight(trip);
    undo.push(releaseHotel);
    await reserveHotel(trip);
    await chargeCard(trip); // pivot
  } catch (err) {
    // newest first (real code wraps this in CancellationScope.nonCancellable,
    // so it still runs after the workflow is cancelled)
    for (const fn of undo.reverse()) await fn(trip);
    throw err;
  }
}

The one rule: workflow code must be deterministic

Because the workflow function is re-run from the top during recovery, it must make the same decisions every time it runs with the same history. So inside workflow code:

• no time.Now() or Date.now(): use workflow.Now(ctx) (Go), which returns the recorded time;
• no random numbers, no direct network or database calls, no goroutines started with go: use activities, workflow.Go, and the SDK's versions;
• no time.Sleep: use workflow.Sleep(ctx, 72*time.Hour), a durable timer that survives restarts;
• no for k := range someMap when the order affects what you do: Go randomizes map iteration order on every run, so sort the keys first;
• no Go select on channels: use workflow.NewSelector(ctx), the SDK's deterministic version.

Break the rule and recovery fails with a "non-determinism" error. The TypeScript SDK runs workflows in a sandbox that replaces Date and Math.random with deterministic versions; Go relies on you (and an optional static checker).

What else you get

• Timers. workflow.Sleep for three days costs nothing: the workflow isn't running, just recorded as "wake me at …".
• Signals. An outside event (a person clicking "approve refund") can be delivered into a running workflow, which can wait for it with a timeout. That's how you build a human-in-the-loop stepHuman-in-the-loop stepA step in a long process that waits for a person (approve a refund, check a document), often for hours or days, with a timeout that decides what happens if nobody answers..
• Visibility. A web UI listing every workflow, its status, and its full history, which is the "where is trip-42 stuck?" answer from §06 for free.

Staff-level: designing sagas that survive real life

Compensations are product decisions

"Undo" sounds technical, but every compensation is a business action with a cost and a customer experience. Design them with the product owner, step by step:

Anything that reaches the outside world (emails, SMS, push notifications, webhooks to partners, physical goods) can't be unsent. Put them after the pivot as retriable steps, or design an explicit "correction" message for them.

Timeouts: decide what silence means

A saga waits on other services, sometimes on people. For every wait, write down two things: how long, and what happens then.

• Per step: a timeoutTimeoutA limit on how long to wait for something (a connection, a response) before giving up with an error, so a slow or dead peer can't hang your program forever. on each call (seconds), then retries with backoff. After the last retry fails, either compensate or park the saga for a person.
• Per saga: an overall deadlineDeadlineA point in time ("by 12:00:01.500") after which a call is abandoned. Unlike a per-step timeout, one deadline is shared by every step and every service working on the same request.. "If the trip isn't confirmed within 15 minutes, compensate and email the customer." Without it, a saga can sit half-done for weeks.
• Humans: "Refunds over $1,000 need a manager's approval" is a human-in-the-loop stepHuman-in-the-loop stepA step in a long process that waits for a person (approve a refund, check a document), often for hours or days, with a timeout that decides what happens if nobody answers.. Decide what happens when nobody approves in 48 hours: escalate, auto-approve, or auto-reject.

Pending holds (§07) need their own expiry (held_until) so a crashed saga's leftovers clean themselves up.

Observability: a saga is a long-running thing you must be able to see

A request finishes in milliseconds; a saga can live for days. You need to answer "where is trip-42?" and "how many sagas are stuck?" at any moment.

• The saga table is your dashboard. status, step, updated_at answer most questions. Alert on status IN ('running','compensating') AND updated_at < now() - interval '15 minutes': that's a stuck saga.
• One ID everywhere. Put the saga ID on every command, event, log line and database row as a correlation IDCorrelation IDOne ID (here the saga ID) carried on every message, log line and database row that belongs to the same business process, so you can find all of them later., so one search finds everything about trip-42.
• Traces. Distributed tracing (each step as a spanTracing / spanDistributed tracing records the path of one request through your code and services. Each timed step (one middleware, one DB query) is a span, so you can see where the time went.) shows the timeline across services. Long sagas outlive a single trace, so link the step traces by the saga ID.
• Metrics that matter: sagas started, completed, compensated, and stuck, per saga type; compensation rate by failing step (a sudden jump in "hotel rejected" is a business signal, not just a technical one); and the time each step takes, at the p99p99 (99th percentile)The value that 99% of measurements are below — i.e. how bad the slowest 1% is. Averages hide slow outliers; p99 shows them..

Versioning sagas that are already running

You deploy a new version that inserts a "reserve airport transfer" step in the middle. There are 3,000 sagas in flight, each with a saved step = 2, and step 2 now means something else. Options, from simplest:

• Store step names, not indexes, and make new steps tolerate being skipped for old sagas.
• Version the saga definition: save definition_version in the row and keep the old step list running until every old saga has finished.
• Workflow engines have built-in versioning APIs for exactly this (Temporal's workflow.GetVersion). Use them from day one.

Sometimes the best saga is no saga

Every saga is extra code, extra states, extra failure modes, and a window where users see inconsistent data. Before writing one, ask whether the data that must change together should live together.

• Redraw the boundary. If "reserve stock" and "create order" always change together and must be atomic, maybe they belong in one service with one database and a plain transaction. Services that must constantly coordinate transactions are a sign of a distributed monolithDistributed monolithA system split into many services that still have to change and deploy together. You pay the cost of networks and many deploys without getting independence in return.: all the costs of separate services with none of the independence. See Monolith vs microservices and Domain-driven design for how to draw boundaries around data that changes together.
• Co-locate the data. In a sharded database, choose the shard key so related rows share a shard (Sharding §04), and the "distributed" transaction becomes local.
• Reserve ahead. Give each service a budget it can spend locally: a warehouse is pre-allocated 100 units it can sell without asking anyone, and only talks to the central stock service when it runs low. (This is often called the escrow or reservation pattern.)
• Accept eventual consistency where it's harmless. "Loyalty points appear within a minute" doesn't need a saga with compensations; an event and a retrying consumer will do.

Pitfalls gallery

// ✕ If this process restarts between two awaits, nobody remembers the trip
await reserveFlight(trip);
await reserveHotel(trip);   // 💥 deploy happens here
await chargeCard(trip);

Check yourself

Predict the answer before you click. Each explanation points back to the section that covers it.

Cheat sheet

-- Postgres 2PC, for reference (needs max_prepared_transactions > 0)
BEGIN; UPDATE …; PREPARE TRANSACTION 'trip-42-hotel';
COMMIT PREPARED 'trip-42-hotel';     -- or ROLLBACK PREPARED, from any session in that database
SELECT gid, prepared FROM pg_prepared_xacts;   -- anything old here is holding locks