Saga Pattern for Cross-Shard Business Transactions

A saga trades atomicity for availability: each step commits immediately on its own shard, and failure is handled by undoing what was already done rather than by holding locks. This guide builds one that is safe in production — durable state, idempotent steps, compensations that tolerate a moved world, and reconciliation. It implements the practical alternative to 2PC from Distributed Transactions & Cross-Shard Consistency, part of Cross-Partition Querying & Aggregation Strategies.

Prerequisites

Step 1 — Model the saga as a durable state machine

The state must be a row, not an object in memory:

CREATE TABLE saga_instance (
    saga_id       uuid PRIMARY KEY,
    saga_type     text        NOT NULL,
    state         text        NOT NULL CHECK (state IN
                    ('running','compensating','completed','failed','needs_operator')),
    current_step  int         NOT NULL DEFAULT 0,
    payload       jsonb       NOT NULL,
    attempts      int         NOT NULL DEFAULT 0,
    last_error    text,
    created_at    timestamptz NOT NULL DEFAULT now(),
    updated_at    timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX saga_instance_active_idx
    ON saga_instance (state, updated_at)
    WHERE state IN ('running','compensating');

CREATE TABLE saga_step_log (
    saga_id     uuid    NOT NULL REFERENCES saga_instance,
    step_index  int     NOT NULL,
    direction   text    NOT NULL CHECK (direction IN ('forward','compensate')),
    outcome     text    NOT NULL,
    recorded_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (saga_id, step_index, direction)
);

The step log’s primary key is what makes resumption safe: a step already recorded as forward/succeeded is skipped rather than repeated when a worker picks the saga up again.

Operational note: The partial index on active states keeps the resume query cheap as the table accumulates millions of completed sagas. Completed rows are still worth keeping for audit; they simply should not be scanned.

DBA tip: Store payload as jsonb and never mutate it. A saga that rewrites its own inputs mid-flight cannot be replayed for debugging, and replay is the main tool available when one goes wrong.

Step 2 — Write forward steps and compensations in pairs

Every step is defined with its undo alongside it, so the pair cannot drift:

Compensations run in reverse, and only for completed steps Steps one through three completed and step four failed. Compensation runs for step three, then two, then one — the reverse of the order in which they succeeded. Step four is not compensated because it never took effect, and running its compensation would undo something that never happened. 1 · reserve stocksucceeded 2 · debit walletsucceeded 3 · create shipmentsucceeded 4 · notify carrierfailed permanently cancel shipment refund wallet release stock Reverse order matters when steps have dependencies: releasing stock before cancelling the shipment would leave a shipment referencing inventory that is already promised to someone else. 
# saga_transfer.py
STEPS = [
    Step(
        name="debit_source",
        forward=lambda ctx: shard_for(ctx["src"]).execute(
            """INSERT INTO ledger_entries (id, account_id, amount, kind, saga_id)
               VALUES (%(entry_id)s, %(src)s, -%(amount)s, 'transfer_out', %(saga_id)s)
               ON CONFLICT (id) DO NOTHING""", ctx),
        compensate=lambda ctx: shard_for(ctx["src"]).execute(
            """INSERT INTO ledger_entries (id, account_id, amount, kind, saga_id)
               VALUES (%(comp_id)s, %(src)s, +%(amount)s, 'transfer_reversal', %(saga_id)s)
               ON CONFLICT (id) DO NOTHING""", ctx),
    ),
    Step(
        name="credit_destination",
        forward=lambda ctx: shard_for(ctx["dst"]).execute(
            """INSERT INTO ledger_entries (id, account_id, amount, kind, saga_id)
               VALUES (%(credit_id)s, %(dst)s, +%(amount)s, 'transfer_in', %(saga_id)s)
               ON CONFLICT (id) DO NOTHING""", ctx),
        compensate=None,   # last step: nothing after it can fail
    ),
]

Two properties do the heavy lifting. Every write is an INSERT ... ON CONFLICT DO NOTHING on a deterministic id derived from the saga, so re-running a step is a no-op. And the compensation is a new ledger entry rather than an update of a balance, so it is correct regardless of what happened to the account in between.

Saga states and every transition between them A saga starts running and advances through its forward steps to completed. A failed forward step moves it to compensating, which walks the completed steps backwards to failed. A compensation that exhausts its retries moves the saga to needs_operator, which is a terminal state requiring a human. Every transition is a durable row update, so a process restart resumes exactly where it left off. createdpayload durable runningforward steps, idempotent completedterminal, happy path compensatingwalking steps backwards needs_operatorcompensation exhausted retries failedterminal, fully compensated The state that matters operationally is needs_operator. It is rare, it always means money or inventory is in an inconsistent place, and it must page a human rather than sit in a dashboard nobody reads.

Operational note: A step with no compensation must be last, or the saga has an unreachable rollback. If a non-final step cannot be undone — sending an email, for instance — restructure so it happens after everything reversible has succeeded.

SRE tip: Give each compensation its own deterministic id (comp_id above). Reusing the forward step’s id means the ON CONFLICT guard silently swallows the compensation.

Step 3 — Drive it from a resumable worker

The worker is the only thing that advances state, and it must be safe to run in multiple copies:

# worker.py — claim, advance, release; safe to run N instances
def tick(db):
    saga = db.query_one("""
        UPDATE saga_instance
           SET state = state, updated_at = now(), attempts = attempts + 1
         WHERE saga_id = (
               SELECT saga_id FROM saga_instance
                WHERE state IN ('running','compensating')
                  AND updated_at < now() - interval '5 seconds'
                ORDER BY updated_at
                FOR UPDATE SKIP LOCKED
                LIMIT 1)
     RETURNING *""")
    if not saga:
        return

    steps = STEP_REGISTRY[saga.saga_type]
    try:
        if saga.state == "running":
            step = steps[saga.current_step]
            step.forward(saga.payload | {"saga_id": saga.saga_id})
            record(db, saga, saga.current_step, "forward", "succeeded")
            advance_or_complete(db, saga, steps)
        else:
            step = steps[saga.current_step]
            if step.compensate:
                step.compensate(saga.payload | {"saga_id": saga.saga_id})
            record(db, saga, saga.current_step, "compensate", "succeeded")
            retreat_or_fail(db, saga)
    except TransientError as exc:
        db.execute("UPDATE saga_instance SET last_error=%s WHERE saga_id=%s",
                   (str(exc), saga.saga_id))            # retried on the next tick
    except PermanentError as exc:
        begin_compensation(db, saga, str(exc))

Operational note: FOR UPDATE SKIP LOCKED is what allows several workers without coordination — each claims a different saga and none block. The updated_at filter provides the retry backoff for free.

DBA tip: Distinguish transient from permanent errors explicitly. Treating every error as transient means a business rejection retries forever; treating every error as permanent means a network blip triggers an unnecessary compensation.

Step 4 — Alert on stuck sagas and reconcile the invariant

Two safety nets, both cheap:

-- sagas that have not progressed
SELECT state, count(*), max(now() - updated_at) AS oldest
FROM   saga_instance
WHERE  state IN ('running','compensating','needs_operator')
GROUP  BY state;

-- the invariant this saga exists to preserve
SELECT sum(amount) AS should_be_zero
FROM   ledger_entries
WHERE  kind IN ('transfer_out','transfer_in','transfer_reversal')
  AND  created_at >= current_date - 1;
Two checks that between them catch every saga failure The stuck-saga query finds operations that started and never reached a terminal state, which is a liveness problem. The ledger reconciliation sums transfer entries and expects zero, which finds correctness problems including duplicated effects and missing compensations. A saga system needs both, because each is blind to what the other catches. Liveness check — stuck sagas Correctness check — ledger sums to zero catches: worker stopped, poison payload, a step that neither succeeds nor fails misses: a saga that completed successfully while doing the wrong thing twice runs: every minute catches: duplicated credits, missing compensations, manual edits by an operator misses: an operation that never started, because nothing was written to compare runs: daily, over the previous day The pairing is the point. Liveness proves every operation reached a terminal state; correctness proves the terminal states add up. A system with only the first can be consistently wrong; one with only the second cannot tell you why. Publish both as metrics. A reconciliation that runs and reports zero is the daily evidence that the design still works.

Operational note: The reconciliation must run against the shards, not against a cache or a warehouse copy. A warehouse that is itself fed by the same buggy path will agree with the bug.

SRE tip: Record the reconciliation result even when it is zero. A gap in the series is itself a signal — it means the check stopped running, which is how a broken invariant goes unnoticed for a quarter.

Verification

Force a failure and confirm the compensation runs:

# staging only: make the second step fail permanently
with fault_injection(step="credit_destination", error=PermanentError("account frozen")):
    saga_id = start_transfer(src=8842, dst=1190, amount=2500)

wait_until(lambda: saga_state(saga_id) == "failed", timeout=60)
SELECT step_index, direction, outcome, recorded_at
FROM   saga_step_log WHERE saga_id = '…' ORDER BY recorded_at;
 step_index | direction  |  outcome  |          recorded_at
------------+------------+-----------+-------------------------------
          0 | forward    | succeeded | 2026-08-03 10:02:11.104+00
          1 | forward    | failed    | 2026-08-03 10:02:11.482+00
          0 | compensate | succeeded | 2026-08-03 10:02:16.903+00

Then confirm the ledger for that account nets to zero, and that re-running the worker does not add a second reversal.

Failure mode table

Failure mode Root cause SRE mitigation
Saga stuck in running forever the worker died mid-step, or an error type was neither retried nor escalated claim rows with FOR UPDATE SKIP LOCKED and an updated_at timeout so another worker resumes; alert on the oldest active saga
Compensation applied twice the compensating write reused the forward step’s idempotency id, or had none give compensations their own deterministic id and a unique constraint; log each compensation in saga_step_log with a primary key that prevents repeats
Compensation is impossible a non-final step had an irreversible external effect, such as a sent email reorder so irreversible steps run last; where that is impossible, escalate to needs_operator and page rather than silently completing

FAQ

Where should saga state be stored?

In a durable store that survives the failure of any participating shard — a dedicated database, or the shard of whichever entity owns the operation, provided that shard is a participant in every step. Storing it in application memory or a cache means a process restart loses the knowledge that a compensation is owed, which is the one thing the pattern exists to guarantee.

What makes a good compensating action?

One that is idempotent, that works on state which has since changed, and that is semantically rather than physically an undo. Reversing a debit by writing a compensating credit entry works even if the account balance has moved; restoring a saved balance value does not, because it silently discards every write that happened in between.

How long should a saga retry before giving up?

Forward steps should retry with backoff for as long as the failure looks transient — minutes, not hours — and then compensate. Compensations should retry effectively forever, because abandoning a compensation leaves the system in the inconsistent state the saga was designed to avoid. When a compensation exhausts its automatic attempts, it belongs in a human queue, not in a log line.

Back to Distributed Transactions Cross Shard Consistency