Consistency, Idempotency and Reconciliation

Executive summary

A money system can be slow and survive; it cannot be wrong without knowing it and survive. Correctness rests on invariants that must always hold: balances derive from postings, value is conserved as equal debits and credits, and every instruction has exactly one outcome. Distribution threatens all three, because a retry can pay twice, a stale read can over-authorize, and a timeout can leave two services disagreeing about what happened.

This paper treats alignment as a chain of techniques, each closing a gap that the previous one leaves open. A small strongly consistent core protects the invariants; idempotency makes every effect happen once; the outbox and compensation move work across boundaries that no single transaction can span; and reconciliation proves, on a schedule, that records agree. Because the cost of a duplicate exceeds the cost of an omission, the system does nothing when an outcome is unknown and resolves the doubt first.

• Protect invariants that span rows with serializable isolation or a single writer per account, and reserve eventual consistency for projections.
• Make every money-moving request idempotent with a scoped key stored beside its outcome, and carry an end-to-end identifier across external boundaries.
• Move work across services with local transactions, the transactional outbox, idempotent receivers and compensation, never two-phase commit across banks.
• Record both effective and posting dates, order changes by version rather than arrival, and treat an unknown outcome as a query, not a retry.
• Engineer reconciliation as a pipeline that extracts, normalizes, matches, classifies, resolves and evidences, and back it with integrity monitors.

The paper closes with failure-injecting tests, an operating model that pairs finance and engineering owners, the metrics that show the discipline is healthy, and a staged path to operating maturity.

Why consistency is different for money

In most software a retry is harmless and a stale read is cosmetic; in a money system a retry can pay an employee twice, and a stale read can authorize a payment that the balance no longer supports.

Correctness for money rests on invariants that must always hold. Balances are derived from postings rather than stored and edited, so the history of movements is the truth and the balance a derivation. Value is conserved, because every debit has an equal credit. And every instruction has exactly one outcome: paid once, neither lost nor duplicated.

Distribution threatens each invariant. A service may crash after committing a change but before announcing it, or a call may time out with its result unknown. The two failure directions do not cost the same. An omission is usually visible quickly and the funds are still held, while a duplicate leaves the system and must be clawed back from a counterparty who may be unwilling or unable to return it. The architecture is therefore biased toward doing nothing when an outcome is unknown, until the doubt is resolved.

The correctness contract

This paper commits to a contract with three clauses, and every mechanism it describes serves one of them. Every effect happens once, however many times a request is delivered or retried. Every record that must agree is brought into agreement on a known schedule. And every mismatch is detected and resolved with evidence, by an authorized transaction that leaves a trail.

The architecture and the running examples

The mechanics assume a common architecture: a double-entry, append-only ledger at the center as the system of record, with idempotency and reconciliation treated as design principles rather than afterthoughts. This paper develops how those principles work in practice. Two examples recur. Payroll disbursement pays many parties from an approved run and must never pay one of them twice. Debt payment allocation applies many small inbound payments to the right accounts, each exactly once.

Consistency models and transactions

Inside one database, transactions provide atomicity, consistency, isolation and durability (ACID). Isolation is the property most often misjudged, because engines offer it in levels that trade correctness for concurrency. Read committed, the default in several widely used databases including PostgreSQL, Oracle and SQL Server (MySQL's InnoDB engine defaults to repeatable read), prevents reading uncommitted data but still allows transactions to interfere in ways that matter for money.

A lost update occurs when two transactions read a value, each computes a new one and each writes it back, so that one overwrites the other; two debt allocations posted concurrently can both read the same principal. Snapshot isolation prevents many anomalies but permits write skew, where two transactions make disjoint writes that together break an invariant, as when two withdrawals each pass a shared balance check and both post. Where an invariant spans rows, snapshot isolation is not enough.

Level names are not portable, which makes the choice easy to get wrong. Oracle's SERIALIZABLE level and PostgreSQL's REPEATABLE READ level both provide snapshot isolation, so both permit write skew; PostgreSQL's SERIALIZABLE level adds the checks that prevent it. Teams should confirm what their engine's levels actually guarantee rather than trusting the name.

Protecting the invariants

Two remedies fit money. Serializable isolation makes transactions behave as if they ran one after another, at the cost of blocking or of aborted transactions that the caller must retry. Alternatively, all writes for one account are routed through a single writer, so that its changes are serialized by construction. Balance-affecting writes deserve one or the other. Concurrency control is then either optimistic, detecting conflicts at commit through a version check, or pessimistic, locking a hot row before reading it.

Why distributed transactions are avoided

When a change must span services or reach a bank, the classical answer is two-phase commit (2PC), in which a coordinator asks every participant to prepare and then to commit. 2PC ties each participant's availability to the coordinator, holds locks across the network, and leaves prepared participants blocked if the coordinator fails at the wrong moment. External institutions, moreover, enroll in no one's transaction manager. Money systems therefore avoid it across services and across parties.

The resolution is a small strongly consistent core with eventual consistency around it. The ledger is the core, serialized per account; projections such as balance views, statements and analytics are rebuilt from it and may lag by seconds. Pat Helland's principle, that systems larger than one transaction are built from independent entities exchanging idempotent messages, organizes the design around that core.

CAP, PACELC and where money sits

Two well-known results frame the choice. The CAP theorem, conjectured by Eric Brewer and proved by Seth Gilbert and Nancy Lynch, shows that when a network partition separates replicas, a system cannot remain both consistent, in the strict linearizable sense, and available: it must refuse some requests or risk giving divergent answers. Brewer himself later described the popular summary, pick any two of three properties, as misleading. Partitions are faults that a distributed system cannot opt out of, so the real choice is what to give up when one occurs. Daniel Abadi's PACELC extends the point to normal operation: if there is a partition (P), the trade-off is between availability (A) and consistency (C); else (E), it is between latency (L) and consistency (C).

For money the answer differs by component. The ledger chooses consistency in both cases, refusing or delaying a posting rather than risking two balances, and accepts the latency of a strongly consistent write. Projections and channels choose availability and low latency, and show how fresh their data is. The figure below positions the main approaches by the scope of the invariant and the latency the operation can tolerate.

Idempotency and effectively-once processing

An operation is idempotent when performing it many times has the same effect as performing it once. Reading a balance is naturally idempotent; appending a payment is not, unless the system is told how to recognize a repeat. Idempotency is the engineering of that recognition, and it realizes the contract's first clause, that every effect happens once.

Keys and fingerprints

A money-moving request carries an idempotency key, supplied by the caller and unique to the intended effect. The server stores the key with the outcome of the first execution, in the same transaction as the effect, and a later request with the same key returns the stored outcome instead of acting again. The key is scoped to a caller and an operation so that unrelated requests cannot collide, and it is retained long enough to outlast any retry, then expired under a published policy.

A stored key also carries a fingerprint of the original request, such as a hash of amount, payee and account. A key that returns with a different payload is rejected rather than served, because a reused key with changed content signals a client error or an attack, not a retry. A request that arrives while the first attempt with the same key is still in progress is told to retry later rather than executed in parallel.

These conventions are being standardized for HTTP APIs. An IETF Internet-Draft, still a work in progress, defines an Idempotency-Key request header and recommends that a server answer a reused key with a different payload with status 422, a concurrent duplicate with 409, and a missing key on an operation that requires one with 400. Public payment APIs follow the same pattern: Stripe, for example, saves the status code and body of the first request made with a key, compares the parameters of any retry with the original and returns an error if they differ, and may remove keys once they are at least 24 hours old.

Inboxes, boundaries and batches

Delivery is the same problem in another guise. Most message brokers guarantee at-least-once delivery, so a consumer will eventually see a duplicate. An idempotent receiver, or inbox, records the identifiers of messages it has processed in the same transaction as its work and ignores a repeat. At-least-once delivery plus deduplication yields effectively-once processing: the effect happens once even though delivery may repeat. This is the honest name for what is loosely called exactly-once. Exactly-once delivery cannot be guaranteed over an unreliable network, and broker features described as exactly-once, such as Kafka's transactions, apply within the broker's own boundary; they do not extend to a database write in another system or an instruction sent to a bank.

Beyond the system's edge, idempotency depends on identifiers that both sides carry. ISO 20022 gives each level its own: a message identification for the message or file, which many banks check for duplicates; a payment information identification for each batch; and an end-to-end identification that the initiating party assigns to each transaction and that is passed on unchanged through the payment chain. Reusing the same identifiers on a resend lets both the bank's duplicate checks and the originator's reconciliation recognize the repeat; identifiers minted fresh on each attempt defeat both. Scopes therefore run from a single request over the Hypertext Transfer Protocol (HTTP) to a payment instruction, each with its own key, store and retention.

Retries must be paired with keys. A retry with exponential backoff and jitter is correct only when every attempt carries the same key; the resend trap is to answer a timeout with a freshly generated file, which pays again if the first one arrived. Batch runs earn idempotency by keying results to the run and partition, so that a failed partition reruns without duplicating rows and a crashed allocation resumes without reapplying payments.

State machines, the outbox and compensation

Every money-moving thing, whether a payment, a run or an allocation, is modeled as an explicit state machine with named states and allowed transitions. Illegal transitions are rejected rather than patched, and each transition is a compare-and-set: the system moves a payment from submitted to settled only if it is still submitted, so a duplicate event or a concurrent worker cannot advance it twice. Explicit terminal states (settled, rejected, returned and canceled) say when a thing is done and may not change again.

Local transactions and the outbox

A step usually must change state and tell others that it did so. Doing these as two separate writes is the dual-write problem: a crash between them leaves the state changed and the announcement lost, or the reverse. The transactional outbox removes the gap by writing the outgoing message into an outbox table in the same local transaction as the state change. A relay, either polling the table or tailing the database's transaction log, then publishes the message and marks it sent. Publication becomes at-least-once, and idempotent receivers absorb the repeats.

Compensation, not rollback

Across steps there is no rollback, because committed local transactions cannot be undone by a coordinator and external effects cannot be recalled. The saga, introduced by Hector Garcia-Molina and Kenneth Salem as a way to break a long-lived database transaction into a sequence of shorter ones, provides the alternative: it runs a sequence of local transactions and, when a later step fails, runs compensating transactions that semantically undo the steps already completed. Microservice architectures adopted the idea for work that spans services. In a ledger, compensation is a reversing entry that posts the opposite movement with its own reason and approval, so the journal shows the action, the reversal and the people involved. A compensation obeys accounting rules; it is not a silent delete.

Pivot steps and unknown outcomes

Some actions cannot be compensated cleanly: once a payment file reaches the bank, the money may move. In the classification popularized by Chris Richardson, a saga consists of compensatable steps, a pivot step and retriable steps. Reversible steps are placed before the pivot, so the saga can still back out; the pivot is attempted only when everything before it has succeeded; and the steps after it are designed to be retried until they complete, because the saga is now committed to finishing. Ordering the work around the pivot is a design decision, not an afterthought.

Ordering, time and dates

Money has two clocks. The effective date is when an event belongs economically; the posting date is when the system recorded it. They diverge whenever information arrives late, such as a payment that settled on the last day of a month but is reported on the first. Keeping both dates, rather than collapsing them into one, makes correct period reporting and retroactive adjustment possible.

Late and out-of-order events

Events do not arrive in the order in which they occurred. A bank may report a settlement days after the fact, a reversal may appear before the original, and two updates to one account may arrive swapped. A system that trusts arrival order will compute wrong balances. The defense is to order by data, not by delivery: each account carries a sequence number or version that increases with every change, so a stale or duplicate update is detected because its version is not the one expected.

A late event is applied with its effective date, not the date it arrived, so the economic history stays correct even when the wall clock has moved on. An allocation that lands after a statement was produced posts to the period it belongs to and triggers a corrected view, rather than being forced into the present period.

Trusted time and cut-offs

Timestamps that carry financial weight come from a trusted source, the database or a time service, not from client clocks that drift, differ by device and can be set by anyone. Cut-offs decide which business day an instruction belongs to and are defined in a named time zone; a day boundary that ignores zones will misclassify work near midnight. Every stored time carries its zone or is held in Coordinated Universal Time (UTC) and rendered locally.

Period close is where late arrivals meet the calendar. A closed period is not reopened; an item found after close posts into the open period with its original effective date, as an adjustment that explains itself. This keeps closed books stable while still reflecting the truth, and it is the same mechanism that payroll uses for retroactive pay and debt management for backdated interest. The ledger's immutability and dual dating make it safe.

Reconciliation as a discipline

Reconciliation is the discipline of comparing two records that should agree and resolving every difference. Preventive controls reduce errors; reconciliation proves, continuously, that the errors which slipped through are found. It is not a spreadsheet exercise performed under deadline but an engineered pipeline that runs on a schedule and produces evidence.

The reconciliations a money system runs fall into the recognizable types shown in the table above. They confirm that recorded movements happened, that summaries agree with detail, that what was approved is what was paid, that two populations agree (such as payroll against human resources headcount), and that an instruction, its acknowledgment and its statement line agree three ways.

Matching rules and timing

Matching rules range from exact to tolerant. An exact match on an identifier is the strongest, and it is the goal wherever an end-to-end identifier exists. A tolerance match accepts small, explained differences such as a rounding remainder. A one-to-many match links one settlement to several instructions, or several receipts to one invoice, which is common in debt payment allocation.

Not every difference is an error. A timing difference is a real item that the two sides record at different moments, such as a payment sent today and settled tomorrow, and it clears when the lagging side catches up. A true break will not resolve on its own. Confusing the two wastes effort, so the engine ages items and separates those still within their settlement window from those that have overrun it.

Breaks and the engine

A break has a lifecycle: it is classified by type, aged from the day it arose, assigned to an owner who can resolve it, and closed only by an authorized correcting transaction in the domain services, never by editing the record to make the difference disappear. A reconciliation engine has a fixed shape, so that adding a reconciliation is configuration over it rather than fresh code.

Integrity controls and detection

Detection is the second half of the contract: if an effect somehow happens twice, or a record drifts, the system should find it quickly and by itself. The cheapest detective control is the total. A control total counts records and sums amounts in a file or batch, and the receiver checks the count and sum before processing; a hash total does the same for identifiers that have no meaningful sum. ACH files are a familiar example: each batch and file control record carries an entry count, total debits and credits, and an entry hash computed from the receiving banks' routing numbers. A batch whose totals do not match is quarantined, not processed.

Two assertions guard the ledger. The balanced-entry assertion refuses any journal entry whose debits and credits do not sum to zero, so an unbalanced movement cannot be posted at all. Checkpoint rebuilds recompute balances from the journal on a schedule and compare them with the stored balances; a mismatch means that something wrote a balance outside the posting path, and it is caught within a day rather than at the annual close.

Invariant monitors and sweeps

Invariant monitors assert properties of the whole, as queries that run continuously. The sum of all account balances equals the sum of all postings. No account that may not go negative holds a negative balance. Suspense and clearing accounts return to zero within their expected window. A violation is an incident, because these monitors encode, as running code, the invariants that the design was required to state.

Two sweeps catch the classic failures. A duplicate sweep looks for two effects that share an identifier, or a payee and an amount within a window, catching a double payment that slipped past the keys. A sequence-gap sweep looks for missing numbers in a per-account or per-file sequence, catching a lost event before it becomes a lost payment. Both feed the same exception handling as reconciliation.

Drift and alerting

Projections and replicas drift. Drift detection compares a projection with its source of truth and repairs or rebuilds it when they differ beyond a bound. These checks are business signals, not only technical metrics: a rising count of duplicates caught, or a suspense account that will not clear, points to a weakening control, and alerts are routed to the people who own the outcome. Data quality rules and lineage on the feeds into the ledger make such signals easier to trace to their cause.

Testing and verification

Correctness that is not tested is a hope. The invariants, the idempotency and the reconciliation described in this paper each have a technique that targets the way they fail. Property-based tests assert an invariant, for example that balances always equal postings, or that a reversal returns an account to its prior balance, and let a generator throw many sequences of operations at it, searching for the case that a hand-written test would miss.

Injecting the failures that matter

The dangerous failures happen between a write and a publish, so tests must inject them. A harness kills a worker after the local commit but before the relay, after the broker accepts a message but before the consumer commits, and during a batch partition, then asserts that the system recovers to exactly one effect. Idempotency is verified by abuse: double-submit tests send the same request twice, concurrently and after a delay, and assert one effect and one stored outcome; replay tests redeliver a captured stream, in order and shuffled, and assert that the inbox absorbs the repeats.

Drills, fixtures and parallel runs

Unknown outcomes are rehearsed with a payment adapter that can be told to time out, to acknowledge late, or to report an instruction that the system believes is pending. The drill confirms that the system queries before resending and never issues a duplicate. Reconciliation is tested with fixtures that contain deliberate breaks (a duplicate, an omission, a timing difference, a one-to-many case, and amounts inside and outside tolerance), and the test asserts that each is classified correctly and that only true breaks reach the queue.

Two practices close the gap to production. A parallel run drives the old and new systems with the same inputs during a migration and compares results before the new system takes over, guarding against silent divergence. And state machines are reviewed explicitly, as diagrams with every transition named, because an unreviewed transition is where an illegal move hides. All of these tests belong in the delivery pipeline, as gates that a change must pass before it reaches production.

Operating model and metrics

Consistency and reconciliation are run by people as much as by code, and the operating model names who owns what. Each reconciliation has two owners side by side: an engineering owner for the pipeline that produces it, and a finance or control owner who adjudicates the breaks and signs that the position is clean. A reconciliation with no business owner becomes a report that nobody reads; one with no engineering owner decays as formats change.

Breaks carry service levels. A high-value break is investigated within hours, a routine break within a few days, and every break has an age beyond which it is escalated. These service levels are tied to the month-end calendar, because an unresolved break at close either delays the books or is carried as a known exception with a reason. The calendar, not an average, defines what timely means.

Evidence for auditors

Auditors ask the system to show that money moved only when it should have and that every figure can be explained. The internal control framework of the Committee of Sponsoring Organizations of the Treadway Commission (COSO), and the requirements for internal control over financial reporting (ICFR) under the Sarbanes-Oxley Act, expect controls to be designed, operating and evidenced. A system built along the lines of this paper produces that evidence as a by-product: the history of an instruction, the reconciliation result for a date, the record of who approved what. Whether a given entity is in scope is a question for counsel.

The measure of a control is not that it exists, but that it produces evidence a stranger can trust.

Metrics and tooling

A few metrics show whether the discipline is healthy, and their trend matters more than their level. Reconciliation tooling can be built or bought, a trade-off to record rather than a matter of principle: packaged engines bring connectors, rules and case management and suit standard matching at volume, while a built engine fits unusual sources and keeps the logic beside the ledger. The same break lifecycle, ownership and evidence apply either way.

Recurring pitfalls

The failures of consistency and reconciliation are repetitive, and most invert a principle from the earlier sections. They are collected here so that a design review can check for each.

A staged path

No team builds all of this at once. A staged path puts the parts that are expensive to retrofit first and defers the rest until volume justifies it.

1. Foundations. A ledger with serializable or single-writer postings, idempotency keys on money-moving requests, explicit state machines with terminal states, and both effective and posting dates recorded from the start.
2. Effectively-once. The transactional outbox, idempotent receivers, retries with backoff under the same key, and a pending state with a status query for unknown outcomes.
3. Reconciliation and detection. A reconciliation engine for the first critical flow, control totals, the balanced-entry assertion, checkpoint rebuilds and invariant monitors, with breaks aged and owned.
4. Operating discipline. Break service levels tied to the close calendar, evidence on demand, metrics read as trends, and verification through fault injection, replay and reconciliation fixtures in the pipeline.

Conclusion

Keeping money and records aligned in a distributed system is not one technique but a chain of them, each covering a gap the last leaves open. Strong consistency protects the invariants where they live. Idempotency makes every effect happen once despite retries and redelivery. State machines, the outbox and compensation carry work safely across boundaries that no single transaction can span. Careful handling of time and order keeps the economic history true. And reconciliation, backed by integrity monitors, proves continuously that the records agree and resolves the differences that remain.

The commitments are few and are made early: post under strong isolation, carry a key on every money-moving request, never treat an unknown outcome as a failure, and reconcile from the first release with evidence that resolves every break. Payroll disbursement and debt payment allocation are then variations on one disciplined pattern rather than special cases, and the same pattern carries into billing, disbursements and any other system that moves money.

Glossary

CAP theorem

The result that, during a network partition, a replicated system cannot be both linearizably consistent and available; it must give up one of the two.

Compensating transaction

A new business transaction, such as a reversing entry, that semantically undoes a completed step rather than rolling it back.

Control total

A count of records and a sum of amounts used to confirm that a file or batch is complete.

Effectively-once processing

At-least-once delivery combined with deduplication, so that an effect happens once even though delivery may repeat.

End-to-end identifier

A reference set by the originator that travels unchanged through a payment chain, so that repeats can be detected and payments reconciled.

Idempotency key

A caller-supplied identifier stored with a request's first outcome, so that a repeat returns that outcome instead of acting again.

Idempotent receiver

A consumer, or inbox, that records processed message identifiers and ignores a message it has already handled.

Optimistic concurrency control

Letting transactions proceed and detecting conflicts at commit through a version or condition check.

PACELC

An extension of CAP: if there is a partition, a system trades availability against consistency; else, it trades latency against consistency.

Pivot step

The step in a saga after which the process is committed to completing, typically an irreversible external action.

Reconciliation break

A difference between two records that should agree and that will not resolve on its own.

Saga

A sequence of local transactions, each with a compensating action that runs if a later step fails.

Serializable isolation

The isolation level under which transactions behave as if they had run one after another.

Snapshot isolation

An isolation level that gives each transaction a consistent view of the data, which still permits write skew.

Three-way match

Matching an instruction, its acknowledgment and its statement line to confirm a payment end to end.

Transactional outbox

A pattern that writes an outgoing message in the same local transaction as the state change, for a relay to publish.

Two-phase commit (2PC)

A protocol in which a coordinator asks all participants to prepare and then to commit, which blocks participants if the coordinator fails between the phases.

Write skew

An anomaly in which two transactions make disjoint writes that together break an invariant that each alone preserved.