Tracing a Reconciliation Break — A Worked Example
A £4,812 difference traced through four hypotheses to a duplicate charge caused by a missing idempotency key — what the ledger showed, why the customer had not complained, and the two fixes.
This is an illustrative example. The scenario and figures are composed to show the method, not drawn from a named client engagement.
The daily reconciliation reports a difference. The processor's settlement file says one thing, the ledger says another, and the gap is £4,812.00.
What follows is the investigation, including the two hypotheses that were wrong.
What was known at 09:15#
| Amount | |
|---|---|
| Processor settlement file | £487,301.40 |
| Ledger, same period | £482,489.40 |
| Difference | £4,812.00 |
The difference is positive: the processor says more money moved than the ledger records. Money arriving that you did not expect is not good news — it means your record of what happened is incomplete.
Hypothesis 1 — timing (wrong)#
The obvious first answer. Transactions authorised just before the cut-off can settle on the other side of it.
Checked: the previous day's break was £0.00, and the following day's file reconciled cleanly against the ledger for that period. A timing difference would have reversed. Not timing.
Hypothesis 2 — a fee or adjustment (wrong)#
Processor fees, chargebacks and adjustments sometimes appear in settlement without a matching ledger entry, if the fee posting is a separate process.
Checked: the fee lines were present and already reconciled separately. Not fees.
Hypothesis 3 — a whole missing batch (wrong, but useful)#
£4,812.00 is a suspiciously round-ish figure. Was a batch of transactions missing from the ledger entirely?
Filtering the settlement file for transactions with no matching ledger entry returned nothing. Every settled transaction had a ledger pair.
That result reframed the problem: the break was not a missing transaction. It was an extra one — the same transaction settled twice.
Hypothesis 4 — duplicate charges (correct)#
Grouping the settlement file by the internal order reference produced six references appearing twice:
| Order ref | Amount | Times settled |
|---|---|---|
| ORD-88214 | £1,299.00 | 2 |
| ORD-88377 | £249.00 | 2 |
| ORD-88401 | £899.00 | 2 |
| ORD-88455 | £1,499.00 | 2 |
| ORD-88502 | £399.00 | 2 |
| ORD-88613 | £467.00 | 2 |
| Total duplicated | £4,812.00 |
Exactly the break.
The cause#
All six were submitted between 14:02 and 14:09, during a period when the processor's response times spiked. The application's HTTP client had a 10-second timeout and a retry.
The sequence: request sent · processor authorises · response takes longer than 10 seconds · client times out · client retries · processor authorises again.
The retry carried no idempotency key, so the processor treated it as a new payment — correctly, because it had no way to know otherwise. The ledger recorded one authorisation per order, because the application only saw one success.
Why nobody had complained#
Two of the six customers had already contacted support and been refunded, logged as "billing query" rather than as a duplicate charge. The other four had not noticed yet.
That is the part worth sitting with. Without the reconciliation, this would have surfaced over weeks as unconnected customer complaints, each handled individually, with nobody seeing the pattern or the cause.
The reconciliation did not just find money. It found a defect.
The two fixes#
Immediate: the four unrefunded customers were refunded that day, before they noticed, with an explanation. Ledger corrected by new entries — the original entries were not edited, because the ledger is append-only.
Permanent: a client-generated idempotency key per logical payment, stored server-side with the result, so a repeat returns the original outcome rather than authorising again. Tested by deliberately submitting the same payment twice and confirming one authorisation.
Additionally: the retry path now queries by internal reference before retrying, so a timeout is resolved rather than assumed.
What transfers#
A positive break is worse than a negative one. Money you did not expect means your record of events is wrong, and the cause is usually a duplicate.
Group by your own reference, not by amount. The duplicates were invisible in the totals and obvious the moment they were grouped by order.
A timeout is not a failure. It is an unknown, and treating it as a failure is precisely how this happens. See the payment flow diagram.
The break process is the control. The reconciliation ran daily and would have found this on any day it occurred. What made it useful was that somebody investigated it the same morning rather than carrying it forward.
See fintech and the launch readiness checklist.