Seven Ways Deterministic Record Matching Breaks at Enterprise Scale

The control report says 97.0% auto-match. The number is true. It is also the least informative number on the page.

Underneath it sits a month of items that did not match, and the queue does not behave the way a residual is supposed to behave. A residual should be the hard tail of the same distribution — the same problem, only worse. Pull fifty items off the top of this one and you do not find hard cases. You find the same handful of shapes, over and over: an invoice settled in two transfers, one wire covering thirty-one invoices net of two credit notes, a receipt three pounds light after somebody's bank took a spread, an item booked on the last day of a period and settled on the first day of the next.

Not one of those is difficult. An analyst clears each in minutes by opening two screens and applying a rule that would fit on an index card. The engine could not, and not because the engine is weak. It could not because it was asked a question that none of these items is an answer to.

The queue is only the visible half of the failure. The other half sits inside the 97%: items matched confidently to the wrong counterpart, which nobody will ever look at, because the point of an auto-match is that nobody looks.

Synthetic data/Figures below are generated, not observed. No employer data appears on this site.

The 97.0% above, the seven sketches below and the table at the end are all invented.

Widening the rules does not converge

Each shape in that list gets a rule written for it. A band for the currency case. A subset search for the aggregated case. A wider window at period ends. Every one is locally correct, and the aggregate is a rule set that grows faster than the problem it covers.

Start from what the engine computes. For a ledger item $a$ and a settlement record $b$ it evaluates three tests and requires all of them

where $k$ is the reference key, $v$ the signed amount and $t$ the date. Each bracket is one when the condition inside it holds and zero otherwise, so $m$ is one only when all three hold together. The engine then resolves the surviving pairs into an assignment subject to $\sum_b x_{ab} \le 1$ and $\sum_a x_{ab} \le 1$: at most one $b$ per $a$, at most one $a$ per $b$.

Widening a tolerance does not make a match more likely to be right. It makes more pairs eligible. If settlement records for the counterparty arrive with local density $\rho$ per unit of value per unit of time near $(v(a), t(a))$, the expected size of the candidate set for $a$ is

the product of the two window widths. Double the amount band and double the date window and the candidate set quadruples. The engine still has to choose one, and it chooses by whatever secondary ordering it was configured with, which is usually date, which is usually arbitrary.

Nor does the rule set converge. With $c$ recurring classes, and items free to present several at once — a short-paid invoice, netted into an aggregated remittance, across a period boundary, in a second currency — a rule set that adjudicates class by class has $2^c - 1$ contexts it must be correct in. At $c = 7$ that is 127, and the seventh rule is written by somebody who has never read the first.

What the matcher assumes without saying so

Three assumptions do all the work in that expression, and none of them appears anywhere in the configuration.

Same value: the amount on one side equals the amount on the other. Same window: the two events happened close enough together on one shared measure of time. One-to-one: the correspondence between ledger items and settlement records is a bijection on the matched subset — one invoice, one payment. Underneath all three sits a fourth, stable key, which is the only one anybody recognises as an assumption, and therefore the only one with a standard remedy already attached to it.

Enterprise settlement is not a bijection. It is a many-to-many correspondence between sets, sometimes with sign changes, sometimes straddling a reporting boundary, whose members are re-keyed in transit. Written honestly, the object being recovered is a pair of subsets

which is subset-sum in a business suit — the problem Karp listed as Knapsack among his original twenty-one. Tolerance is a parameter of the wrong model, so no setting of it arrives at the right one.

The seven classes

A matcher applies its predicate in a fixed order: block on the reference key to build a candidate set, compare amounts, compare dates, resolve the survivors one-to-one. Four gates. An item is reported at the first gate that rejects it, and that gate is frequently not the assumption that actually broke.

That gap is why the tolerance always ends up attached to the wrong parameter. A split settlement is a cardinality failure reported as an amount mismatch, so an amount tolerance is what gets added, and an amount tolerance cannot fix cardinality. It only widens $\epsilon$ until the wrong things match.

1. Split settlement

An invoice for 88,400.00 is settled by two transfers: 60,000.00 on the 14th and 28,400.00 on the 22nd. Both carry the invoice reference. Both are obvious to anyone who looks at them.

The key gate passes, because the reference survived. The value gate rejects both transfers, because neither equals the invoice. The engine emits three exceptions for one event: two unapplied receipts and one invoice still open at its full face value. Nobody reading the queue can see that the three are the same thing, because a queue is a list of records, not a list of events.

The assumption that broke is one-to-one. The gate that fired was value. Widening $\epsilon$ far enough to swallow a 28,400.00 gap is not a tolerance but an abdication, so nobody does it, and the class stays in the queue permanently.

2. Aggregated remittance

A single wire of 412,905.66 settles thirty-one invoices, net of two credit notes, from a counterparty that pays on a fortnightly cycle. The remittance advice arrives separately, as a spreadsheet attached to an email, in a layout the sender changes without telling anyone. Sometimes it does not arrive.

The reference field on the wire carries the payer's own batch number, so the key gate has nothing usable to block on and the engine falls back to the account. Every open item on the account becomes a candidate, and the value gate rejects all of them. One exception is reported. Thirty-three items are unresolved.

Some engines answer with a subset rule that hunts for the combination of open items summing to the wire. On an account with four hundred open items that is subset-sum, and an engine attempting it must either cap the search depth, which misses the true combination, or accept the first combination inside tolerance, which finds a false one and closes it.

3. Currency conversion and rounding drift

An invoice is raised for 47,318.20 in the counterparty's currency. The payer converts at their own rate on their own value date, the receiving bank takes a spread, and the sub-ledger was revalued at the month-end rate. The receipt lands 3.41 away from the carrying amount. The same class covers allocation remainders: a 100.00 charge split three ways is 33.33, 33.33 and 33.34, and the third one never matches anything.

Here the violated assumption and the firing gate agree. Value equality is genuinely false, and the value gate says so. This is the only one of the seven where a tolerance is the correct instrument.

It is also the class that poisons the other six. Set $\epsilon$ to 5.00 to absorb the drift and every open item within 5.00 of a receipt becomes a candidate. On a high-volume account that is never one item.

4. Timing across a period boundary

A payment is initiated on the 31st, value-dated the 1st, and posted to the bank statement on the 2nd. The ledger item it settles was booked in the period that closed on the 31st. Nothing is wrong with any of those dates. They are four correct answers to the question when, from four systems that were each asked a slightly different question.

The window gate rejects the pair at $\delta = 3$ days and accepts it at $\delta = 10$. That looks like a free fix. It is not: a ten-day window at a period boundary reaches back into a period that is closed, reported and signed. The tolerance that repairs the match breaks the accounting, and it does so quietly, because a matched item raises no exception and nobody reopens a closed period to check.

5. Reference-field mutation

An invoice leaves the billing system as INV-2026-0088412. The payment file format truncates the reference to twelve characters. The counterparty's payables system substitutes its own purchase-order number. What arrives in the bank statement narrative is PO 44120/88412 PART.

This class fails differently from the rest. It does not produce a bad candidate, it produces no candidate, because the key gate is also the blocking step: the item never enters a comparison at all. The symptom is a null result, which is why the class is systematically over-represented in queues triaged by exception message rather than by cause.

Approximate string comparison recovers much of it. It also introduces the failure the record-linkage literature spends most of its own time on: a confident match to a neighbouring reference.

6. Partial settlement and short pay

An invoice for 12,600.00 is paid at 12,180.00. The 420.00 difference is not drift. It is a deduction: a damaged-goods claim, an unearned early-payment discount, a disputed freight charge, a promotional allowance the counterparty has decided to take without asking.

Structurally this is class 3. The value gate rejects it, value equality is what broke, and the two shapes are indistinguishable to the engine. The difference between them is intent, and intent is not a field.

Which is why this is a control problem rather than a matching problem. Any amount tolerance wide enough to absorb currency drift absorbs small deductions too, and absorbs them silently. The item closes, the receivable clears, and a write-off never reaches anyone with the authority to approve it. The engine has not made a matching error. It has made a policy decision on nobody's behalf.

7. Reversal and rebook

An item is posted, reversed the next day, and reposted with corrected coding under a new document number. Three records, one economic event, and a net of zero across the first two.

The payment, when it arrives, quotes the original document, which is now closed. The key gate rejects it. But the violation underneath is arity: three ledger records correspond to one settlement, and the engine has no representation for that at all.

The degradation here is the worst in the taxonomy. If the rebook carries the same amount as the original — and it usually does, because only the coding changed — then the reversal is a perfect amount-and-date match for some other open item of the same value. The engine can close two items, both wrongly, and because the pair nets to zero every control total still agrees. A balance identity is a necessary condition and not a sufficient one, which is a failure that recurs wherever an identity is used as a validity check.

A worked example

The population below is internally consistent — the counts sum, the minutes are counts times a median handling time — and calibrated against nothing. Treat its shape as a hypothesis about where effort concentrates.

One monthly cycle on a mid-sized receivables book: 240,000 items presented to the matcher, 232,800 auto-matched, 7,200 exceptions. The headline is 97.0%. I would expect the same seven shapes on bank, intercompany and custody reconciliations with different weights on each class, and that portability is the part of the taxonomy I would defend hardest.

That is 7,200 items and 67,710 analyst minutes, or 1,128 hours. Across a twenty-one day cycle it is a little over seven full-time analysts, and every one of them is working the three per cent. Two classes alone — aggregated remittance and reference-field mutation — are 22.2% of the items and 47.4% of the minutes.

Two things in that table matter more than the headline.

The first is the inversion. Currency drift is 29.3% of the queue and 6.2% of the effort: high volume, trivial to clear. Aggregated remittance is 13.6% of the queue and 31.8% of the effort. A dashboard that ranks classes by queue depth ranks them almost exactly backwards, and an automation programme that follows the dashboard spends its budget on the cheapest class in the book.

The second is what the table cannot contain. Draw 1,000 items at random from the 232,800 that auto-matched and have an analyst re-derive each one from source. Suppose six come back wrong. That is a point estimate of 0.6%, with a 95% interval of roughly 0.2% to 1.3%, and 0.6% of 232,800 is about 1,400 items — larger than six of the seven exception classes taken individually, in a population that by construction nobody inspects. There is no queue for these. The only way to see them is to go looking, which is the evaluation-without-ground-truth problem wearing different clothes.

Limits

Three honest ones.

The taxonomy is descriptive and I have not built the classifier. Assigning an unseen exception to one of seven classes is itself a matching problem, the boundaries blur — a short-paid invoice, netted into an aggregated remittance, across a period boundary is three classes and one item — and until that assignment is automatic the taxonomy is a way of thinking rather than a control.

The classes are not mine, and neither is most of the machinery. Record linkage has a sixty-year literature: Newcombe on automatic linkage of vital records, the Fellegi and Sunter probabilistic model, Winkler's string comparators and expectation-maximisation estimation at the US Census Bureau, Christen's synthesis of blocking and entity resolution. The many-to-many amount problem is subset-sum, and one-to-one resolution is the assignment problem Kuhn solved in 1955. What I claim is the framing: that these seven are violations of a small fixed set of unstated assumptions, and that the gate reporting a failure is usually not the assumption that broke.

The numbers are the weakest part. The item counts are plausible; the analyst-minute distribution is a guess, and it is the guess the whole argument about misallocated effort rests on.

Where this goes

The reframe, taken at its word, is the piece that precedes this one. If what the engine should recover is a correspondence between subsets, across systems and across time, then reconciliation is a graph problem and the seven classes are seven edge types rather than seven exceptions. What comes next is the part I have not built: a classifier that routes an unseen exception to the assumption it violated rather than to the gate that reported it. The rest of the pillar sits on the research page.

The question worth holding onto is not what your auto-match rate is. It is this: of the items that auto-matched last month, how many has anyone ever checked?