Why Data Masking Is Not Enough for Enterprise AI

A finance team wires an assistant into their systems. Legal asks the reasonable question — what happens to our data? — and engineering gives the reasonable answer: it is masked. Account numbers are tokenised. Names are replaced with surrogates. Amounts are bucketed. Nothing identifying reaches the model.

Six weeks later someone asks the assistant why a particular month closed late. It answers: the two largest counterparties in that region settle on a fourteen-day cycle, one of them nets across subsidiaries before remitting, and the reconciliation rule that tolerates a three-day variance does not apply to netted remittances, so those items age into the exception queue.

Every field in that answer was masked. The answer was not.

A redaction that redacted nothing

The masking worked exactly as specified. No account number appeared. No legal entity was named. The surrogate keys held. If you audited the request payload you would find nothing that a data-loss-prevention rule would flag, because there was nothing in it to flag.

What leaked was not a field. It was a shape: how many counterparties concentrate the volume, how they are related to each other, on what rhythm they pay, and which internal rule they collide with. A competitor who learned only that would know something worth knowing. An adversary who learned it would know where to push.

This is the failure mode that keeps recurring in enterprise AI deployments, and it is not a masking bug. It is a category error. Masking is a control on the representation of data. The thing an enterprise needs to protect is mostly not in the representation.

Why more masking does not fix it

The instinctive response is to mask harder — tokenise more fields, coarsen the buckets, strip the free text. This runs into a wall that the statistical disclosure control literature mapped out decades before anyone attached a language model to a warehouse.

The first problem is that the useful signal and the sensitive signal are frequently the same signal. If you coarsen settlement timing enough to hide the fourteen-day cycle, you have also destroyed the assistant's ability to answer any question about settlement timing — which is what it was deployed to do. Masking trades utility for confidentiality along a single axis, and on that axis the enterprise use case sits uncomfortably close to the sensitive end.

The second problem is that masking is per-record and the disclosure is cross-record. Replacing supplier ACME-4471 with S_00318 everywhere is consistent pseudonymisation. It is also a perfect structure-preserving map: the adjacency of the graph, the degree distribution, the concentration ratios, and the temporal pattern all survive the substitution unchanged. You have renamed the nodes and kept the network.

The third problem is that masking is stateless and disclosure is cumulative. A masking function sees one payload. It has no way to know that this is the fortieth question in a sequence whose answers, taken together, reconstruct something no single answer contained.

The reframe: disclosure is a property of the session

The useful move is to stop asking what is in this payload and start asking what does the counterparty know now that they did not know before.

That reframing is not new — it is how disclosure risk has been reasoned about in official statistics since the 1970s. What is new is the delivery mechanism. A statistical agency releases a table once and can reason about it as a fixed artifact. An enterprise assistant emits a fresh, query-shaped, natural-language release every few seconds, tailored to exactly what the asker wanted to know, for as long as the asker keeps asking.

Once you accept that framing, the three failures above stop looking like three unrelated bugs and start looking like three axes of the same object. I have been calling the resulting framework DRRC — disclosure risk and rule confidentiality — and it sits in the research programme alongside the reconciliation work it grew out of. The rest of this piece is the threat model, stated plainly enough to be argued with.

Three leaks masking cannot reach

Relationship disclosure

Pseudonymisation preserves structure by construction. That is the point of it — consistent surrogates are what make the data still joinable, and joinable data is what makes it useful. But a graph whose node labels are surrogates and whose edges are intact is still the graph.

Concretely, if an assistant can answer which of our suppliers also supply our competitors or which counterparties cluster into the same settlement behaviour, it is answering a question about edges. No node label is required. And the structural properties that make such a graph re-identifiable — a handful of very-high-degree nodes, an idiosyncratic bipartite pattern, a distinctive timing signature — are exactly the properties that survive every masking scheme that preserves referential integrity.

Rule disclosure

The second leak is the one enterprises consistently underweight, because it does not look like data at all.

An assistant that is useful in a finance function has to know the rules: what tolerance applies to which class of item, which exceptions route to which reviewer, what makes a match acceptable. Those rules are frequently the most valuable thing in the building. They are the encoded outcome of years of operational learning, and unlike the underlying data they are not protected by any regulation — nobody's rule set is personally identifiable information.

So the rules sit outside the scope of the privacy programme, get loaded into context to make the assistant work, and are then recited on request, in fluent prose, to whoever asked. The rules in question are frequently the tolerance and exception logic that sits on top of a reconciliation process — which is worth reading about on its own terms, because the reason those rules are valuable is the same reason they are complicated.

Cumulative disclosure

The third leak is the one that defeats per-request review, and it is worth being precise about.

Model the sensitive quantity as a random variable $X$ — the thing you do not want disclosed. Let $a_1, a_2, \dots, a_k$ be the answers the system has emitted so far. Conditioning never increases entropy, so

$$H(X \mid a_1, \dots, a_k) \;\le\; H(X \mid a_1, \dots, a_{k-1})$$

for every $k$. Uncertainty about $X$ is monotonically non-increasing across the session. It does not recover between requests.

A per-request control tests the increment

$$\Delta_k \;=\; H(X \mid a_{<k}) - H(X \mid a_{\le k})$$

against some threshold, and passes every request for which $\Delta_k$ is small. But the quantity that actually determines whether you have disclosed $X$ is the total

$$I(X; a_{\le k}) \;=\; H(X) - H(X \mid a_{\le k}) \;=\; \sum_{j \le k} \Delta_j$$

and a sum of arbitrarily small terms is not arbitrarily small. Forty questions that each move the needle by two percent do not leave you with ninety-eight percent of your uncertainty intact. This is the same accounting that differential privacy formalises as a composition budget, and the same accounting that almost no enterprise AI deployment does at all.

A worked example

Synthetic data/Figures below are generated, not observed. No employer data appears on this site.

Take a synthetic supplier ledger: 400 suppliers, 12 buying entities, 18 months of invoices. Every identifier is replaced with a consistent surrogate, every amount is bucketed to the nearest 5%, and every date is jittered by up to two days. By any per-field audit, this data is masked.

Now ask three questions that a masked-data policy would wave through, because none of them requests an identifier:

"Which supplier accounts for the largest share of invoice volume for entity E1?" The answer is a surrogate key. It is also a statement that one supplier dominates a specific entity, and the degree distribution that produced it is unchanged by masking. That is relationship disclosure.

"Why did this item not auto-match?" To be useful the assistant explains the tolerance and the exception path. That is rule disclosure, and no masking scheme touches it, because the rule was never in the data.

"How does that share compare to the same month last year, and to the other eleven entities?" Individually a trend question. In sequence with the previous two, it is the third term in a sum that is quietly converging on the supplier concentration profile of the entire business. That is cumulative disclosure.

None of the three requests contained a protected field. All three were answered correctly. The masking never failed.

Limits

Three honest ones.

The framework above names failure modes; it does not price them. Saying that cumulative disclosure exists is easy, and every practitioner nods. Estimating $I(X; a_{\le k})$ over an open-domain natural-language session, cheaply enough to run inline, is the hard part, and I am not claiming to have solved it here.

It is also not the case that masking is useless. Masking is a good control against exactly the threat it was designed for — regulated identifiers reaching systems that should not hold them — and dropping it would be a straightforward downgrade. The argument is that it is necessary and not sufficient, and that the gap between those two words is where enterprise deployments are currently sitting.

Finally, the framing is mine and the failure modes are not. Relationship disclosure is graph de-anonymisation, cumulative disclosure is composition, and both have substantial literatures. What I think is under-served is the enterprise-confidentiality case — where the asset is a business relationship or a business rule rather than a person — because that case falls outside every privacy regulation that funds this research.

Where this goes

Next in this pillar I take relationship disclosure on its own terms: what an anonymised supplier graph actually gives away, and why the structural properties that make a graph useful are the same ones that make it identifying. After that, rule confidentiality — the leak nobody has a control for, because it does not look like data. And if the question you are actually being asked is what evidence a control tester will accept for any of this, that is a different conversation.

If you deploy assistants inside a business with something to protect, the question I would ask first is not what did we mask. It is what does the session know.