At step nine the agent asked for a supporting schedule it had already asked for at step seven, received the same transport error, and asked again. At step twenty the cap fired, and it wrote three fluent paragraphs describing a close pack it had not assembled.
Two teams in the same organisation had built agents for that job: walk the period-end checklist, pull each account's balance and its prior period, fetch the schedule behind it, flag any movement over the stated threshold, write the commentary line. Same base model, same API, same weights. They built the evaluation set together from a shared task inventory, so there was no argument about what counted as success.
One team's agent finished most of the set. The other produced that transcript, or something close to it, on a third of the set.
The post-mortem was written against the model. Not ready for multi-step work in this domain. Revisit after the next release.
Nothing about the model differed. What differed was the tool surface — a wrapper for every internal endpoint on one side, task-shaped operations on the other. What an error looked like when a call failed. What was still in the context window at step nine. Whether a retry was permitted to repeat itself. Whether anything at all checked the work before the loop declared victory.
Those decisions were made by four people over five weeks, none of whom thought they were making a decision.
Why upgrading the model does not close the gap
The instinctive fix is to wait for a better model. It is a reasonable instinct: model capability is the one variable that improves without anyone in the building doing anything, and there is a leaderboard to watch meanwhile.
Write task performance as , determined by the model and the harness around it. Let both vary — across generations, across the harnesses a competent team might plausibly build for this task. The law of total variance splits the outcome without any modelling assumptions at all:
The first term is the spread from swapping models inside a fixed harness — the quantity a model evaluation reports. The second is the spread from swapping harnesses, and almost nobody measures it, because the postmortem holds fixed at whatever it happened to be and never registers that it was a variable.
If the second term is the larger one on your task, a model upgrade is a small multiplier on a large error. It lifts both harnesses. The gap between the two teams survives it, because the gap was never in the term the upgrade touches.
There is a second-order problem underneath. A harness is tuned, informally, to the failure modes of the model it grew up around: the retry rule that exists because the old model kept malforming one argument, the prompt paragraph that exists because it once looped on a null result. Swap the model and those adaptations do not go neutral, they go stale. Upgrading without re-deriving the harness has a real chance of regressing.
And you cannot diff what nobody wrote down. Ask a team to produce the harness that ran and you get a prompt template, a helper module, a truncation buried in an HTTP client, and a retry decorator someone added on a Friday.
The harness is an artefact
The move is to stop treating the code around the model as glue and start treating it as an engineered artefact with a specification, a version, an owner and a changelog. It is not plumbing between a model and a business. It is the entire interface between a general-purpose reasoner and a specific institution: every affordance the model has, every fact it can see, every consequence it can cause, and every signal it gets back about whether any of it worked.
Stated that way the asymmetry is hard to miss. Choosing a model is a procurement decision with a benchmark, a review and a meeting. Choosing a harness is forty small decisions spread across a quarter, each locally reasonable, none recorded.
A model is chosen once, deliberately. A harness is chosen a hundred times, by whoever was closest to the keyboard.
Once the harness is a named artefact you can enumerate its surfaces, and once you can enumerate them you can ask the only question worth asking about each one: was this decided, or did it default?
Six surfaces
Each has dimensions you can name, and a failure it defaults to.
The tool surface
Its dimensions are granularity, naming and idempotence.
Granularity defaults to a one-to-one mirror of an internal service API, because that API already exists and wrapping it takes an afternoon. But its granularity was chosen for a different consumer — a service-to-service caller with a type checker, a retry library and a developer reading reference documentation. A model has none of those. Too fine, and the agent spends its budget on assembly with a selection error available at every step. Too coarse, and a single tool takes a free-form argument encoding an entire query, so every mistake surfaces as one opaque failure a long way from the intent that caused it. The granularity that works is the granularity of the task, which is rarely the granularity of the service.
Naming is not cosmetic: name and description are the entire basis on which the model discriminates between tools. Two operations whose names differ by a qualifier will be confused, and the confusion is silent — the near neighbour returns plausible data and the run continues, wrong, with no error anywhere.
Idempotence needs an explicit answer per tool, because retry is the loop's main recovery move. Left to default, nobody labels it, and the harness ends up either retrying everything — which turns a timeout into a duplicate write — or retrying nothing, which turns a transient failure into a dead run.
Error semantics
Of the six, this is the one most consistently inherited rather than designed.
An error returned to a model is not a log line. It is the next thing in the context window and the only evidence the model has for choosing its next action. It is prompt, not telemetry.
Three properties decide whether it works. Does it say what failed. Does it say
why, in terms of the action that was taken. Does it say what would have been
valid instead. 500 Internal Server Error has none of the three.
Invalid parameter has the first. end_date '2026-02-30' is not a calendar date; supply an ISO date inside the open period has all three.
The consequence is not ergonomic, it is control flow. An error the model cannot act on produces a retry loop: with nothing telling it what to change, the model re-issues substantially the same call with cosmetic variation until the budget is gone. An error that names the violated constraint and the admissible values produces a recovery in one step, and the transcript reads like a competent operator hitting a snag. Same failure, same tool, same model, different string.
- A stable code an alert can fire on
- A stack trace that locates the defect
- Terseness, so the log stays cheap
- The constraint that was violated, named
- The argument that violated it, quoted back
- The values that would have been accepted
- Whether repeating the call is safe
Context assembly
The dimensions here are what enters the window, why, and what leaves.
The default is append-everything, then drop from the front on overflow. That policy is precisely backwards. The oldest tokens are the task definition, the constraints and the acceptance criteria — the things that stayed true for the whole run. The newest tokens are the last tool payload, frequently four thousand rows of which two matter. The default evicts the instructions and preserves the noise, which is why long runs drift into confident irrelevance shortly after the first overflow.
The deliberate version pins what is invariant, compresses tool output at the moment of receipt rather than at the moment of pressure, and maintains a short explicit record of what has been established so far. Compaction is a lossy rewrite of the record, which makes it a control in its own right — the evidence a reviewer reads afterwards is whatever survived it, and that is one reason agents in SOX-controlled environments are harder than they look. And what is in the window is also what leaves the building, which is a separate problem masking does not solve.
The control loop
Stop conditions, retry policy, and how failure is represented back to the model.
The default stop condition is a step cap, which is not a stop condition but a timeout wearing one. It fires at the worst available moment, mid-sequence, leaving partial state that nobody designed for. Real stop conditions are positive: a post-condition holds; the model asks to stop and a check agrees; the model requests something it is not permitted to have; progress has stalled — the same call producing the same error twice.
Retry policy has one rule that costs nothing to adopt and changes the shape of every transcript: a retry must alter at least one thing — the argument, the tool, or the plan. A retry that alters nothing is a loop with extra billing. Two identical failures should escalate out of the loop into a report, not into a third attempt.
State
The dimension is what persists between steps, and who owns it.
There are three places state can live. The context window: visible to the model, lossy, and rewritten every time you compact. The harness's own store: durable, and invisible to the model unless something deliberately surfaces it. The system of record: authoritative, and being mutated by the agent's own actions while it runs. Left to default, state lives only in the transcript, which means compaction silently edits it and the model's belief about the world diverges from the world the moment a call half-succeeds.
The question to answer explicitly is which of the three wins when a fact appears in two of them, and what reconciles them when they disagree. That is the same question a distributed system asks about a cache, and refusing to answer it produces the failure everyone recognises: an agent that reports completing a step it started and abandoned.
The verification boundary
The dimension is where the check lives, and there are three options in increasing order of strength — the model's own judgement, a post-condition evaluated by the harness, a constraint enforced by the target system.
Left to default there is no boundary, so the evidence that a task was done is the agent's own assertion that it was done. The model observes only what tools return to it; if nothing re-reads the world after a write, done means no more than I emitted a call that did not raise. The boundary earns its keep twice: at run time it is the stop condition worth having, and at review time it is the only thing that produces evidence someone outside the team can accept. Where no ground truth exists to check against, evaluating agents without ground truth becomes its own problem.
A worked example
Synthetic dataFigures below are generated, not observed. No employer data appears on this site.
Take a constructed suite of 120 close-pack tasks over a synthetic set of books, in four classes, run against one model under two harnesses. Harness A mirrors an internal service API tool for tool, passes transport status codes through as errors, appends and truncates from the front, caps the run at twenty steps, retries a failed call unchanged, and keeps no record outside the transcript. Harness B exposes nine task-shaped operations, returns errors carrying the violated constraint and the admissible values, pins the task specification and a running fact record, requires each retry to change an argument or a tool, and checks a post-condition before a write counts as done.
These numbers are constructed for this suite, not measured on any system. They illustrate a shape; the claim they carry is structural, not quantitative.
| Task class | Tasks | Harness A | Harness B |
|---|---|---|---|
| Single lookup, one tool call | 40 | 92% | 95% |
| Multi-step retrieval, no failure injected | 30 | 68% | 87% |
| Recoverable tool failure injected | 30 | 24% | 79% |
| Long horizon, fourteen or more steps | 20 | 15% | 61% |
| Weighted total | 120 | 56% | 83% |
Three things are worth reading off that table.
The first class barely separates them, 92 against 95. That is the class most benchmarks are built from — one call, one answer, no recovery, no memory — and it is the class least sensitive to everything this piece is about. A leaderboard delta measured on single-call tasks transfers to your agent in proportion to how much of your work looks like single-call tasks.
The gap opens where recovery is required: 24 against 79 on the class with an injected tool failure. In one harness the failure arrived as a status code; in the other it arrived as a sentence naming the constraint and the admissible values.
It opens again on the long-horizon class, 15 against 61, and the cause is eviction. The runs that failed under Harness A did not fail at the step that needed step-two evidence. They failed earlier, at the moment that evidence was dropped to make room for a payload.
Limits
Three honest ones.
The hypothesis is unproven, as labelled. If a future model became reliable enough to recover from uninformative errors and to rebuild an evicted task specification unaided, the harness term would shrink and the argument with it. I expect that shrinkage to be partial rather than total.
Very little here is novel, and the parts that are not have literatures with names. Tool granularity and naming is affordance design in Donald Norman's sense, dressed in JSON. That a notation's usability decomposes into enumerable dimensions rather than into taste is Green and Petre's cognitive dimensions of notations. Error-message design has its own literature in programming education; Brett Becker and colleagues on compiler error messages is where I would start. A harness is also, structurally, an environment with a language model inside it, which reinforcement learning has been shaping for decades. The enumeration is mine. The ideas underneath it are not.
The framing has a failure mode of its own. A harness that compensates for a weak model becomes scaffolding a strong one has to work around, and every surface named here is a place to over-engineer. Writing the harness down makes its coupling to a model version visible, not absent.
Where this goes
Next in this pillar is the measurement problem the verification boundary walked into: how to evaluate an agent when nobody can supply the ground truth its output would be scored against, which is the normal condition in reconciliation and control work rather than an edge case.
If you are about to file a post-mortem against the model, the question I would ask first is not which model was this. It is can anyone here produce the harness that ran.