Worked Example — The Agent That Passed Every Test
A worked example of an agent that scored 96% in testing and failed in its first week — why mocked tools guaranteed the result, and what replaced them.
This is an illustrative example. The agent, figures and timeline are invented. The cause — a test suite where every tool always succeeds — is close to universal in the first version of any agent test suite.
The situation#
A logistics company built an agent to handle delivery exception cases: find the shipment, determine what went wrong, rebook or refund, and tell the customer.
Testing was thorough by the standards the team knew. 220 cases, 96% pass rate, run on every change. It went live on a Monday.
By Thursday it had been switched off.
What happened in production#
| First 3 days | |
|---|---|
| Cases attempted | 1,840 |
| Completed correctly | 61% |
| Wrong action taken | 12% |
| Stalled and handed over | 27% |
Twelve per cent taking a wrong action is what stopped it. Among them: 14 shipments rebooked to the wrong depot, 7 refunds issued twice, and one customer told their parcel had been delivered when the tracking system had returned an error.
The suite had 96% pass. Production had 61%. Something about the suite was measuring a different system.
The cause#
Every tool in the test suite was mocked, and every mock always succeeded.
The mocks had been written from the API documentation, which describes the successful response. They returned a well-formed shipment record, in under 10 milliseconds, every time.
Production tools do other things:
| What the real tool does | Frequency | Mock behaviour |
|---|---|---|
| Returns a shipment with null fields | ~8% | Never |
| Times out | ~2% | Never |
| Returns two matches for one reference | ~3% | Never |
| Returns a stale record | ~1% | Never |
| Rejects a rebooking with a business-rule error | ~6% | Never |
| Succeeds slowly enough to matter | ~4% | Never |
Ninety-six per cent pass against tools that cannot fail. The suite tested the agent's reasoning on perfect inputs, which is the situation it handles best and the one that needs the least testing.
The double refunds are the clearest illustration. The refund tool occasionally returns a timeout after the refund has been issued. The agent, seeing a timeout, retried. In testing this never occurred, so the agent had never been observed retrying a non-idempotent action.
What replaced the mocks#
Recorded real responses. Three weeks of actual tool traffic was captured and turned into a response library — 1,400 real responses including every failure shape above. Cases now draw from it rather than from a hand-written ideal.
Deliberate failure injection. Every case runs three times: once clean, once with a randomly chosen tool failure, once with latency. A case only passes if the agent behaves acceptably in all three, where acceptable includes stopping and handing over.
Idempotency assertions. Any tool with a side effect now carries a case asserting that the agent does not repeat it after an ambiguous result. This is the assertion that would have prevented the double refunds, and it did not exist because the failure it guards against could not occur in testing.
Trajectory assertions. Cases assert which tools may be called, in what order, with what arguments — not just the final outcome. The wrong-depot rebookings all reached a plausible final state by a path that read the wrong record two steps earlier.
The rebuilt suite#
| Old | New | |
|---|---|---|
| Cases | 220 | 180 |
| Runs per case | 1 | 3 (clean / failure / latency) |
| Tool responses | Hand-written, always succeed | 1,400 recorded, including failures |
| Pass rate on first run of the new suite | — | 58% |
| Runtime | 6 min | 19 min |
Fifty-eight per cent. The agent that had scored 96% scored 58 against tools that behave like the real ones — and 58 is close to the 61% it achieved in production, which is the first sign the suite was measuring the right system.
Getting from 58 to production-ready#
Six weeks, and almost none of it was prompt work.
Tools were made idempotent where possible — refund and rebooking take an idempotency key, so a retry after a timeout is safe by construction rather than by the agent being careful.
Tool errors were made informative. error: failed became a typed error stating whether a retry could help. The agent stopped retrying things that could never succeed.
Ambiguous results became explicit. Two shipment matches for one reference now returns both with a flag, rather than the first. The agent asks rather than picking.
A hand-over path was designed rather than being what happens when the step budget runs out. The agent now recognises a small set of conditions — ambiguity, a business-rule rejection, an unclear tool state — and hands over immediately with the transcript.
The result#
| Launch | Relaunch | |
|---|---|---|
| Suite pass rate | 96% (mocked) | 91% (recorded, with failure injection) |
| Production completed correctly | 61% | 88% |
| Wrong action taken | 12% | 0.4% |
| Handed over | 27% | 11% |
| Gap between suite and production | 35 points | 3 points |
The last row is the one the team now watches. A suite whose result does not predict production is not a slightly optimistic suite — it is measuring a different system.
What was learned#
Mocks written from documentation test the happy path only. Documentation describes success; production supplies the rest.
Record real tool traffic. Three weeks of capture produced a library no team would have thought to write, including failure shapes nobody knew existed.
Test the agent against failing tools, not just failing inputs. The agent's reasoning was never the problem. Its behaviour when a tool misbehaved was, and it had never been observed.
Watch the gap between suite and production. Thirty-five points was the finding. Three points means the suite is worth running.