Worked Example — The Suite That Passed 100% and Caught Nothing
A worked example of a test suite that had been quietly disarmed — how assertions were loosened one incident at a time, what the audit found, and the rule that stopped it recurring.
This is an illustrative example. The team, timeline and figures are invented. The mechanism — a suite loosened one reasonable decision at a time until it asserted nothing — is ordinary enough that most teams have some of it.
The situation#
A product team shipped an AI feature that generated shift handover notes for a hospital ward. The suite had 180 tests and had passed on every run for eleven weeks.
In week twelve a nurse reported that a handover note had listed a medication that was not in the record. Investigation found the same class of error in 14 notes over the previous month.
The suite had passed on every one of the days those notes were generated.
What the tests were actually asserting#
The audit took an afternoon and was uncomfortable reading.
| Tests | What it checked | |
|---|---|---|
| Asserted the response was non-empty | 71 | That something came back |
| Asserted the response contained any of 3+ common words | 44 | Effectively nothing |
| Asserted valid JSON, no field checks | 28 | Shape only, contents unchecked |
| Asserted a specific fact was present | 24 | Genuine |
| Asserted something must not appear | 3 | Genuine |
| Skipped | 10 | — |
Twenty-seven of 180 tests could fail. The remaining 153 would pass on any output the model produced, including an empty summary, a refusal, or a note about a different patient.
How it got that way#
Nobody disarmed the suite. It happened one reasonable decision at a time, and each decision was defensible on the day it was made.
Week 1. Tests asserted exact output. They failed constantly, because the model does not produce identical text twice. Engineers spent more time updating tests than writing code.
Week 2. Exact matching was relaxed to substring matching. Better, still fragile — "patient was administered" versus "patient received" broke tests that were describing correct behaviour.
Week 4. Substring matching was relaxed to "contains at least one of these keywords". This is the change that removed the suite's ability to fail, and it went through review without comment, because it looked like the same kind of pragmatic loosening as the week before.
Week 6. Two tests that still failed intermittently were skipped, with a comment saying they would be revisited.
Week 9. A new engineer, copying the established pattern, wrote 40 new tests in the keyword style. That is now the house style.
Every step was a response to a real problem — a brittle suite that cried wolf. The problem was real and the remedy removed the alarm rather than fixing it.
What was rebuilt#
The suite was reconstructed over two weeks around a rule: an assertion that cannot fail is not an assertion. Every test had to be demonstrated failing against a deliberately broken output before it was accepted.
That single requirement — write the test, then prove it can fail — eliminated 153 tests in an afternoon, because none of them could be made to fail by any means.
The replacement suite separates what may vary from what may not:
Structural, must be identical every run. Valid JSON, required fields present, medication list is an array, every entry has name and dose, no field null.
Factual, checked by presence not wording. A transcript stating a medication was administered must produce a note containing that medication. Phrasing free, fact required.
Negative, must never appear. No medication not in the source. No patient name other than the subject. No date outside the shift window. These three did not exist before, and the first of them is the assertion that would have caught the original defect on day one.
Repetition. Every case runs five times. The original defect appeared roughly one time in seven, so a single-run suite would have missed it even with the right assertion.
The result#
| Before | After | |
|---|---|---|
| Tests | 180 | 96 |
| Tests that can fail | 27 | 96 |
| Runs per case | 1 | 5 |
| Runtime | 4 min | 11 min |
| Defects found in the first week | — | 6, including two unrelated to the original |
Ninety-six tests that can fail found six defects in a week. One hundred and eighty that mostly could not had found none in eleven.
The two unrelated defects are the part the team found most sobering. Both had been present for weeks, both were the kind a person notices immediately, and neither had been reported — because the notes were reviewed by people who assumed the system had been tested.
What was learned#
Prove every assertion can fail. Writing the test and then breaking the output on purpose takes an extra minute per test and is the only thing that distinguishes a test from a decoration.
Brittleness and looseness are both failures, and the cure for one produces the other. The right response to a suite that fails on wording is to assert on facts, not to assert on nothing.
Negative assertions are the ones that matter and the ones nobody writes. All three added here were absent, and one of them was the whole incident.
Run non-deterministic tests more than once. A defect at a one-in-seven rate passes a single-run suite most days, and each pass makes the team more confident.