Behavioural Regression Report — Sample
A worked example of testing an AI system after a model version change — what moved, what stayed, the improvements that broke downstream systems, and the decision the numbers supported.
Markdown. No sign-up, no email.
This is an illustrative example. The system, versions and figures are invented. The situation is not: a provider updates a model, nothing in your code changes, and the behaviour does.
Behavioural regression — model version change#
| System | Document classification and extraction, 40,000 documents a month |
| Change | Provider model version updated. No change to our code, prompt or data. |
| Notice given | 30 days, with a deprecation date for the previous version |
| Suite | 600 held-out documents, plus 240 case-based behavioural tests |
| Run | Both versions, same day, same inputs |
1. Should we take the update?#
Yes, with two changes first, and a two-week overlap.
The new version is better on the measures that matter and worse on two that break downstream systems. Both are fixable in a day. The overlap exists because two behaviours changed in ways the suite could not have anticipated, and there will be others it did not find.
2. Aggregate#
| Previous | New | Change | |
|---|---|---|---|
| Document type correct | 94.3% | 95.8% | +1.5 |
| All 6 fields correct | 86.9% | 89.4% | +2.5 |
| Refused or empty | 0.8% | 0.5% | −0.3 |
| Median latency | 1.4 s | 1.1 s | −0.3 s |
| Cost per document | $0.049 | $0.041 | −16% |
Better on every aggregate measure. If this were the whole report, the decision would be trivial — which is exactly why aggregates are not the whole report.
3. What moved, case by case#
Of 600 documents, 71 changed outcome. Aggregates hid most of this: 44 improved and 27 regressed, netting to +17.
| Count | |
|---|---|
| Was wrong, now correct | 44 |
| Was correct, now wrong | 27 |
| Both wrong, differently | 9 |
The 27 regressions are the report. Grouped:
| Regression | Count | Note |
|---|---|---|
| Handwritten annotations now ignored | 11 | Previously extracted, now skipped |
| Two document types merged | 9 | Similar layouts, distinguished before |
| Multi-page documents truncated at page 4 | 5 | Only affects documents over 4 pages |
| No pattern found | 2 | — |
The handwritten case is the one to note. Nobody knew the previous version handled handwritten annotations, because nobody had tested for it. It was working by accident, a downstream process had come to depend on it, and the dependency became visible only when it stopped.
4. The improvements that broke things#
Two behaviours improved and broke downstream systems in doing so. This is the category teams do not anticipate.
Dates are now normalised to ISO format. Previously returned as written. The downstream system parses the previous formats and fails on ISO. Strictly an improvement; entirely a breaking change.
Currency values now include the symbol. Previously bare numbers. The receiving field is numeric and rejects the string.
Both were found by the behavioural suite because it asserts on output shape, not only on correctness. An accuracy-only suite would have scored both as improvements and passed them straight into a production failure.
5. What stayed the same#
Worth checking, and worth reporting.
- Refusal behaviour on documents outside the 14 types is unchanged
- No new failure mode on poor-quality scans
- Prompt injection attempts in document text still fail, all 40 cases
- Field-level confidence remains uncalibrated in both versions — high confidence does not predict correctness in either
The last one is a standing finding rather than a regression. It has been true for three versions and it is why the empty-field routing exists.
6. Cost of the two changes#
| Effort | |
|---|---|
| Accept ISO dates downstream | Half a day |
| Strip the currency symbol before the numeric field | Two hours |
| Add handwritten annotation cases to the suite | Two hours |
| Total | Under a day |
Against a 16% reduction in cost per document, about $330 a month at current volume, plus the accuracy improvement.
7. Recommendation#
- Make the two downstream changes before switching.
- Add the 11 handwritten cases to the suite so the behaviour is asserted rather than assumed, whichever version is running.
- Run both versions in parallel for two weeks, comparing on live traffic, before retiring the old one. The suite found 71 differences; production will find more.
- Accept the truncation regression for now. Documents over 4 pages are 1.2% of volume and they already route to a person.
- Diarise the deprecation date. The overlap is only available until then, and it is the single most useful thing the provider gave us.
Notes on using this format#
Aggregates hide the report. A net +17 concealed 27 regressions and two breaking improvements. Compare case by case or you are not testing, you are scoring.
Assert on shape, not only on correctness. Both breaking changes were improvements in accuracy. A suite that only measures whether the answer is right would have passed them.
Some behaviour works by accident. The handwritten annotations were never specified, never tested, and depended on. A version change is how you find these, and the only remedy is to add each one to the suite as it is discovered.
Use the overlap. A parallel run on live traffic is the only test performed on the real distribution, and it is available only until the deprecation date.