Checklist · Regression Testing

AI Regression Testing Checklist

Catch degradation in systems whose output legitimately varies, suite structure, what triggers a run, pinning, and reading a result without being fooled by the aggregate.

About the numbers here. Numbers here are thresholds the checklist proposes, not measurements.

Download the checklist (Word) Read the full guide

Markdown. No sign-up, no email.

System: _______________ Change: _______________ Date: _______

1. Suite structure#

[ ] Golden set. Must always pass. Small, stable, high-consequence
[ ] Regression set. Every production failure, permanently
[ ] Exploratory set. Newer cases, allowed to fail while investigated
[ ] The three are kept separate, not merged
[ ] Promotion path defined: exploratory → golden once stable

2. Pinning: you cannot detect drift against a moving baseline#

[ ] Model version pinned. Not "latest"
[ ] Temperature and sampling fixed
[ ] Prompt versioned in the repository
[ ] Corpus snapshotted for the golden set
[ ] Scorer version fixed

3. What triggers a run#

[ ] Prompt change
[ ] Model version change. The most common cause of silent degradation
[ ] Retrieval or chunking change
[ ] Tool definition change
[ ] Document corpus change. The system changed even though the code did not
[ ] Scheduled run regardless of changes

4. What counts as a regression here#

[ ] Correctness dropped. A passing case now fails
[ ] Consistency dropped. 10/10 became 7/10. Same nominal result, different system
[ ] Cost rose at flat quality. Prompt grew, score did not move
[ ] Refusal behaviour changed. It now answers what it used to decline

5. Reading the result#

[ ] Per-case deltas reported, not only the aggregate
[ ] Aggregate treated as a dashboard number, not a verdict
[ ] Cases that broke identified individually
[ ] Distinguished: correctness failure vs consistency failure
[ ] Cause checked in order, prompt, model, corpus, tools
[ ] Asked whether the old behaviour was actually correct

A suite reporting "84%, was 84%" can be hiding three broken cases offset by three improved ones.

6. Hygiene#

[ ] Every production failure added the same day, not "when we get time"
[ ] Outputs archived, not just verdicts
[ ] Test case updates are deliberate and noted, not silent
[ ] Cases that have never failed and never will are pruned

7. Speed: a slow suite gets skipped exactly when it matters#

[ ] Golden set runs in minutes
[ ] Full suite runs nightly
[ ] Runs parallelised
[ ] Results cached on unchanged prompt + model + input
[ ] Fails fast on structural checks before paying a judge

8. CI#

[ ] Golden set blocks on failure
[ ] Someone is accountable for a red build
[ ] Alerting on consistency drop, not just pass/fail
[ ] Alerting on cost per completed task rising

Sign-off#

	Name	Date
Run by
Result	Pass / Fail / Accepted with exceptions

Back to Regression Testing