AI Regression Testing Checklist

System: _______________ Change: _______________ Date: _______

1. Suite structure#

• [ ] Golden set — must always pass. Small, stable, high-consequence
• [ ] Regression set — every production failure, permanently
• [ ] Exploratory set — newer cases, allowed to fail while investigated
• [ ] The three are kept separate, not merged
• [ ] Promotion path defined: exploratory → golden once stable

2. Pinning — you cannot detect drift against a moving baseline#

• [ ] Model version pinned — not "latest"
• [ ] Temperature and sampling fixed
• [ ] Prompt versioned in the repository
• [ ] Corpus snapshotted for the golden set
• [ ] Scorer version fixed

3. What triggers a run#

• [ ] Prompt change
• [ ] Model version change — the most common cause of silent degradation
• [ ] Retrieval or chunking change
• [ ] Tool definition change
• [ ] Document corpus change — the system changed even though the code did not
• [ ] Scheduled run regardless of changes

4. What counts as a regression here#

• [ ] Correctness dropped — a passing case now fails
• [ ] Consistency dropped — 10/10 became 7/10. Same nominal result, different system
• [ ] Cost rose at flat quality — prompt grew, score did not move
• [ ] Refusal behaviour changed — it now answers what it used to decline

5. Reading the result#

• [ ] Per-case deltas reported, not only the aggregate
• [ ] Aggregate treated as a dashboard number, not a verdict
• [ ] Cases that broke identified individually
• [ ] Distinguished: correctness failure vs consistency failure
• [ ] Cause checked in order — prompt, model, corpus, tools
• [ ] Asked whether the old behaviour was actually correct

🔴 A suite reporting "84%, was 84%" can be hiding three broken cases offset by three improved ones.

6. Hygiene#

• [ ] Every production failure added the same day, not "when we get time"
• [ ] Outputs archived, not just verdicts
• [ ] Test case updates are deliberate and noted, not silent
• [ ] Cases that have never failed and never will are pruned

7. Speed — a slow suite gets skipped exactly when it matters#

• [ ] Golden set runs in minutes
• [ ] Full suite runs nightly
• [ ] Runs parallelised
• [ ] Results cached on unchanged prompt + model + input
• [ ] Fails fast on structural checks before paying a judge

8. CI#

• [ ] Golden set blocks on failure
• [ ] Someone is accountable for a red build
• [ ] Alerting on consistency drop, not just pass/fail
• [ ] Alerting on cost per completed task rising

Sign-off#

Back to Regression Testing