Checklist · Test Automation

AI Test Automation Checklist

Build an automated evaluation pipeline — what to automate in what order, making model-as-judge trustworthy, keeping the suite fast, and what belongs in CI.

Download the checklist Read the full guide

Markdown. No sign-up, no email.

System: _______________ Date: _______ Owner: _______________

1. Automate in this order#

[ ] 1. Structural checks — parses, fields present, arithmetic, no personal data
[ ] 2. Regression cases — every production failure
[ ] 3. Refusal and safety — should-refuse, permissions, known injections. Binary, no judge needed
[ ] 4. Rubric scoring — last, because least reliable

Deliberately not automated:

[ ] Exploratory testing
[ ] Usability judgement
[ ] Deciding what "good enough" means

2. Pipeline#

[ ] Fixed case set with metadata
[ ] Runner executes cases × repeats
[ ] Outputs archived, not just verdicts — needed to investigate weeks later
[ ] Scorers separate from the runner
[ ] Per-case deltas reported, not only an aggregate

3. Model-as-judge, made trustworthy#

[ ] Used only for narrow, closed questions ("is this claim supported: yes/no")
[ ] Not used for broad quality ("rate this 1–10")
[ ] Judge is a different model from the one under test
[ ] Judge pinned and versioned
[ ] Calibrated against human scoring on a sample
[ ] Re-calibrated periodically
[ ] Rubric max 4 dimensions
[ ] Judge asked for a verdict and a reason

4. Speed#

[ ] Runs parallelised up to the rate limit
[ ] Results cached on prompt + model + input
[ ] Tiered: structural on every commit, full suite nightly
[ ] Fails fast on structural before paying a judge
[ ] Unnecessary cases pruned

5. CI wiring#

Stage	Contents	Blocking
Every commit	Structural + refusal + injection regression	[ ] Yes
Every merge	Golden set, 3 runs	[ ] Yes
Nightly	Full suite incl. rubric, 5 runs	[ ] Report only
On model change	Everything + security suite	[ ] Yes

[ ] The model-change trigger is actually wired up

6. Alerting#

[ ] Pass-rate drop on the golden set — blocks
[ ] Consistency drop — 10/10 → 7/10 is a real regression
[ ] Cost per completed task rising at flat quality
[ ] Refusal accuracy falling — answers what it used to decline
[ ] Latency p95

7. Ownership and economics#

[ ] QA owns the standard and thresholds
[ ] Engineering owns keeping it green
[ ] Someone is accountable for a red build
[ ] Suite cost is a small percentage of production inference spend
[ ] Suite is trusted enough that people act on a red result

Sign-off#

	Name	Date
Built by
Owner

Back to Test Automation