# AI Test Automation Checklist

**System:** _______________  **Date:** _______  **Owner:** _______________

## 1. Automate in this order

- [ ] **1. Structural checks** — parses, fields present, arithmetic, no personal data
- [ ] **2. Regression cases** — every production failure
- [ ] **3. Refusal and safety** — should-refuse, permissions, known injections. Binary, no judge needed
- [ ] **4. Rubric scoring** — last, because least reliable

Deliberately **not** automated:

- [ ] Exploratory testing
- [ ] Usability judgement
- [ ] Deciding what "good enough" means

## 2. Pipeline

- [ ] Fixed case set with metadata
- [ ] Runner executes cases × repeats
- [ ] **Outputs archived, not just verdicts** — needed to investigate weeks later
- [ ] Scorers separate from the runner
- [ ] **Per-case deltas reported**, not only an aggregate

## 3. Model-as-judge, made trustworthy

- [ ] Used only for **narrow, closed questions** ("is this claim supported: yes/no")
- [ ] **Not** used for broad quality ("rate this 1–10")
- [ ] Judge is a **different model** from the one under test
- [ ] Judge pinned and versioned
- [ ] **Calibrated against human scoring** on a sample
- [ ] Re-calibrated periodically
- [ ] Rubric max 4 dimensions
- [ ] Judge asked for a verdict **and a reason**

## 4. Speed

- [ ] Runs parallelised up to the rate limit
- [ ] Results cached on `prompt + model + input`
- [ ] Tiered: structural on every commit, full suite nightly
- [ ] Fails fast on structural before paying a judge
- [ ] Unnecessary cases pruned

## 5. CI wiring

| Stage | Contents | Blocking |
|---|---|---|
| Every commit | Structural + refusal + injection regression | [ ] Yes |
| Every merge | Golden set, 3 runs | [ ] Yes |
| Nightly | Full suite incl. rubric, 5 runs | [ ] Report only |
| **On model change** | Everything + security suite | [ ] Yes |

- [ ] The model-change trigger is actually wired up

## 6. Alerting

- [ ] Pass-rate drop on the golden set — blocks
- [ ] **Consistency drop** — 10/10 → 7/10 is a real regression
- [ ] **Cost per completed task rising** at flat quality
- [ ] **Refusal accuracy falling** — answers what it used to decline
- [ ] Latency p95

## 7. Ownership and economics

- [ ] QA owns the standard and thresholds
- [ ] Engineering owns keeping it green
- [ ] Someone is accountable for a red build
- [ ] Suite cost is a small percentage of production inference spend
- [ ] Suite is trusted enough that people act on a red result

## Sign-off

| | Name | Date |
|---|---|---|
| Built by | | |
| Owner | | |