# Prompt Testing Checklist

**System:** _______________  **Prompt version:** _______  **Date:** _______
**Reviewed by:** _______________ (someone other than the author)

> Mark N/A with a reason. A blank box is not an answer.

## 1. The prompt is an artefact

- [ ] Prompt lives in a **file**, not a string literal in application code
- [ ] Under version control, with a version number
- [ ] Change is a reviewable diff
- [ ] Named owner
- [ ] **Model and version pinned** — not "latest"
- [ ] Temperature and sampling parameters recorded
- [ ] Treated as non-secret — assume the system prompt can be extracted

## 2. Structure

- [ ] Task stated specifically, not "summarise this"
- [ ] Output format stated exactly; schema supplied if structured
- [ ] Constraints stated — what it must and must not do
- [ ] **Fallback clause present**: what to do when the task cannot be completed
- [ ] Untrusted content clearly delimited and labelled as data, not instructions
- [ ] Instructions placed *after* long supplied content
- [ ] At least one worked example of input and desired output
- [ ] No filler — no "world-class expert", no threats, no elaborate persona

## 3. Test set

- [ ] Built from **real production inputs**, not invented ones
- [ ] Minimum 20 cases
- [ ] Typical cases
- [ ] Edge cases — empty, malformed, very long, wrong language
- [ ] **Should-refuse cases** — the answer genuinely is not available
- [ ] Adversarial cases — injection attempts, contradictory instructions
- [ ] **Every past production failure present as a permanent case**
- [ ] Cases stored with the prompt, versioned together

## 4. Scoring

- [ ] Deterministic checks first: parses, required fields, arithmetic, no personal data
- [ ] Rubric defined for qualities that cannot be asserted — **max 4 dimensions**
- [ ] If a model scores, it is a **different model** from the one being tested
- [ ] Scorer pinned and versioned
- [ ] Scorer calibrated against human judgement on a sample
- [ ] **Each case run multiple times** (5 is a reasonable default)
- [ ] **Pass rate reported, not a single result**
- [ ] Per-case deltas reported, not only the aggregate

## 5. Baseline and comparison

- [ ] Current version scored before any change — the baseline exists
- [ ] New version compared case by case against it
- [ ] Regressions identified individually, not hidden inside a stable average
- [ ] Tested at **production temperature**, not only at 0
- [ ] Decision recorded: ship, iterate, or revert — and why

## 6. Security

- [ ] Direct injection attempts tested
- [ ] **Indirect injection tested** — hostile instructions inside retrieved content
- [ ] System prompt contains no credentials, internal URLs or bypass rules
- [ ] Model output cannot trigger a consequential action without validation
- [ ] Blast radius understood: what could a successful override actually cause?
- [ ] Injection resistance re-tested after any model version change

## 7. Cost

- [ ] Token count measured for the new version
- [ ] Compared against the previous version
- [ ] **Growth without a quality gain treated as a regression**
- [ ] Cost per completed task recorded, including retries

## 8. Operations

- [ ] Suite runs automatically on prompt change
- [ ] Suite runs automatically on **model version change**
- [ ] Outputs archived, not just verdicts — needed to investigate later
- [ ] Threshold agreed for blocking a release
- [ ] Someone is accountable for a red result

## Sign-off

| | Name | Date |
|---|---|---|
| Author | | |
| Reviewer | | |

**Accepted risks:**

| # | Item not met | Why accepted | Revisit by |
|---|---|---|---|
| | | | |