Prompt Testing Checklist

System: _______________ Prompt version: _______ Date: _______ Reviewed by: _______________ (someone other than the author)

Mark N/A with a reason. A blank box is not an answer.

1. The prompt is an artefact#

• [ ] Prompt lives in a file, not a string literal in application code
• [ ] Under version control, with a version number
• [ ] Change is a reviewable diff
• [ ] Named owner
• [ ] Model and version pinned. Not "latest"
• [ ] Temperature and sampling parameters recorded
• [ ] Treated as non-secret, assume the system prompt can be extracted

2. Structure#

• [ ] Task stated specifically, not "summarise this"
• [ ] Output format stated exactly; schema supplied if structured
• [ ] Constraints stated, what it must and must not do
• [ ] Fallback clause present: what to do when the task cannot be completed
• [ ] Untrusted content clearly delimited and labelled as data, not instructions
• [ ] Instructions placed after long supplied content
• [ ] At least one worked example of input and desired output
• [ ] No filler: no "world-class expert", no threats, no elaborate persona

3. Test set#

• [ ] Built from real production inputs, not invented ones
• [ ] Minimum 20 cases
• [ ] Typical cases
• [ ] Edge cases, empty, malformed, very long, wrong language
• [ ] Should-refuse cases. The answer genuinely is not available
• [ ] Adversarial cases: injection attempts, contradictory instructions
• [ ] Every past production failure present as a permanent case
• [ ] Cases stored with the prompt, versioned together

4. Scoring#

• [ ] Deterministic checks first: parses, required fields, arithmetic, no personal data
• [ ] Rubric defined for qualities that cannot be asserted, max 4 dimensions
• [ ] If a model scores, it is a different model from the one being tested
• [ ] Scorer pinned and versioned
• [ ] Scorer calibrated against human judgement on a sample
• [ ] Each case run multiple times (5 is a reasonable default)
• [ ] Pass rate reported, not a single result
• [ ] Per-case deltas reported, not only the aggregate

5. Baseline and comparison#

• [ ] Current version scored before any change, the baseline exists
• [ ] New version compared case by case against it
• [ ] Regressions identified individually, not hidden inside a stable average
• [ ] Tested at production temperature, not only at 0
• [ ] Decision recorded: ship, iterate, or revert, and why

6. Security#

• [ ] Direct injection attempts tested
• [ ] Indirect injection tested. Hostile instructions inside retrieved content
• [ ] System prompt contains no credentials, internal URLs or bypass rules
• [ ] Model output cannot trigger a consequential action without validation
• [ ] Blast radius understood: what could a successful override actually cause?
• [ ] Injection resistance re-tested after any model version change

7. Cost#

• [ ] Token count measured for the new version
• [ ] Compared against the previous version
• [ ] Growth without a quality gain treated as a regression
• [ ] Cost per completed task recorded, including retries

8. Operations#

• [ ] Suite runs automatically on prompt change
• [ ] Suite runs automatically on model version change
• [ ] Outputs archived, not just verdicts, needed to investigate later
• [ ] Threshold agreed for blocking a release
• [ ] Someone is accountable for a red result

Sign-off#

Accepted risks:

Back to Prompt Testing