Checklist · Prompt Testing

Prompt Testing Checklist

Run through before a prompt change reaches production — versioning, test coverage, scoring, security and cost. Free, editable Markdown.

Download the checklist Read the full guide

Markdown. No sign-up, no email.

System: _______________ Prompt version: _______ Date: _______ Reviewed by: _______________ (someone other than the author)

Mark N/A with a reason. A blank box is not an answer.

1. The prompt is an artefact#

[ ] Prompt lives in a file, not a string literal in application code
[ ] Under version control, with a version number
[ ] Change is a reviewable diff
[ ] Named owner
[ ] Model and version pinned — not "latest"
[ ] Temperature and sampling parameters recorded
[ ] Treated as non-secret — assume the system prompt can be extracted

2. Structure#

[ ] Task stated specifically, not "summarise this"
[ ] Output format stated exactly; schema supplied if structured
[ ] Constraints stated — what it must and must not do
[ ] Fallback clause present: what to do when the task cannot be completed
[ ] Untrusted content clearly delimited and labelled as data, not instructions
[ ] Instructions placed after long supplied content
[ ] At least one worked example of input and desired output
[ ] No filler — no "world-class expert", no threats, no elaborate persona

3. Test set#

[ ] Built from real production inputs, not invented ones
[ ] Minimum 20 cases
[ ] Typical cases
[ ] Edge cases — empty, malformed, very long, wrong language
[ ] Should-refuse cases — the answer genuinely is not available
[ ] Adversarial cases — injection attempts, contradictory instructions
[ ] Every past production failure present as a permanent case
[ ] Cases stored with the prompt, versioned together

4. Scoring#

[ ] Deterministic checks first: parses, required fields, arithmetic, no personal data
[ ] Rubric defined for qualities that cannot be asserted — max 4 dimensions
[ ] If a model scores, it is a different model from the one being tested
[ ] Scorer pinned and versioned
[ ] Scorer calibrated against human judgement on a sample
[ ] Each case run multiple times (5 is a reasonable default)
[ ] Pass rate reported, not a single result
[ ] Per-case deltas reported, not only the aggregate

5. Baseline and comparison#

[ ] Current version scored before any change — the baseline exists
[ ] New version compared case by case against it
[ ] Regressions identified individually, not hidden inside a stable average
[ ] Tested at production temperature, not only at 0
[ ] Decision recorded: ship, iterate, or revert — and why

6. Security#

[ ] Direct injection attempts tested
[ ] Indirect injection tested — hostile instructions inside retrieved content
[ ] System prompt contains no credentials, internal URLs or bypass rules
[ ] Model output cannot trigger a consequential action without validation
[ ] Blast radius understood: what could a successful override actually cause?
[ ] Injection resistance re-tested after any model version change

7. Cost#

[ ] Token count measured for the new version
[ ] Compared against the previous version
[ ] Growth without a quality gain treated as a regression
[ ] Cost per completed task recorded, including retries

8. Operations#

[ ] Suite runs automatically on prompt change
[ ] Suite runs automatically on model version change
[ ] Outputs archived, not just verdicts — needed to investigate later
[ ] Threshold agreed for blocking a release
[ ] Someone is accountable for a red result

Sign-off#

	Name	Date
Author
Reviewer

Accepted risks:

#	Item not met	Why accepted	Revisit by

Back to Prompt Testing