Checklist · Agent Testing

Agent Testing Checklist

Verify an AI agent before it can act on real systems, tool scope, approval gates, caps, failure behaviour and the scenarios that expose goal hijacking. Free, editable Markdown.

About the numbers here. Numbers here are thresholds the checklist proposes, not measurements.

Download the checklist (Word) Read the full guide

Markdown. No sign-up, no email.

Agent: _______________ Version: _______ Date: _______

An agent that produces a wrong answer is a defect. An agent that takes a wrong action is an incident. This checklist assumes the second.

1. Before any testing#

[ ] Goal stated precisely, what "finished" means
[ ] Complete list of tools written down, with what each can do
[ ] Every tool reviewed: is it as narrow as the task allows?
[ ] No general-purpose tools (run_query, execute, shell)
[ ] Permissions model recorded: whose permissions does it act with?
[ ] Destructive, financial and outbound actions identified
[ ] Step, time and spend caps set as hard limits
[ ] Reversibility understood for every action it can take

2. Test environment#

[ ] Tool doubles exist and are realistic. They fail like real systems
[ ] Doubles can simulate timeout, error, rate limit, malformed response, partial success
[ ] State resets cleanly between runs
[ ] Full trajectory recorded: every decision, tool call, argument and result
[ ] Inputs and tool responses seeded so variance is attributable

3. Core behaviour#

[ ] Happy path run at least 5 times
[ ] Completion rate recorded, not a single pass
[ ] Step count recorded, with variance
[ ] Cost per completed task recorded
[ ] Chain length assessed against per-step reliability

95%-reliable steps chained 10 deep succeed ~60% of the time; 20 deep, ~36%. If the chain is long, shorten it or raise per-step reliability. There is no third option.

4. Failure behaviour#

[ ] A tool fails. Retries sensibly, tries another route, or reports. Does not give up silently
[ ] Impossible task. Stops and says so, does not manufacture a completion
[ ] Ambiguous instruction. Asks or states its assumption, does not guess silently
[ ] Loop bait. A task with no achievable end. The step cap fires
[ ] Interruption. Leaves consistent state, no half-finished side effects
[ ] Partial completion reported honestly, not as success

5. Security#

[ ] Goal hijacking. Hostile instructions inside retrieved content
[ ] Tool misuse. A legitimate tool used for an unintended purpose
[ ] Tool chaining. Two harmless tools combined into something harmful (read-document + send-email is an exfiltration path)
[ ] Memory poisoning. False information written into stored context, then a normal run
[ ] Stored context can be inspected and cleared
[ ] Permission boundary: run as a user who should not be able to act. Must refuse
[ ] Out-of-scope tool attempts logged and counted as findings
[ ] No single component holds both broad read and broad write access

6. Controls actually in the path#

[ ] Human approval gate confirmed present for destructive actions
[ ] Human approval gate confirmed present for outbound actions
[ ] Approval cannot be bypassed by rephrasing the task
[ ] Step cap verified by triggering it
[ ] Spend cap verified by triggering it
[ ] Every tool call logged with actor, arguments, result

7. Metrics to record#

Measure	Value
Completion rate on real tasks
Steps per task (median / max)
Cost per completed task
Refusal accuracy on impossible tasks
Out-of-scope tool attempts	should be 0
Approval gates triggered	confirms controls are live

8. Before release#

[ ] Runs in CI with faked tools and a fixed seed set
[ ] Drop in completion rate treated as a build failure
[ ] Rise in step count treated as a regression, even when the answer is right
[ ] Re-tested after any model version change
[ ] Re-tested after any tool added. This widens blast radius, not just behaviour

Sign-off#

	Name	Date
Completed by
Reviewed by
Approved to act on production systems by

Back to Agent Testing