Checklist · Chaos Engineering

AI Chaos Engineering Checklist

Run a controlled failure experiment against an AI system — hypothesis, blast radius, stop conditions, the failures worth injecting, and what "graceful" has to mean.

Download the checklist Read the full guide

Markdown. No sign-up, no email.

System: _______________ Experiment: _______________ Date: _______

Not "unplug things and see". A hypothesis test with a stop condition.

1. Design the experiment#

[ ] Steady state defined and measured first — what normal looks like, numerically
[ ] Hypothesis written before running it
[ ] Blast radius stated: environment, traffic, data
[ ] Stop condition stated — what ends the experiment immediately
[ ] Team present and available
[ ] Announced — an unannounced experiment that looks like an incident wastes everyone's time
[ ] Off-peak

Steady state:   ____________________________________
Hypothesis:     ____________________________________
Blast radius:   ____________________________________
Stop condition: ____________________________________

2. Provider failures to inject#

[ ] Complete outage — sustained errors from the model API
[ ] Slow, not down — 30s instead of 2s. Harder and more common
[ ] Rate limited — sustained 429s
[ ] Malformed responses — truncated JSON, empty content, unexpected schema
[ ] Partial degradation — responses arrive but are poor. No error to catch

3. Surrounding failures#

[ ] Retrieval unavailable — does it refuse, or answer ungrounded from training data?
[ ] Retrieval returns empty — not an error. Does it say "I don't know"?
[ ] Tool failures (agents) — timeout, error, wrong shape
[ ] Stale corpus — nothing errors, answers confidently wrong
[ ] Spend cap reached mid-task

4. "Graceful" verified — define before testing#

[ ] Honest — user told something is wrong, not shown a spinner forever
[ ] Bounded — one slow dependency does not stall unrelated features
[ ] Lossless — queued work survives, or is discarded with the user informed
[ ] Idempotent — a retry cannot duplicate an effect
[ ] Recoverable — resumes without intervention when the dependency returns
[ ] Visible — something alerted. Silent degradation lasts longest

5. The findings people usually get#

[ ] Timeouts longer than intended, threads piling up
[ ] Retry without backoff, extending the outage
[ ] No user-facing message at all
[ ] Non-idempotent operations retried, producing duplicates
[ ] Queued work lost on restart

6. Afterwards#

[ ] Result recorded against the hypothesis — matched or not
[ ] Surprises recorded. These are the value
[ ] Findings fed into runbooks and the incident plan
[ ] Fixes made, then experiment re-run to verify
[ ] Next experiment scheduled

Record#


Hypothesis held?	Yes / No / Partly
What surprised us
Actions, with owners and dates
Re-run scheduled for

Sign-off#

	Name	Date
Experiment lead
System owner

Back to Chaos Engineering