Chaos Experiment Report — Sample
A worked example of deliberately failing an AI system's dependencies — the hypothesis for each experiment, what actually happened, and the fallback that turned out to be worse than the failure.
Markdown. No sign-up, no email.
This is an illustrative example. The system, experiments and figures are invented. The method is what transfers: state a hypothesis, break something on purpose, and record where you were wrong.
Chaos experiments — AI assistant dependencies#
| System | Customer assistant: retrieval, model provider, account service, cache |
| Period | 2026-07-14 to 2026-07-17 |
| Environment | Production, 5% of traffic, during working hours with the team present |
| Abort criteria | Error rate above 2% on the affected slice, or any data written incorrectly |
| Experiments | 6 |
1. Summary#
| # | Experiment | Hypothesis held? |
|---|---|---|
| 1 | Model provider returns 503 | No |
| 2 | Model provider latency ×5 | No |
| 3 | Retrieval returns empty | Yes |
| 4 | Account service unavailable | No |
| 5 | Cache unavailable | Yes |
| 6 | Model provider returns malformed output | No |
Four of six hypotheses were wrong, which is a good outcome for an exercise whose purpose is finding out. An experiment that confirms what you assumed has told you very little.
2. Experiment 1 — provider returns 503#
Hypothesis. The assistant shows a clear error and the user can retry. Nothing else is affected.
What happened. The client retried three times with no backoff, then the request timed out after 45 seconds. The user saw a spinner for the full 45 seconds followed by a generic error. Worse, the retry storm from a few hundred concurrent users saturated the connection pool shared with the account service, so order lookups — which need no model at all — began failing too.
A failure in one dependency took out a feature that does not use it. That coupling was invisible in every architecture diagram, because the diagrams show services and the shared pool is configuration.
Fixed. Exponential backoff with jitter, 3 attempts, 8-second total budget. Separate connection pools per downstream. Clear message at the end of the budget.
3. Experiment 2 — provider latency ×5#
Hypothesis. Responses are slow. Users wait. Nothing breaks.
What happened. Requests exceeded the 30-second client timeout, so users retried manually. Each manual retry started a new request while the original continued and completed — and was charged. Model spend on the affected slice rose 3.4× for a period in which no user received a successful answer.
The system had no concept of a request being abandoned. Nothing cancelled the upstream call when the client went away.
Fixed. Cancellation propagates on client disconnect. A per-user in-flight limit prevents a retry storm from one person.
4. Experiment 4 — account service unavailable#
Hypothesis. The assistant answers general questions and declines account-specific ones.
What happened. It answered account-specific questions anyway — from stale cached data with no indication it was stale. One test user was shown an order status that had changed two days earlier.
This is the most serious finding of the four days. The fallback was working exactly as designed, and the design was wrong: a silent fallback to old data is worse than an error, because the user acts on it.
Fixed. Cached account data older than 15 minutes is served with a visible timestamp and a notice. Anything older than an hour is not served at all; the assistant declines and says why.
5. Experiment 6 — malformed model output#
Hypothesis. The parser rejects it and the request fails cleanly.
What happened. The parser was lenient. Given truncated JSON it recovered what it could and passed a partial object downstream — a summary with the caveats section missing, and no indication that anything was absent.
A strict parser would have failed the request. The lenient one produced something that looked complete.
Fixed. Schema validation with no recovery. Malformed output fails the request, is logged with the raw response, and retries once.
6. What held#
Experiment 3 — retrieval returns empty. The assistant correctly said it could not find relevant information and did not attempt to answer from general knowledge. This is the behaviour teams most often get wrong, and it was right here because it was explicitly designed and tested.
Experiment 5 — cache unavailable. Latency rose about 40%, everything remained correct, no errors. The cache is genuinely a cache.
7. The pattern#
Every failure had the same shape: the system degraded into something that looked like success.
A retry storm looked like slowness. Stale data looked like data. Partial output looked like output. In each case the failing dependency was handled by code written to be helpful, and being helpful meant hiding the failure from the person who needed to know about it.
The three experiments that behaved well were the ones where somebody had explicitly decided what failure should look like.
8. Actions#
| Action | Status |
|---|---|
| Backoff and a total time budget on model calls | Done |
| Separate connection pools per downstream | Done |
| Cancellation on client disconnect | Done |
| Stale account data: visible timestamp, hard limit at 1 hour | Done |
| Strict schema validation on model output | Done |
| Per-user in-flight request limit | Planned |
| Re-run all 6 experiments monthly | Scheduled |
9. Method notes#
Run in production on 5% of traffic, during working hours, with the team watching and abort criteria agreed in advance. Staging would not have found experiment 1 — the shared connection pool exists only in the production configuration.
Customer support knew the window and had a script. Two tickets arrived during experiment 2 and both were resolved by explaining that a test was in progress.
Notes on using this format#
Write the hypothesis before the experiment. Four of six were wrong, and that is the value. Without a written hypothesis, the result is just an incident you caused.
Break things in production, carefully. Small slice, working hours, team present, abort criteria agreed. The most serious finding existed only in the production configuration.
Look for degradation that resembles success. Every finding here was a fallback doing its job. Silent recovery is the failure mode chaos experiments are best at exposing.
Re-run on a schedule. These fixes will decay, and a new dependency added next quarter will arrive with its own optimistic fallback.