Worked Example — The Fallback That Caused the Incident

This is an illustrative example. The company, outage and figures are invented. The pattern — recovery logic written optimistically, never exercised, and worse than the failure it handles — is what chaos experiments exist to find.

The situation#

A pricing platform used an AI feature to generate quotes for complex insurance products. It had a fallback: if the model provider was unavailable, fall back to a rules engine written three years earlier.

The fallback had been added at launch, reviewed, approved, documented, and never tested against a real provider failure.

The outage#

The provider had a 90-minute degradation. Not an outage — requests succeeded, slowly, some timing out.

The fallback triggered on timeout. The rules engine produced quotes. Nothing alerted, because from the platform's point of view nothing failed. Quotes were being generated throughout.

What the fallback actually did#

The rules engine had not been updated in three years. It had been kept as a safety net and excluded from the release process that maintained the pricing tables, on the reasonable-sounding grounds that it was not in the live path.

The fallback worked exactly as designed. It generated a quote for every request. It was never told that its pricing was three years stale, because nothing in the system knew that was a property worth checking.

The 147 bound policies were honoured. That was never in question and it is the cost of the gap.

Why nothing alerted#

Three assumptions, each reasonable on its own.

The fallback was treated as success. A quote was returned, so no error was recorded and no alert fired. Monitoring measured whether quotes were produced, not which system produced them.

Nothing measured fallback rate. Had anyone been able to see 1,890 quotes from a path that normally serves none, the window would have closed in minutes rather than 90.

Nobody owned the fallback. Excluded from releases, absent from the runbook, mentioned in one design document. It had no owner because it was never expected to run.

The first game day#

Six weeks later, the team ran their first chaos exercise. Four experiments, production, 5% of traffic, 90 minutes, with abort criteria agreed in advance.

Experiment 1 — provider returns errors. Hypothesis: fallback engages, quotes are marked as fallback-priced, alert fires within a minute.

Result: fallback engaged. No marking — that had been built but only applied to the customer receipt, not the internal record. Alert fired at 4 minutes, not 1. Partial pass, and the gap between the two is what the exercise was for.

Experiment 2 — provider slow, not failing. Hypothesis: same as experiment 1.

Result: fallback did not engage. The timeout was set above the degraded latency, so requests waited 40 seconds and returned real quotes very slowly. Customers abandoned. This is the original incident's near-miss twin, and it fails in the opposite direction: the fallback that fired when it should not have, in the incident, would not fire when it should here.

Experiment 3 — pricing table service unavailable. Hypothesis: clear error, no quote.

Result: quotes generated from a cache with no age limit. The cache was 6 days old. Fail — the same class of defect as the original incident, in a different component, found before it cost anything.

Experiment 4 — both providers unavailable. Hypothesis: graceful refusal.

Result: graceful refusal, correct message, quote request queued for callback. Pass.

What changed#

Fallback output is marked everywhere — internal records, monitoring, the quote itself. A quote generated by the fallback is visibly a different thing.

Fallback rate is a monitored metric with an alert at any sustained non-zero value. It is normally zero, which makes it the cleanest signal in the system.

The fallback is in the release process. Pricing tables update in both engines or in neither. This is the change that would have prevented the incident entirely, and it had been excluded on the grounds that the fallback was not live.

Timeouts were set from measured latency distributions, not from a round number, so slow and failing are both handled.

Every cache has a maximum age, beyond which it is not served.

Game days run monthly. Six subsequent exercises have found four further problems, one of which was another optimistic fallback added by a different team in the interim.

What was learned#

Untested recovery logic is a liability, not a safety net. It runs at the worst moment, having never been exercised, and it runs with confidence.

A fallback must announce itself. Silent degradation is what turned 90 minutes into 147 mispriced policies. Had a single dashboard shown quotes coming from the wrong engine, this is a ten-minute incident.

Anything excluded from the release process will be stale when it runs. The exclusion was the whole defect, and it was justified by the belief that the fallback would never be needed.

Test the degraded case, not just the failed one. Experiment 2 found a system that handles total failure and not partial failure, which is the more common event by a wide margin.

Back to Chaos Engineering