Worked Example — The Deprecation Notice Nobody Read
A worked example of a forced model migration with no test suite — what broke on cutover day, why the rollback did not exist, and what the team built in the fortnight afterwards.
This is an illustrative example. The company, dates and figures are invented. The situation is not: providers deprecate model versions, the notice arrives by email, and the email goes to whoever created the account.
The situation#
An insurance company used an AI feature to triage claims into fast-track, standard and investigate. It had run for fourteen months and was trusted.
The provider sent a deprecation notice for the model version in use, with 90 days and a clear migration path. It went to the email address on the account — a former employee's, forwarded to a shared inbox that four people could access and nobody read.
The model was retired on a Tuesday morning. The application began receiving errors at 09:04.
The first four hours#
| 09:04 | Errors begin. Claims triage returns failures. |
| 09:20 | Escalated. Nobody knows why. |
| 09:55 | Provider status page checked — clean. |
| 10:30 | Someone finds the deprecation notice in the shared inbox |
| 10:45 | Decision: point at the successor version |
| 11:10 | Deployed. Errors stop. |
| 11:10 – 16:00 | Claims triaged by the new model, unverified |
The switch itself was one configuration line. The problem was everything after 11:10.
What changed silently#
Three days later a claims manager noticed the investigate queue had roughly doubled. The review took a week and found four behavioural changes, none of which produced an error.
More conservative triage. Claims that had been fast-tracked were now standard, and standard were now investigate. Roughly 22% of claims moved a category. Not wrong, exactly — defensible on each individual claim, and materially different in aggregate.
Confidence values on a different scale. The old model's scores clustered 0.6–0.95. The new one's clustered 0.4–0.8. A downstream rule auto-approved anything above 0.85, and that rule stopped firing almost entirely. Nobody had documented that the threshold was calibrated to a specific model version.
Longer reason text. The reason field went from around 40 words to around 90. It is displayed in a fixed-height panel in the claims system, so reasons were being truncated mid-sentence for every claim.
Different handling of missing fields. The old model inferred a claim type from context when the field was absent. The new one returned "unknown", and 3% of claims routed to a manual queue that had been almost empty for a year.
The cost#
| Downtime | 2 hours 6 minutes |
| Claims triaged before the changes were understood | ~1,400 |
| Claims re-triaged manually | 1,400 |
| Auto-approval rate during the period | 31% → 4% |
| Backlog cleared in | 9 days |
| Staff time | ~180 hours |
The two hours of downtime were the cheapest part. Everything expensive came from three days of running a changed system while believing it was the same one.
Why there was no rollback#
The old version was retired. There was nothing to roll back to — the only path was forward.
This is the property that distinguishes a provider deprecation from an ordinary dependency upgrade, and it is why the 90 days matter so much. During the notice period both versions run and can be compared. After it, one of them does not exist. The team had a comparison window and spent it not knowing it was open.
What was built in the fortnight after#
A behavioural suite. 300 real claims with known correct triage, plus 80 cases asserting on shape: confidence range, reason length, field presence, handling of missing data. Two of the four silent changes — confidence scale and reason length — are shape assertions, and no accuracy-focused suite would have caught either.
Provider notices routed to a team address, monitored, with a standing calendar entry to check deprecation schedules monthly. The most embarrassing item on the list and the cheapest.
Model version pinned explicitly in configuration, with the pin recorded in the deployment log. Previously the application requested an alias that the provider re-pointed.
A calibration document stating which downstream thresholds depend on model output scales. There were three. Nobody had known.
A migration runbook: run the suite against the new version, diff case by case, quantify the deltas, adjust thresholds, then switch — with a parallel period wherever the notice period allows one.
The next deprecation#
It arrived seven months later, with 60 days.
| Forced migration | Planned migration | |
|---|---|---|
| Notice used | 0 days | 41 days |
| Suite run against the new version | No suite | Day 2 |
| Cases that changed outcome | Unknown | 34 of 300 |
| Shape assertions failed | Unknown | 2 (confidence range again) |
| Threshold adjustments needed | Discovered in production | 1, made before cutover |
| Parallel running | None | 2 weeks |
| Downtime | 2 h 6 min | 0 |
| Claims re-triaged | 1,400 | 0 |
The confidence range failed again on the next version, which is the strongest argument for that particular assertion. It is not a one-off property of one migration; it moves whenever the model does.
What was learned#
A deprecation notice is a comparison window. It is the only period in which both versions exist and can be run side by side, and it is finite.
Errors are the easy failures. The outage was two hours. The three days of silently different behaviour cost 180 hours.
Assert on shape, not only on accuracy. Confidence scale and reason length broke downstream systems while the triage decisions themselves remained defensible.
Know which thresholds are calibrated to a model. There were three, none documented, and one of them silently stopped an entire auto-approval path.
Route provider notices to a team, not a person. The whole incident began with an email nobody was responsible for reading.