Worked Example — The One-Word Change Nobody Reviewed
A worked example of an untracked prompt edit that ran for three weeks — how it was found, what it cost, and the three controls that made prompts as reviewable as code.
This is an illustrative example. The company, timeline and figures are invented. The underlying condition — prompts edited in a console, outside version control, by whoever has the login — is extremely common.
The situation#
A subscription business used an AI feature to draft responses to cancellation requests. A support manager, entirely reasonably, wanted the drafts to try harder to retain the customer.
They opened the provider console and changed one line of the system prompt:
"Acknowledge the request and explain the options available."
became
"Acknowledge the request and explain the options available, including any retention offer the customer may be eligible for."
It took two minutes. There was no pull request, because prompts did not live in the repository. There was no evaluation, because none existed for this feature. Nobody else knew.
What happened over the next three weeks#
The drafts did mention retention offers, which is what was wanted.
They also mentioned offers the customer was not eligible for. The model had no eligibility data — the prompt asked for offers "the customer may be eligible for", and with nothing to check against, it produced plausible ones. Discounts that did not exist. Free months that were not on the price list. A loyalty tier the company had retired in 2024.
Support agents approved and sent the drafts. From an agent's point of view they looked entirely normal, and the offers looked like something the system knew about and they did not.
How it was found#
Not by monitoring. By finance.
The retention team's month-end reconciliation showed 41 accounts with discounts that did not match any campaign. Tracing them back through the ticket history took two days and led to the drafts.
Three weeks between the change and the discovery, and the discovery was a side effect of somebody counting money.
The cost#
| Accounts offered a discount that did not exist | 41 |
| Honoured, because the offer had been made in writing | 41 |
| Direct cost over the contract term | ~£12,400 |
| Staff time to trace, contact and resolve | ~60 hours |
| Accounts that cancelled after the offer was corrected | 6 |
The company honoured every offer, which was the right decision and was never in question. The cost was the consequence of the change, not of the response to it.
Why nothing caught it#
Four things, each of which is a control that did not exist.
The prompt was not in version control. There was no diff, no author, no date and no review. Reconstructing the change required the provider's console audit log, which retains 30 days — the change was 22 days old when they looked.
There was no evaluation set. Nothing existed to run the new prompt against, so there was nothing to run.
Nothing detected the change. The application read the prompt from the provider at startup. No alert, no log entry, no deployment record.
The drafts were reviewed by people who could not verify them. An agent reading "you may be eligible for a 20% loyalty discount" has no way to know whether that is true. The review step existed and could not catch this class of error.
The three controls#
Prompts moved into the repository. They are now files, deployed with the application. The provider console is read-only for everyone; the credential that could edit it was rotated and the permission removed.
This was the change with the most resistance — it made a two-minute edit into a pull request and a deploy. The counter-argument that carried it: a two-minute edit that runs for three weeks unreviewed is not agility, it is an absence of review.
An evaluation set was built. Ninety cases drawn from real cancellation tickets, with expected behaviours including negative ones: must not state a specific discount percentage, must not name an offer, must not commit to anything requiring eligibility data.
Run against the changed prompt afterwards, it failed 34 of 90. It would have caught this in four minutes.
Factual commitments moved out of the model. Retention offers are now looked up from the offers system and passed in as data. If the customer is eligible for nothing, the prompt receives nothing and the draft cannot invent one. The model no longer has the opportunity to be wrong about eligibility, which is stronger than instructing it not to be.
The result#
| Before | After | |
|---|---|---|
| Prompt changes reviewed | No | Every one |
| Time to detect an unreviewed change | 3 weeks | Not possible |
| Evaluation before a prompt ships | None | 90 cases, 4 minutes |
| Offers stated that do not exist | 41 in 3 weeks | 0 in 5 months |
The support manager's original request was, incidentally, a good one. It shipped six weeks later with eligibility data wired in, and retention on cancellation requests improved.
What was learned#
A prompt is production code. It has no compiler, no type system and no reviewer by default, which makes it more dangerous to edit casually than the code around it, not less.
Instructions cannot supply facts. Asking a model to mention offers the customer is eligible for, without giving it eligibility data, is asking it to invent them. Pass the data or remove the instruction.
Console access is production access. Anyone who can edit the prompt can change the behaviour of the system without leaving a trace outside a 30-day log.
Negative cases are where the value is. The 34 failures were nearly all negative assertions — things the draft must not say. A set built only from good examples would have passed the broken prompt.