Debugging a Prompt — A Worked Example
A support-ticket classifier stuck at 71%, taken to 93% through four changes measured one at a time — with the change that made it worse and the one that mattered most.
This is an illustrative example. The scenario and figures are composed to show the method, not drawn from a named client engagement.
A support team classifies incoming tickets into eight categories so they route to the right queue. A prompt does the classification. It works about seven times in ten, which is not good enough to route on, and the team has been "improving the prompt" for three weeks without the number moving.
The starting position#
The original prompt, in full:
Classify this support ticket into one of these categories: Billing, Technical,
Account, Feature Request, Bug Report, Refund, Complaint, Other.
Ticket: {ticket}Measured: 71% correct on a set of 200 real tickets that had been hand-labelled by the support lead.
Establishing that test set took a morning and was the reason the next three weeks were productive rather than more of the same. Before it existed, "better" meant "the three examples I keep pasting in look right".
Where the errors actually were#
Not evenly distributed — which is the point of looking rather than guessing:
| Category | Accuracy | Share of errors |
|---|---|---|
| Billing | 94% | 4% |
| Technical | 88% | 9% |
| Account | 82% | 11% |
| Refund | 79% | 8% |
| Bug Report | 48% | 31% |
| Feature Request | 51% | 26% |
| Complaint | 77% | 7% |
| Other | 66% | 4% |
Two categories produced 57% of all errors, and they were being confused with each other. A ticket saying "the export button doesn't do anything, it should let me pick a date range" is genuinely both.
Three weeks of general prompt polishing could never have found that. Ten minutes with a confusion breakdown did.
Four changes, measured one at a time#
Change 1 — define the ambiguous boundary. Added: "If the ticket describes something that does not work as documented, it is a Bug Report. If it describes something that works as documented but the user wants it to behave differently, it is a Feature Request. If both, choose Bug Report."
71% → 84%.
Change 2 — add examples for the confused pair only. Four examples, two of each, chosen from tickets the model had got wrong.
84% → 89%.
Change 3 — force structured output. The API's structured output with an enum of the eight categories, rather than asking for a category name in prose. This removed a class of failure that was not a classification error at all: responses like "This appears to be a Bug Report" that the parser did not match.
89% → 93%.
Change 4 — chain-of-thought. Added "Think step by step before answering."
93% → 91%. Reverted.
That last one is worth dwelling on. It is standard advice, it was applied in good faith, and on this task it made things worse — the model reasoned itself into over-thinking short, unambiguous tickets. Without a test set it would have been kept, because it feels like an improvement.
What did not help#
- Making the prompt more polite
- Adding "You are an expert support analyst" — no measurable change
- Adding "This is very important" — no measurable change
- Listing all eight categories with long descriptions — 86%, worse than the short boundary rule, because it buried the one distinction that mattered
The final state#
93%, with the remaining errors concentrated in genuinely ambiguous tickets where the support lead herself was inconsistent on relabelling. That is the ceiling for this task, and knowing that stopped further effort being spent.
Two operational additions:
- A confidence threshold below which the ticket goes to a triage queue rather than a category
- Every misrouted ticket reported by the team is added to the test set permanently
What transfers#
Build the test set first. Everything else here depended on it. Three weeks of unmeasured iteration produced nothing; one morning of labelling made the next changes obvious.
Look at where the errors are before changing anything. Two categories held 57% of the errors. General improvements would have moved the wrong number.
Change one thing at a time and measure. Otherwise you cannot tell which change helped, and you will keep the one that hurt.
Be willing to revert standard advice. Chain-of-thought is good guidance and it was wrong here. Test it on your own case rather than adopting it as a rule.
See prompt engineering, the production checklist, and the prompt template for the structure this ended up in.