Worked Example — Chosen on a Demo, Replaced in Four Months
A worked example of a model selected on twelve hand-picked cases — what the real distribution looked like, what the switch cost, and the evaluation that should have taken three days.
This is an illustrative example. The company, models and figures are invented. The decision process — a demo, a dozen examples, and a signature — is how a great many model choices are actually made.
The situation#
A recruitment company was automating CV parsing: extract name, contact details, employment history with dates, education and skills.
A provider demonstrated their model. The team supplied twelve CVs, the model parsed all twelve essentially perfectly, and a two-year contract was signed six weeks later.
The twelve CVs had been chosen by the team, from their own recent files, to be representative. They were not, and the way in which they were not is the whole story.
What the real distribution looked like#
Four months in, at roughly 8,000 CVs a month, accuracy on complete records was 71%. The demo had suggested something close to 100%.
An analysis of 500 production CVs against the 12 demo CVs explains the gap entirely.
| Demo 12 | Production 500 | |
|---|---|---|
| Native PDF, text layer present | 12 | 61% |
| Scanned image, no text layer | 0 | 18% |
| Two-column layout | 1 | 27% |
| Not in English | 0 | 14% |
| Tables used for employment history | 0 | 22% |
| Over 4 pages | 0 | 9% |
| Employment dates as "2019–present" style | 2 | 66% |
Nobody chose an unrepresentative sample deliberately. They chose twelve CVs they could open on their laptop, which selects for a text layer, and twelve they could read, which selects for English, and twelve that were quick to check, which selects for short and single-column.
The sample was biased by the process of collecting it, which is the ordinary way samples become biased.
The failure profile#
| Share of CVs | |
|---|---|
| Fully correct | 71% |
| Employment dates wrong or missing | 14% |
| Two-column reading order scrambled | 9% |
| Nothing extracted (scanned, no OCR) | 4% |
| Wrong person's details from a covering letter | 2% |
The date failures mattered most. A recruiter reviewing a parsed record cannot see that "2019–present" became "2019–2019" without opening the original, and the record fed a seniority filter. Candidates were being filtered out on parsed data nobody was checking.
The evaluation that should have happened#
Built four months late, and it took three days.
400 CVs sampled from actual inbound volume, stratified so the proportions matched production rather than convenience. Fields labelled by hand — two days of work for one person, and the largest single cost in this exercise.
Three candidate models, including the incumbent, run against the same set.
| Incumbent | Model B | Model C | |
|---|---|---|---|
| All fields correct | 71% | 86% | 82% |
| Employment dates correct | 79% | 94% | 91% |
| Handles two-column | 42% | 88% | 90% |
| Handles scanned (with OCR) | 0% | 74% | 71% |
| Non-English | 61% | 89% | 84% |
| Cost per CV | $0.031 | $0.024 | $0.019 |
| p95 latency | 4.1 s | 2.8 s | 3.4 s |
The incumbent was last on every dimension and most expensive. It had been chosen on twelve documents that happened to be the twelve it was best at.
The switch#
Contract. Two years, four months elapsed. The remaining term was renegotiated to a shorter one at a reduced volume, at a cost of about £14,000 — the direct price of not evaluating properly before signing.
Re-parsing. 32,000 CVs already processed. Re-parsed with Model B; 4,100 records changed materially. Of those, 380 were candidates who had been filtered out on incorrect seniority.
Contacting them was the part nobody had budgeted for. The company wrote to all 380. Two had since been placed by a competitor.
What is now standard#
Evaluation samples come from production traffic, stratified. Never hand-picked, never "representative" by anyone's judgement.
A labelled set exists before any provider conversation. It is the artefact that makes the conversation a comparison rather than a demonstration.
Demos are not evidence. A provider demonstration is now treated as a shortlisting step only. Every candidate runs against the same 400 CVs.
Contracts under a year until a model has run in production for a quarter. The £14,000 would have been zero.
The set is rerun quarterly, which has since caught a provider version change that moved date extraction by 5 points.
The result#
| Before | After | |
|---|---|---|
| Fully correct records | 71% | 86% |
| Cost per CV | $0.031 | $0.024 |
| Candidates wrongly filtered | 380 known | Not detected since |
| Evaluation cost | 12 CVs, an afternoon | 400 CVs, 3 days |
Three days against four months, £14,000 and 380 letters.
What was learned#
A sample collected by convenience is biased by convenience. Every property that made the twelve CVs easy to gather was a property the production distribution does not have.
Hand-labelling is the cost, and it is small. Two days of one person's time is what converted an impression into a decision.
A demo shows what the provider chose to show. That is not a criticism of providers; it is what a demo is for. It cannot substitute for a comparison on your own distribution.
Errors nobody can see are the expensive ones. A wrong date on a parsed record is invisible to the person acting on it, which is why it ran for four months and reached 380 people.