Worked Example — Six Months Chasing the Leaderboard
A worked example of a team that switched models on every benchmark release — what each switch actually cost, and what happened when they built a set for their own task instead.
This is an illustrative example. The models are anonymised and the figures are invented. The behaviour — treating a public leaderboard as a decision procedure — is common wherever the leaderboard is the only number available.
The situation#
A team building a clinical documentation assistant had adopted a rule that sounded rigorous: use whichever model leads the relevant public benchmarks.
Over six months they switched models four times.
| Month | Switch | Reason |
|---|---|---|
| February | A → B | B took the lead on two benchmarks |
| April | B → C | C released, topped three |
| May | C → B | C had a rate limit that did not suit us |
| July | B → D | D topped the medical-domain benchmark |
Each switch was a week of prompt adjustment, a week of monitoring and a rewrite of whatever had been tuned for the previous model.
What the switches actually did#
They kept no consistent measurement, so the effect was invisible at the time. Reconstructed afterwards from user feedback and edit rates:
| Model | Benchmark rank | Clinician edit rate | Structure errors |
|---|---|---|---|
| A (original) | 4th | 31% | 4% |
| B | 1st | 38% | 11% |
| C | 1st | 34% | 6% |
| D | 1st | 44% | 17% |
The model that led every benchmark performed worst on the actual task. Edit rate — how often a clinician had to change the generated note — rose from 31% to 44% over six months of chasing improvements.
Structure errors are the more serious column. The assistant must produce notes in a fixed clinical structure. Model D writes better prose and follows a rigid template less reliably, and the benchmark that ranked it first rewards exactly the fluency that causes the problem.
What the benchmarks were measuring#
Reading the benchmark documentation — after six months of using them as a decision procedure — found three properties that mattered.
Free-form answers, scored by another model. Ours must fill a rigid template. Fluency helps there and hurts here.
Single-turn questions. Ours takes a 20-minute transcript with interruptions, corrections and multiple speakers.
General medical knowledge. Ours needs almost none — the information is in the transcript. It needs faithful structured extraction, which no benchmark in the set measures.
None of the three benchmarks measured anything the product does. They were not wrong. They were answering a different question, clearly, and the team had assumed the ranking transferred.
What they built instead#
200 real transcripts with clinician-written reference notes, gathered over three weeks with consent. The largest cost in this story and the only thing that ended it.
Four measures, each tied to something that matters:
| Measure | Why |
|---|---|
| Structure compliance | The note is rejected downstream if the template is broken |
| Fact recall against the transcript | Anything stated must be in the transcript |
| No fabricated clinical content | The failure that matters most |
| Clinician edit distance | The measure the users actually feel |
The result of measuring properly#
| A | B | C | D | |
|---|---|---|---|---|
| Structure compliance | 96% | 89% | 94% | 83% |
| Fact recall | 91% | 93% | 94% | 92% |
| Fabricated content | 0.4% | 1.1% | 0.7% | 1.9% |
| Edit distance (lower better) | 0.19 | 0.26 | 0.22 | 0.31 |
| Cost per note | $0.08 | $0.11 | $0.09 | $0.14 |
Model A — the original, ranked fourth publicly, replaced in February — is best on this task and cheapest. They switched back in August.
The six months cost roughly eight weeks of engineering time, a measurable decline in the product, and the goodwill of a clinical team who had watched output quality drop while being told the model was improving.
What is standard now#
The 200-transcript set is the only benchmark that decides anything. Public benchmarks are used to build a shortlist of candidates worth running against it, which is a genuine use.
No model change ships without a run. Twenty minutes, about $9.
The set is rerun quarterly on the model in use, which has since caught a provider version change that cost 3 points of structure compliance.
Fabricated clinical content is a blocking measure. Any model above 1% is not adopted whatever else it does — a threshold that would have excluded both B and D on the day they were chosen.
What was learned#
A benchmark measures its own task. Nothing about a leaderboard position transfers to a different task, a different input shape or a different output format, and none of those differences are visible in the rank.
Constrained output and fluency pull in opposite directions. The models that write best prose followed the template worst, and every benchmark in use rewarded the prose.
Switching has a cost that nobody counts. Four switches consumed eight weeks and left the product worse, and none of the four decisions was individually unreasonable given the information being used.
Three weeks of gathering reference data ended six months of guessing. It is the most common shape in this whole field: the measurement is cheaper than the confusion it removes.