# Model Selection Report — Sample **This is an illustrative example.** The models are anonymised as A, B and C and every figure is invented, deliberately: a real comparison goes stale within weeks, and copying someone else's numbers is exactly the mistake this format exists to prevent. Copy the method. The report answers one question — **which model should run this specific task** — and it refuses to answer any broader one. There is no such thing as the best model; there is a best model for a task, at a volume, under a latency budget. --- ## Model selection — document classification and extraction | | | |---|---| | Task | Classify inbound documents into 14 types and extract 6 fields | | Volume | ~40,000 documents a month | | Latency budget | 4 seconds at the 95th percentile | | Evaluated | 2026-07-14 to 2026-07-25 | | Test set | 600 documents, human-labelled, held out from all prompt development | ## 1. The recommendation, first **Model B.** Not the highest accuracy — Model A scores 2.1 points higher — but it meets the latency budget, costs a third as much at our volume, and its errors are the recoverable kind. The 2.1 points are worth roughly 840 documents a month. The cost difference is $4,100 a month. Both numbers are in section 4, which is where the decision was actually made. ## 2. Accuracy on our task | | Model A | Model B | Model C | |---|---|---|---| | Document type correct | 96.4% | 94.3% | 89.1% | | All 6 fields correct | 88.2% | 86.9% | 74.5% | | At least one field wrong | 11.8% | 13.1% | 25.5% | | Refused or returned nothing | 0.3% | 0.8% | 2.1% | Model C is eliminated here. Its extraction accuracy is not within reach of the other two, and its 2.1% non-response rate would put roughly 840 documents a month into a queue that does not currently exist. ## 3. The errors are not the same shape This section changed the decision, and it does not appear in any benchmark. | Error type | Model A | Model B | Recoverable? | |---|---|---|---| | Field left empty | 2.1% | 7.4% | **Yes** — routed to a person | | Field filled with a plausible wrong value | 6.8% | 2.9% | **No** — looks correct downstream | | Wrong document type, confidently | 2.9% | 2.8% | Partly — caught in review | **Model A is more accurate overall and fails worse.** It guesses. When it cannot read a date, it produces a date, and a plausible wrong date flows into a system that has no way to know it is wrong. Model B leaves the field empty more often. That is a visible failure: empty fields route to a person, and the person fixes them in about 20 seconds. Our downstream process has a queue for empty and no defence at all against wrong. For this task, a recoverable error at 7.4% is preferable to an unrecoverable one at 6.8%. ## 4. Cost and latency at our volume | | Model A | Model B | |---|---|---| | Cost per document | $0.152 | $0.049 | | **Monthly at 40,000** | **$6,080** | **$1,960** | | Median latency | 3.1 s | 1.4 s | | 95th percentile | **6.8 s** | 2.9 s | | Within the 4-second budget | **No** | Yes | Model A fails the latency requirement at the 95th percentile. That alone would settle it, independent of everything above — the budget exists because documents are classified while a customer is waiting. The accuracy difference costs about 840 additional documents needing correction each month, at roughly 20 seconds each: **4.7 hours of human time**. Against $4,120 a month in additional model spend. The arithmetic is not close. ## 5. What we did not test, and why it matters Stated plainly, because an evaluation that hides its gaps invites more confidence than it earned. - **Robustness to poor scans** is under-represented. The test set came from recent documents, which are cleaner than the archive. - **Behaviour on document types outside the 14** was not tested at all. In production these exist and we do not know what any model does with them. - **Stability across model versions** is unknown by definition. Both providers update models; neither guarantees behaviour is preserved. - **Prompt sensitivity** was only lightly probed. Model B may be more sensitive to prompt changes than these results suggest. The first and last of these are the largest risks to the recommendation. ## 6. Why published benchmarks did not decide this Model A leads Model B on every public benchmark either provider cites. On our task the gap is 2.1 points, it reverses entirely once error shape is considered, and it disappears under the latency requirement. Benchmarks measure general capability on tasks that are not ours, with prompts that are not ours, at volumes where cost is not a consideration. They are useful for building a shortlist. **They cannot select a model, because they do not know what a wrong answer costs you.** ## 7. Decision and re-evaluation **Adopt Model B.** Empty-field routing is required before launch — the 7.4% rate depends on those documents reaching a person. Re-evaluate when any of these occur, not on a calendar: - Either provider ships a new version of the model in use - Monthly volume passes 80,000, where the cost gap doubles - The error mix in production diverges from this test set by more than a third - A new document type is added **The 600-document test set is now a permanent asset.** It is the only thing that makes the next comparison a comparison rather than an impression. --- ## Notes on using this format **Lead with the recommendation and the reason it is not the highest score.** That is the sentence people need, and burying it makes the report an argument to be won rather than a decision to be understood. **Classify errors by whether they are recoverable.** It is the most decision-relevant analysis here and it appears in almost no evaluation. A model that fails visibly can be operated around. One that fails plausibly cannot. **Put cost and latency at your real volume in the same table as accuracy.** Separated, accuracy always wins the argument. Together, the trade-off is visible and often decisive. **Say what was not tested.** Four gaps, stated in a paragraph, are what stop this report being read as more certain than it is.