# Internal Benchmark Report — Sample **This is an illustrative example.** The models are anonymised and every figure is invented. A benchmark's numbers are only meaningful for the task and set that produced them, which is precisely why copying someone else's is a mistake. --- ## Internal benchmark — legal document analysis | | | |---|---| | Purpose | Rank candidate models for our own task, and track drift over time | | Set | 400 items across 5 task types, built in-house from our own documents | | Contamination control | Never published, never sent outside the evaluation, rebuilt annually | | Scoring | Automated where objective; two blind human raters where not | | Run | 3 repetitions per item per model | ## 1. Headline ranking | | Score | 95% CI | Cost per item | p95 latency | |---|---|---|---|---| | Model A | 78.4 | ±2.1 | $0.211 | 5.9 s | | Model B | 76.9 | ±2.3 | $0.068 | 2.4 s | | Model C | 71.2 | ±2.6 | $0.052 | 1.9 s | **A and B are not separated by this benchmark.** The 1.5-point gap sits inside overlapping confidence intervals, and reporting "A wins" from these numbers would be reporting noise. Most published comparisons omit the interval entirely, which is how three-point differences become headlines. ## 2. By task type — where the ranking reverses | Task type | Items | Model A | Model B | Model C | |---|---|---|---|---| | Clause extraction | 120 | 84.1 | **86.7** | 79.0 | | Obligation identification | 90 | **81.2** | 77.4 | 70.1 | | Date and party extraction | 80 | 92.0 | 91.4 | 88.2 | | Risk summarisation | 60 | **74.8** | 68.0 | 61.4 | | Cross-reference resolution | 50 | 51.2 | 49.9 | 44.0 | **Model B beats Model A on the task type that is 60% of our volume.** The overall ranking is an average across a distribution that does not resemble our work. Weighted by actual volume rather than by set composition: | | Unweighted | Volume-weighted | |---|---|---| | Model A | 78.4 | 79.1 | | Model B | 76.9 | **82.3** | | Model C | 71.2 | 76.8 | The ranking reverses, and the reversal is worth more than the entire headline table. ## 3. Cross-reference resolution — the honest row Every model scores near or below 51 on cross-reference resolution. This is the row that should be read first by anyone deciding what to automate. At 51%, the task cannot be automated at any of the three. It is 12% of volume and it stays with a person. **A benchmark that reports only an average conceals which parts of a job are currently out of reach**, which is the most actionable thing it could tell you. ## 4. Contamination Taken seriously, because a benchmark a model has seen measures memory rather than capability. - **Built from our own documents**, redacted, never published anywhere - **Never sent to any provider outside the evaluation itself**, and no provider is offered it for improvement purposes - **Canary items:** 12 items contain a distinctive invented phrase. If it ever appears in a model's output unprompted, the set is compromised and rebuilt. - **Rebuilt annually** with fresh documents, keeping 20% of the previous set to bridge the two versions - Providers' published scores on public benchmarks are recorded for context and are **not used in the ranking** The canary check has fired zero times in three runs. ## 5. Where scoring is subjective Two of the five task types cannot be scored automatically. Risk summarisation and obligation identification were rated blind by two people against a rubric, with a third resolving disagreement. | | Agreement between raters | |---|---| | Obligation identification | 88% | | Risk summarisation | 71% | **Risk summarisation at 71% inter-rater agreement is the weakest number in this report.** The raters disagree with each other nearly a third of the time, which places a ceiling on how much weight the 74.8 versus 68.0 gap can carry. It is reported, and the rubric is being tightened before the next run. Reporting rater agreement is what separates a benchmark from a scoreboard. ## 6. Drift tracking The benchmark's second purpose is detecting change in a model we already use. | Run | Model B score | |---|---| | 2026-02 | 76.1 | | 2026-04 | 76.6 | | 2026-06 | 71.8 | | 2026-07 | 76.9 | **The June result is the most valuable data point here.** A 4.8-point drop, well outside the interval, on an unchanged set with unchanged prompts. It coincided with a provider-side update, was raised, and the following version restored the score. Without a stable internal set, that drop would have been experienced as scattered user complaints with no cause. ## 7. What this benchmark cannot tell you - **Nothing about tasks not in it.** Five task types is our work, not all legal work. - **Nothing about robustness to unusual inputs.** Every item is a real document of the kind we process. - **Nothing about behaviour at production volume**, latency under load, or cost at scale beyond the per-item figures. - **Nothing about how the models will behave next month**, as the June row demonstrates. ## 8. Recommendation **Model B**, on volume-weighted performance, at a third of the cost and less than half the latency. **Do not automate cross-reference resolution** at any model. Re-run on every provider version change, and on a fixed quarterly schedule regardless. --- ## Notes on using this format **Report confidence intervals.** The headline gap between the top two models was inside the noise. Without intervals it reads as a result. **Weight by your own volume.** The unweighted ranking and the weighted ranking disagreed on the winner, and only one of them describes the work. **Report the row where everything fails.** It tells you what not to automate, which is more actionable than knowing which model is 1.5 points ahead. **Guard against contamination and say how.** A benchmark that has leaked measures memory. Canary items cost nothing and make the claim checkable.