Retrieval Regression Report — Sample
A worked example of testing retrieval after a knowledge base migration — the golden set as a gate, which queries broke, and why aggregate recall stayed flat while a quarter of the set moved.
Markdown. No sign-up, no email.
This is an illustrative example. The system, migration and figures are invented. The scenario is common: the knowledge base changes, no code changes, and retrieval quietly moves.
Retrieval regression — knowledge base migration#
| System | Internal technical support assistant |
| Change | Content migrated to a new documentation platform. Same material, re-exported. |
| Our code | Unchanged. Same chunker, same embedding model, same query path. |
| Golden set | 220 queries, each with the document that must be retrieved, labelled by hand |
| Run | Before migration and after, same queries, same day |
1. Result#
Do not cut over. Aggregate recall is flat. Twenty-six per cent of the golden set changed outcome, and the losses are concentrated in the queries that matter most.
2. The aggregate that hid everything#
| Before | After | |
|---|---|---|
| Correct document in top 10 | 88.2% | 87.7% |
| Correct document in top 3 | 79.5% | 78.6% |
| Correct document ranked first | 64.1% | 63.2% |
Half a point down across the board. On these three rows the migration is a non-event, and this is what would have been reported had the golden set been scored only in aggregate.
3. Query by query#
| Queries | |
|---|---|
| Improved | 28 |
| Regressed | 30 |
| Unchanged | 162 |
Fifty-eight queries — 26% of the set — moved. They net to −2, which is how a quarter of a system changing behaviour presents as half a point.
The 30 regressions group into three causes, and none of them is subtle once seen.
| Cause | Queries | What the migration did |
|---|---|---|
| Heading hierarchy flattened | 14 | H2/H3 exported as bold paragraphs, so chunking lost structure |
| Tables exported as pipe-delimited text | 9 | Column headers no longer attached to values |
| Code blocks merged with prose | 5 | Chunk boundaries fall mid-example |
| No pattern found | 2 | — |
All three are export format, not content. The material is identical. The structure our chunker depends on was not preserved, and nothing in the migration plan mentioned structure because nothing had made it visible as a dependency.
4. Why the regressions matter more than the improvements#
The 30 regressions include 11 of the 40 highest-frequency queries in the set. The 28 improvements include 2.
Weighted by how often each query is actually asked, the picture inverts:
| Before | After | |
|---|---|---|
| Unweighted recall at 3 | 79.5% | 78.6% |
| Frequency-weighted recall at 3 | 83.1% | 74.8% |
An 8.3-point fall on the queries people actually ask. The unweighted number is the one that looked flat.
The 14 heading-hierarchy failures are concentrated in the configuration and troubleshooting sections, which are the most-read pages in the library. That is not coincidence — heavily structured documents are both the most useful and the most damaged by a flattening export.
5. What did not change#
- Embedding model and query path — deliberately held constant, which is what makes this a clean comparison
- Latency unchanged
- Queries with no correct answer in the library still correctly return nothing useful, 25 of 25
- Duplicate detection unaffected
Holding everything else constant is the reason a cause could be identified at all. Had the platform migration coincided with a chunker change, this report could have described the effect and not the reason.
6. What is needed before cutover#
- Export headings as headings. The platform supports it; the migration used a default template that did not.
- Export tables as tables, or as structured data the chunker can keep intact.
- Keep code blocks as single units with the surrounding explanation.
- Re-run this golden set and require frequency-weighted recall at 3 to be within one point of the current system.
Estimated: three days of export configuration. The alternative — accepting the migration and recovering with chunker changes — was considered and rejected, because it makes the chunker absorb a problem created upstream and leaves the next export equally fragile.
7. What this run added permanently#
Frequency weighting is now part of the golden set. Each query carries how often it is asked. The unweighted figure remains reported, because a rare query that fails is still a failure, but the weighted figure is the gate.
Structure preservation is now a migration requirement, written down. It was an invisible dependency of the retrieval system and it will be an invisible dependency of the next one.
Notes on using this format#
Compare query by query. Twenty-six per cent of the set moved and the aggregate showed half a point. Aggregates measure whether the average changed, not whether the system did.
Weight by frequency, and report both. The unweighted number said the migration was harmless. The weighted number said it removed 8 points from the queries people ask.
Hold everything else constant. One change per comparison is what turns a result into a cause.
Golden sets are cheap and permanent. 220 hand-labelled queries took two days once, and this migration is the third time they have prevented a regression reaching production.