Worked Example — Six Weeks Tuning Chunk Size
A worked example of optimisation without measurement — a team tuning parameters by feel for six weeks, and what two days of building a golden set revealed instead.
This is an illustrative example. The team, timeline and figures are invented. The pattern — tuning a parameter because it is the knob within reach, without anything that could tell you whether it helped — is the most common way time is lost on retrieval systems.
The situation#
An engineering team had built a support assistant over 6,000 product documentation pages. Users said the answers were "hit and miss". The team set out to improve retrieval.
They spent six weeks on chunk size.
What six weeks looked like#
| Week | Change | How it was judged |
|---|---|---|
| 1 | 1,000 → 500 characters | Team tried ~15 questions each. Felt better. |
| 2 | 500 → 750 | Felt about the same |
| 3 | Added 100-character overlap | Two people preferred it, one did not |
| 4 | Overlap to 200 | Inconclusive |
| 5 | Back to 1,000 with 200 overlap | "Probably the best so far" |
| 6 | Switched embedding model | Everything felt different, nothing was measured |
Every judgement came from a handful of ad-hoc questions typed by people who knew what answer they were hoping for. Each rebuild took four hours. Six weeks produced no evidence in either direction, and week 6 changed two things at once, which made the previous five weeks uninterpretable even in principle.
The user complaints did not change.
The two days that ended it#
A new team member asked what turned out to be the only question that mattered: when it fails, is the right document being retrieved at all?
Nobody knew. Building the thing that could answer it took two days.
150 real questions were taken from support logs. For each, someone found by hand which documentation page contained the answer. That is the whole golden set: a question, and the document that must come back.
Then one measurement, against the current configuration:
| Result | |
|---|---|
| Correct document in the top 10 | 62% |
| Correct document in the top 3 | 44% |
| Correct document ranked first | 29% |
| Correct document not retrieved at all | 38% |
Thirty-eight per cent. For nearly four questions in ten, no chunk size would have helped, because the document never entered the candidate set.
What was actually wrong#
Examining those 57 questions took an afternoon and produced three causes.
The search was vector-only. Support questions are full of exact tokens — error codes, part numbers, version strings, configuration key names. Vector search is poor at exactly these, because a code has no semantic neighbourhood. A question about error E-4471 retrieved pages about errors in general.
This accounted for 31 of the 57.
Roughly 900 pages had never been indexed. A pipeline failure eleven months earlier had skipped one documentation section, exited zero, and never been noticed. The section covered installation, which is what a third of support questions are about.
This accounted for 19 of the 57.
Seven questions had no answer in the documentation at all. The material was genuinely missing. This is a content problem masquerading as a retrieval problem, and it is worth separating out precisely so that nobody tries to fix it with an index.
The changes#
Hybrid search. Keyword search run alongside vector search, results merged. Two days of work.
Re-index, with a count check. The 900 pages were indexed, and the pipeline now compares document count against the source and fails if it drops. Half a day.
Reranking. Added afterwards, once retrieval was finding the right material, to improve where it ranked. Three days.
Chunk size was left at 1,000 with 200 overlap — where week 5 had happened to leave it. Once measured properly, chunk size was worth about 2 points across the plausible range, and the golden set could finally say so.
The result#
| Before | After | |
|---|---|---|
| Correct document not retrieved | 38% | 4% |
| Correct document in the top 3 | 44% | 86% |
| Correct document ranked first | 29% | 71% |
| Answers judged correct by support staff | 51% | 84% |
| Elapsed | 6 weeks, no change | 6 days |
What the golden set is worth now#
It has become the most reused artefact the team owns. Since it was built it has caught:
- A documentation platform migration that flattened headings and cost 8 points of weighted recall
- An embedding model version change that moved 40 queries
- A well-intentioned filter that excluded archived pages and took some current ones with it
Two days of hand-labelling, three regressions caught before release. None of the three would have been visible to a person trying fifteen questions and forming an impression.
What was learned#
Measure before optimising. Six weeks went into the parameter that was worth two points, because it was the parameter that was easy to change.
Ask whether the answer is retrievable at all. Recall against a labelled set is the first measurement, it takes two days to make possible, and it separates retrieval problems from ranking problems from content problems.
Change one thing at a time. Week 6 changed chunking and the embedding model together, which destroyed the ability to interpret anything before it.
A silent pipeline failure can look like a retrieval quality problem. Nine hundred missing pages presented as "the assistant is bad at installation questions" for eleven months.