RAG Worked Example — Rebuilding a Search Nobody Trusted

This is an illustrative example. The organisation, timeline and numbers are invented. The sequence is the point: what was tried first, why it failed, and what the evidence actually supported.

The situation#

A 400-person insurance company had built an internal assistant over its policy and procedure library — about 11,000 documents accumulated over a decade. It launched in March. By May, usage had fallen to roughly 30 questions a day across the whole company, from a peak of 400.

The verdict from the claims team was unambiguous and unhelpful: "it makes things up."

What was tried first#

Three things, in order, over six weeks. None of them worked, and the reason is instructive.

A better model. The obvious move, and the most expensive. It changed the tone of the wrong answers. It did not change how often they were wrong, because the model was never seeing the material it needed.

A stricter prompt. "Only answer from the provided context. If the context does not contain the answer, say so." Refusals went from 4% to 31%. The team briefly counted this as progress — fewer wrong answers — until a claims handler pointed out that the assistant was now refusing questions it had previously answered correctly. The passages were not there either way. All the prompt did was make the absence visible.

More retrieved passages. Top 5 became top 20, on the theory that the answer would be in there somewhere. Answers got worse. Twenty passages of mostly-irrelevant material buried the one useful passage in noise, and the model reconciled contradictions between passages that should never have been shown together.

Six weeks, three changes, no improvement. The common thread: every change was made on the generation side of a retrieval problem.

The measurement that redirected the work#

The team then did something they had skipped: they measured retrieval on its own.

They took 150 real questions from the logs and, for each, found by hand which document contained the answer. Then they asked one question of the system — was that document retrieved at all?

The 29% settled the argument. For nearly a third of questions, no prompt and no model could have produced a correct answer, because the material was never in front of it. The six weeks of generation-side work had been spent on the wrong half of the system.

What was actually wrong#

Examining the 43 questions where retrieval failed completely produced two causes, not twenty.

Chunking had been done by character count. Every document was split into 1,000-character pieces regardless of structure. In a policy library this is close to a worst case: the library is full of tables, and a table split from its heading becomes a grid of numbers with no indication of what they measure. Procedures were split from their preconditions, so a retrieved chunk would say what to do without saying when it applied.

One example, reproduced from the evaluation: a chunk containing "Excess: £250 · Waiting period: 14 days · Maximum benefit: £15,000" with no indication of which product it described. It was retrieved for questions about three different products, and it was plausible for all of them.

Superseded documents were indexed alongside current ones. The library had six versions of the claims procedure going back to 2018. None was marked as retired, none carried an effective date the index could see, and the 2019 version was often the closest match because its wording was longer and more explicit. The assistant was quoting withdrawn policy in a confident tone with a correct-looking citation.

What was done#

Two changes on the index side. Neither touched the model or the prompt.

Chunk on structure. Split on the document's own headings, list items and table boundaries, with the parent heading path prepended to every chunk. A table now arrives with its title and the product it belongs to attached. Fixed-length splitting remained only as a fallback for documents with no structure at all — about 8% of the library.

Effective dates and retirement. Each document was given an effective date and a status. Superseded versions were removed from the index, not deleted from the library. Every retrieved passage now carries its effective date into the prompt, so a date-sensitive question can be answered as such.

Both changes were index-side and ran offline. Neither added a millisecond to query latency.

The result#

Same 150 questions, three weeks later.

Usage recovered to roughly 320 questions a day over the following month. The claims team's verdict changed from "it makes things up" to a specific, actionable complaint about two document types that were still poorly handled — which is a much better place to be.

What was learned#

Measure retrieval separately before changing anything on the generation side. The six weeks lost to models and prompts were spent on a system that was working correctly with bad inputs. A single afternoon of retrieval measurement would have redirected the work.

Chunking is a content decision, not a parameter. The right chunk size for this library was never a number. It was "one section, with its heading attached", and no amount of tuning 1,000 against 500 would have found it.

Retiring content is part of the system. Nobody thought of the six historic procedure versions as a technical problem. They were the cause of the most damaging failure the assistant had — confident, well-cited, and wrong.

A refusal rate is not a quality measure. The stricter prompt made the system look safer while making it less useful, and the number it improved was the number being watched.

Back to RAG