BvLogic
Diagram · RAG Testing

Where to Measure a RAG System — Diagram

The three points a retrieval system must be measured at, drawn against the pipeline — why an end-to-end score hides the ceiling, and which metric belongs at each point.

Download the diagram Read the full guide

SVG. No sign-up, no email.

Most teams measure a retrieval system in one place: the answer. That single number cannot distinguish a retrieval failure from a generation failure, and the two need entirely different fixes.

The diagram puts a measurement point at each stage. The important one is the first, because it sets a ceiling everything downstream is trapped beneath.

Three measurement points, three different failures Retrieve: Question () → Search index () → Passages returned () → Was the answer in them? (hit rate / recall@k). Generate: Model () → Answer produced () → Grounded in the passages? (claims traceable to source) → Refused when it should? (on unanswerable questions). End to end: Correct answer rate (), Over-refusal rate (refused when it could have,answered), Cost per answer (), p95 latency (). Retrieve Question Search index Passages returned Was the answer in them? hit rate / recall@k Generate Model Answer produced Grounded in the passages? claims traceable to source Refused when it should? on unanswerable questions End to end Correct answer rate Over-refusal rate refused when it could have answered Cost per answer p95 latency passages + question Measure here The ceiling — nothing downstream can exceed it
Measure at each shaded point separately. An end-to-end score alone cannot tell you which stage failed, and the retrieval number caps everything after it.

Why the first point is the ceiling#

If the correct passage is not retrieved, no model can produce a grounded answer from it. A retrieval hit rate of 60% means the system's honest ceiling is 60%, and every hour spent on prompt wording is spent under that ceiling.

This is the single most common misallocation of effort in retrieval projects: weeks on generation, against a retrieval problem.

Measure it first and separately. Take questions with known answers, run retrieval alone, and record whether the passage containing the answer came back at all.

What each point tells you#

PointMetricA bad number means
Retrievalhit rate / recall@kchunking, embedding, or hybrid search missing
Generationgroundednessthe model is adding claims the passages do not support
Generationrefusal accuracyit answers when it should decline
End to endcorrect answer ratethe combination — diagnose using the two above
End to endover-refusalit declines when the answer was present

The pair that must be read together#

Refusal accuracy and over-refusal move against each other. A system tuned to refuse readily scores well on the first and badly on the second, and looks admirably cautious while being useless.

Report both, always. One without the other is a metric you can trivially satisfy in the wrong direction.

The third lane has no arrows#

Deliberately — those four are not a sequence. They are independent properties of the finished system, measured over the same run, and none is derived from the others.

Practical minimum#

Twenty real questions with known answers. Retrieval hit rate recorded before any tuning. Then groundedness and refusal on the same set, over at least three runs, reported as a pass rate with the run count.

Everything else is refinement on top of those numbers.

See RAG testing for the full method, the RAG guide for the system being tested, and the RAG checklist before shipping one.

Back to RAG Testing