BvLogic
Diagram · Hallucination Testing

Measuring Groundedness — Diagram

How a claim is actually checked against its sources — the decomposition step most teams skip, the three verdicts per claim, and why an answer-level score hides partial fabrication.

Download the diagram Read the full guide

SVG. No sign-up, no email.

"Did the model make something up?" is not a question you can answer about a whole answer. A paragraph can be four supported claims and one invented number, and any answer-level score records that as either right or wrong — both of which are misleading.

Groundedness is measured per claim. The decomposition step below is the one teams skip, and skipping it is why hallucination metrics so often fail to move when the underlying behaviour changes.

One answer, decomposed and checked claim by claim Produce: Question () → Passages retrieved (recorded, not just used) → Answer generated (). Decompose and check: Split into claims (each independently,checkable) → For each: in the passages? () → Verdict per claim (supported / contradicted /,absent) → Score the answer (share of claims supported). Report: Groundedness rate (claims supported ÷ total), Answers with any unsupported claim (the number people care,about), Refusal accuracy (on unanswerable questions), Over-refusal (report with the line above). Produce Question Passages retrieved recorded, not just used Answer generated Decompose and check Split into claims each independently checkable For each: in the passages? Verdict per claim supported / contradicted / absent Score the answer share of claims supported Report Groundedness rate claims supported ÷ total Answers with any unsupported claim the number people care about Refusal accuracy on unanswerable questions Over-refusal report with the line above per claim The step that makes the metric meaningful The verdict that matters most
The shaded step is the one most often skipped. Without it, a partially fabricated answer scores as a single pass or fail and the fabrication is invisible.

The three verdicts, and why "absent" is not "contradicted"#

Supported — the claim is stated in, or directly follows from, the retrieved passages.

Contradicted — the passages say something different. Rare, and serious.

Absent — the passages simply do not address it. This is the common case and the one that matters. The claim may even be true — drawn from the model's training rather than from your documents — and it is still a fabrication with respect to the sources, because nothing in the system verified it.

Collapsing "absent" into "wrong" understates the problem; collapsing it into "fine because it happens to be correct" understates it more.

The two numbers to report#

Groundedness rate — claims supported divided by total claims. Good for tracking change over time.

Share of answers containing at least one unsupported claim — the number that reflects user experience. An answer that is 90% grounded still contains something invented, and a reader has no way to tell which part.

Teams that report only the first can show a healthy 94% while a third of answers contain a fabrication.

Who does the checking#

Humans, for the calibration set. Slow, and it is the only ground truth you have.

A model judge, for scale — with its agreement against those human labels measured first. If the judge agrees with humans 80% of the time, every subsequent figure carries that error and should be quoted as approximate.

Judging one claim against specific passages is a narrow, well-posed question — much better suited to a model judge than "is this a good answer?"

The last lane has no arrows#

Those four are independent properties of the same evaluation run. In particular, refusal accuracy and over-refusal must always be reported together: a system that refuses everything scores perfectly on groundedness and is useless.

See hallucination testing for the full method, where to measure a RAG system for the stage before this, and RAG for the system under test.

Back to Hallucination Testing