Groundedness Audit Report — Sample
A worked example of auditing an AI system for unsupported claims — how groundedness is measured claim by claim, the confident-and-wrong cases that matter most, and what actually reduced them.
Markdown. No sign-up, no email.
This is an illustrative example. The system and every figure are invented to show the shape of an audit that produces fixes rather than anxiety. Copy the method; measure your own system.
The audit measures one thing precisely: what proportion of the claims in an answer are supported by the source material provided. Not whether the answer "seems right" — whether each individual statement can be traced.
Groundedness audit — customer policy assistant#
| System | Assistant answering customer questions from a policy library |
| Audited | 2026-07-20 to 2026-07-27 |
| Sample | 240 answers — 180 from production logs, 60 written to probe known weak areas |
| Method | Each answer split into atomic claims; each claim checked against the retrieved passages |
| Reviewers | Two independent, third resolving disagreement |
1. Headline#
| Answers | Share | |
|---|---|---|
| Every claim supported | 171 | 71% |
| One or more unsupported claims | 52 | 22% |
| Correctly declined to answer | 17 | 7% |
Claim-level, which is the number that matters:
| Claims | Share | |
|---|---|---|
| Total claims across all answers | 1,486 | |
| Supported by a retrieved passage | 1,342 | 90.3% |
| Not supported | 144 | 9.7% |
Ninety per cent of claims supported sounds strong and is not. An answer averages six claims, so a 9.7% claim-level failure rate produces an unsupported statement in roughly a fifth of answers.
2. The unsupported claims, by type#
| Type | Claims | What it looks like |
|---|---|---|
| Plausible elaboration | 61 | Detail that fits, that no passage contains |
| Merged from two passages | 33 | Facts from separate products combined into one answer |
| Outdated but retrieved | 24 | Supported by a passage that is no longer current |
| Arithmetic on retrieved values | 15 | Correct inputs, wrong calculation |
| Contradicts the passage | 11 | Directly wrong |
Plausible elaboration is the largest category and the hardest to notice. The answer is mostly correct; one sentence adds a detail that was never in the source. A reader checking a citation finds the surrounding claims supported and stops checking.
The 33 merged claims share a cause worth naming: two products with similar names, whose passages arrive together and are not distinguishable once inside the prompt.
3. The cases that matter most#
Cross-tabulating unsupported claims against how confidently they were stated:
| Hedged | Stated plainly | Stated emphatically | |
|---|---|---|---|
| Supported claims | 214 | 1,009 | 119 |
| Unsupported claims | 12 | 98 | 34 |
Thirty-four unsupported claims were stated emphatically — "always", "in every case", "you must". These are the most damaging output the system produces, because confidence is the only signal a reader has when they cannot check.
There is no relationship between how confidently the system states something and whether it is supported. That is worth stating plainly to anyone inclined to trust the tone.
4. Where the audit disagreed with itself#
Reviewers disagreed on 31 of 1,486 claims, about 2%. Nearly all were the same question: does a claim need to be stated in a passage, or is a reasonable inference from a passage acceptable?
The rule adopted, after the third reviewer resolved them: an inference is supported only if a competent reader would reach the same conclusion from the passage alone. Anything requiring outside knowledge is unsupported, even when true.
Recording the rule matters more than the 31 claims. Without it, the next audit measures something slightly different and the trend is meaningless.
5. What was changed#
Three changes, in order of what they closed.
Passages now carry their product name and effective date into the prompt, and the answer format requires each claim to name its source. Addressed the merged claims and the outdated ones — 57 of 144.
Arithmetic moved out of the model. Calculations on retrieved values are performed in code and the result is passed in. Addressed all 15 arithmetic failures, which is the only category that goes to zero rather than down.
The answer format now separates what the policy says from what it does not address. Elaboration fell but did not disappear; it is the hardest category and no format change eliminates it.
6. After#
| Before | After | |
|---|---|---|
| Claim-level groundedness | 90.3% | 96.8% |
| Answers with an unsupported claim | 22% | 9% |
| Emphatic unsupported claims | 34 | 6 |
| Arithmetic errors | 15 | 0 |
| Correctly declined | 7% | 11% |
The rise in declines is intended. The system now declines where the library genuinely does not answer the question, which was previously where elaboration came from.
7. What this audit cannot tell you#
It measures grounding, not correctness. A claim supported by a passage that is itself wrong scores as grounded. This audit assumes the library is right; auditing the library is separate work.
It does not measure usefulness. A system that declined every question would score perfectly. That is why the decline rate is reported alongside, and why a rise in it needs a reason.
The 60 probe questions bias the sample toward known weak areas. The production-only figure is 3 points better. Both are reported; neither alone is honest.
Notes on using this format#
Split answers into claims and check each one. Rating whole answers as "correct" or "not" loses the fact that most bad answers are mostly right, which is precisely what makes them persuasive.
Cross-tabulate against confidence. The emphatic unsupported claims are the most damaging output and they are invisible in a single accuracy figure.
Write down the inference rule. Two reviewers will otherwise apply different standards, and the difference between audits will exceed the difference this measures.
Report the decline rate next to groundedness. Otherwise the easiest way to improve the number is to make the system less useful.