Worked Example — The Citations That Made Things Worse
A worked example of a feature that reduced errors and increased harm — how citations raised trust faster than accuracy, what the measurement showed, and the redesign that closed the gap.
This is an illustrative example. The product, study and figures are invented. The effect it describes — a trust signal outrunning the accuracy it signals — is worth designing against whether or not your numbers look like these.
The situation#
A compliance software company added citations to its regulatory assistant. Every claim in an answer now carried a link to the source paragraph.
It was a clear improvement by every measure the team tracked. Groundedness rose from 88% to 94% of claims supported, because the requirement to cite discouraged unsupported elaboration. User satisfaction rose. Support tickets about the assistant fell by a third.
Four months later, a customer's internal audit found that a compliance officer had filed a return based on an assistant answer that was wrong. The citation was present, the citation was real, and the citation did not say what the answer claimed it said.
What the investigation found#
The team pulled 400 answers and checked each citation against the claim it supported.
| Share | |
|---|---|
| Claim supported, citation correct | 91% |
| Claim supported, citation points to the wrong passage | 3% |
| Claim unsupported, citation present and plausible | 6% |
Six per cent is a lower error rate than before citations existed. The harm was higher.
They then ran a study with 60 users of the product, presenting the same 20 answers — half of them containing a known error — in two formats: with and without citations.
| No citations | With citations | |
|---|---|---|
| Answers users checked against the source | 34% | 9% |
| Errors users detected | 41% | 19% |
| Users' stated confidence in the answer | 6.2 / 10 | 8.4 / 10 |
Citations reduced verification by three quarters. A citation reads as evidence that checking has already happened. The presence of a link satisfied the impulse that the link was supposed to serve.
Accuracy improved by 6 points. Detection of the remaining errors fell by 22. The net effect on errors reaching a filing was negative.
The part that was hardest to accept#
The feature was working as designed. It genuinely reduced unsupported claims. Every measure the team had chosen in advance improved.
The measure they had not chosen was whether users caught what remained. No amount of groundedness testing would have surfaced this, because groundedness measures the system and the failure was in the interaction between the system and a person.
The redesign#
Four changes, aimed at making the citation carry its own uncertainty rather than implying there is none.
The quoted text appears inline, not behind a link. The passage that supports the claim is shown. A user who reads three words of it learns more than one who does not click, and almost nobody clicks.
Claims are marked by support type. Directly stated, inferred from the passage, or not found in the sources. The third category previously did not exist — those claims simply appeared without a citation, which readers did not notice.
Inferred claims are visually distinct and carry the reasoning. Roughly a fifth of claims are inferences, and the study showed users had been treating them exactly as they treated direct quotations.
Answers touching a filing deadline or a monetary threshold require the user to expand the source before the answer can be copied. Deliberate friction, applied only where the cost of being wrong is highest, and it was the most contested change internally.
The result#
Same study design, 60 different users, four months later.
| Original citations | Redesigned | |
|---|---|---|
| Answers checked against the source | 9% | 47% |
| Errors detected | 19% | 58% |
| Stated confidence | 8.4 | 7.6 |
| Claim-level groundedness | 94% | 95% |
Confidence fell and that is the improvement. Users are now roughly as confident as the system deserves, and they catch more than half of what it gets wrong — better than either the no-citation or the original-citation version.
The friction on high-stakes answers was measured separately: it adds about 11 seconds and it is where two thirds of the additional detections occur.
What was learned#
Measure the human outcome, not just the system output. Groundedness improved throughout. The thing that got worse — whether errors reached a filing — was never on the dashboard.
A trust signal has to be calibrated to the trust it earns. A citation implies verification. If 6% of them do not support the claim, the implication is wrong 6% of the time and it is being made with complete confidence.
Show the evidence, do not link to it. Nine per cent of users clicked. Forty-seven per cent read text that was already in front of them.
Falling confidence can be the goal. It is an uncomfortable metric to move deliberately and it was the point of the entire redesign.