# RAG Evaluation Report — Sample

**This is an illustrative example.** Every figure below is invented to show the shape of a
useful report. Nothing here describes a real system. Copy the structure; produce your own
numbers.

The point of the format is one thing most RAG evaluations get wrong: **retrieval and
generation are scored separately**. A single "accuracy" number cannot tell you whether the
model reasoned badly or was handed the wrong passages, and those two problems have nothing
in common to fix.

---

## Internal knowledge assistant — evaluation, cycle 4

| | |
|---|---|
| System | Internal policy and procedure assistant |
| Evaluated | 2026-07-28 to 2026-08-01 |
| Question set | 180 questions — 120 sampled from real usage, 60 written to cover known gaps |
| Judged by | Two reviewers independently; disagreements resolved by a third |
| Compared against | Cycle 3 (2026-06-30), same question set |

## 1. Headline

| Measure | Cycle 3 | Cycle 4 | Change |
|---|---|---|---|
| Answer correct and complete | 61% | 74% | +13 pts |
| Answer correct but incomplete | 18% | 14% | −4 pts |
| Answer wrong | 14% | 7% | −7 pts |
| Declined to answer | 7% | 5% | −2 pts |

The improvement came from one change: adding a reranking step. No prompt was edited and the
model was not changed between cycles.

## 2. Retrieval, scored on its own

Measured before the model sees anything: for each question, was the passage containing the
answer retrieved at all?

| Measure | Cycle 3 | Cycle 4 |
|---|---|---|
| Answer present in top 10 | 84% | 86% |
| Answer present in top 3 | 58% | 81% |
| Answer ranked first | 41% | 66% |

**Reading this correctly matters.** Recall at 10 barely moved — the right passage was
already being found. What changed is where it ranked. Cycle 3 retrieved the answer and then
buried it beneath four plausible near-misses that filled the prompt. The model was not the
bottleneck; the ordering was.

The remaining 14% where the answer is not in the top 10 is a different problem entirely, and
reranking cannot touch it. See section 4.

## 3. Generation, given correct passages

Restricted to the 155 questions where the answer was demonstrably in the retrieved set.

| Outcome | Count | Share |
|---|---|---|
| Correct, with a citation that supports it | 128 | 83% |
| Correct, but cited the wrong passage | 9 | 6% |
| Missed an answer that was present | 12 | 8% |
| Contradicted the passage | 6 | 4% |

The 9 mis-citations matter more than their count suggests. An answer with a citation that
does not support it is worse than no citation, because a reader who checks one and finds it
reasonable stops checking.

The 6 contradictions all shared a shape: two retrieved passages disagreed, one current and
one superseded, and the model reconciled them rather than flagging the conflict. Neither
passage carried a visible effective date.

## 4. Failures grouped by cause

Grouping by cause, not by symptom, is what makes a report actionable.

| Cause | Questions | What it looks like | Fixable by |
|---|---|---|---|
| Content genuinely absent | 11 | Confident answer from adjacent material | Writing the missing page |
| Answer split across sections | 8 | Half an answer, correctly cited | Chunking on structure |
| Superseded content still indexed | 6 | Correct-sounding, out of date | Effective dates + retirement |
| Table separated from its heading | 5 | Numbers without their meaning | Chunking on structure |
| Question needed context from earlier | 4 | Answer to a different question | Query rewriting |
| Ambiguous question | 3 | Reasonable answer to one reading | Asking back |

Two of these — **chunking on structure** and **effective dates** — account for 19 of the 37
failures, and both are index-side. They cost nothing at query time.

## 5. What we are not measuring yet

Stated explicitly, because an evaluation that hides its gaps is worse than a smaller honest
one.

- **Latency** is not in this cycle. Reranking added a step; nobody measured what it cost.
- **Cost per question** is not tracked.
- **Real-world satisfaction** is unknown. The 120 sampled questions came from logs, but
  nobody asked the people who asked them whether the answer helped.
- The question set **over-represents policy questions** because that is what early users
  asked. It is not yet a fair sample of current usage.

## 6. Recommendation

1. **Re-chunk on document structure** rather than fixed length. Addresses 13 of 37 failures
   and requires no query-path change.
2. **Add effective dates to the index and retire superseded content.** Addresses 6 failures
   and removes the class where the answer is confidently wrong.
3. **Measure latency and cost per question before the next cycle** — reranking has been
   shipped without either being known.
4. **Do not change the model.** The generation lane is performing well on correct passages,
   and changing it now would make the next comparison uninterpretable.

**Not recommended this cycle:** query rewriting. It addresses 4 failures and adds a step to
the latency budget we have not yet measured.

---

## Notes on using this format

Three properties make the difference between this and a dashboard nobody reads.

**Score the halves separately.** Every decision above follows from splitting retrieval from
generation. Combine them and the only available conclusion is "it got better", which is not
a plan.

**Group failures by cause.** Thirty-seven individual failures are a list. Six causes are a
backlog, ordered by how many failures each one closes.

**Change one thing per cycle.** This cycle changed reranking alone, which is why the +13
points can be attributed at all. Two changes at once and the report can describe what
happened but not why.