RAG Evaluation Report — Sample
A worked example of what a retrieval-augmented generation evaluation should produce — retrieval and answer scored separately, failures grouped by cause, and a recommendation that follows from the numbers.
Markdown. No sign-up, no email.
This is an illustrative example. Every figure below is invented to show the shape of a useful report. Nothing here describes a real system. Copy the structure; produce your own numbers.
The point of the format is one thing most RAG evaluations get wrong: retrieval and generation are scored separately. A single "accuracy" number cannot tell you whether the model reasoned badly or was handed the wrong passages, and those two problems have nothing in common to fix.
Internal knowledge assistant — evaluation, cycle 4#
| System | Internal policy and procedure assistant |
| Evaluated | 2026-07-28 to 2026-08-01 |
| Question set | 180 questions — 120 sampled from real usage, 60 written to cover known gaps |
| Judged by | Two reviewers independently; disagreements resolved by a third |
| Compared against | Cycle 3 (2026-06-30), same question set |
1. Headline#
| Measure | Cycle 3 | Cycle 4 | Change |
|---|---|---|---|
| Answer correct and complete | 61% | 74% | +13 pts |
| Answer correct but incomplete | 18% | 14% | −4 pts |
| Answer wrong | 14% | 7% | −7 pts |
| Declined to answer | 7% | 5% | −2 pts |
The improvement came from one change: adding a reranking step. No prompt was edited and the model was not changed between cycles.
2. Retrieval, scored on its own#
Measured before the model sees anything: for each question, was the passage containing the answer retrieved at all?
| Measure | Cycle 3 | Cycle 4 |
|---|---|---|
| Answer present in top 10 | 84% | 86% |
| Answer present in top 3 | 58% | 81% |
| Answer ranked first | 41% | 66% |
Reading this correctly matters. Recall at 10 barely moved — the right passage was already being found. What changed is where it ranked. Cycle 3 retrieved the answer and then buried it beneath four plausible near-misses that filled the prompt. The model was not the bottleneck; the ordering was.
The remaining 14% where the answer is not in the top 10 is a different problem entirely, and reranking cannot touch it. See section 4.
3. Generation, given correct passages#
Restricted to the 155 questions where the answer was demonstrably in the retrieved set.
| Outcome | Count | Share |
|---|---|---|
| Correct, with a citation that supports it | 128 | 83% |
| Correct, but cited the wrong passage | 9 | 6% |
| Missed an answer that was present | 12 | 8% |
| Contradicted the passage | 6 | 4% |
The 9 mis-citations matter more than their count suggests. An answer with a citation that does not support it is worse than no citation, because a reader who checks one and finds it reasonable stops checking.
The 6 contradictions all shared a shape: two retrieved passages disagreed, one current and one superseded, and the model reconciled them rather than flagging the conflict. Neither passage carried a visible effective date.
4. Failures grouped by cause#
Grouping by cause, not by symptom, is what makes a report actionable.
| Cause | Questions | What it looks like | Fixable by |
|---|---|---|---|
| Content genuinely absent | 11 | Confident answer from adjacent material | Writing the missing page |
| Answer split across sections | 8 | Half an answer, correctly cited | Chunking on structure |
| Superseded content still indexed | 6 | Correct-sounding, out of date | Effective dates + retirement |
| Table separated from its heading | 5 | Numbers without their meaning | Chunking on structure |
| Question needed context from earlier | 4 | Answer to a different question | Query rewriting |
| Ambiguous question | 3 | Reasonable answer to one reading | Asking back |
Two of these — chunking on structure and effective dates — account for 19 of the 37 failures, and both are index-side. They cost nothing at query time.
5. What we are not measuring yet#
Stated explicitly, because an evaluation that hides its gaps is worse than a smaller honest one.
- Latency is not in this cycle. Reranking added a step; nobody measured what it cost.
- Cost per question is not tracked.
- Real-world satisfaction is unknown. The 120 sampled questions came from logs, but nobody asked the people who asked them whether the answer helped.
- The question set over-represents policy questions because that is what early users asked. It is not yet a fair sample of current usage.
6. Recommendation#
- Re-chunk on document structure rather than fixed length. Addresses 13 of 37 failures and requires no query-path change.
- Add effective dates to the index and retire superseded content. Addresses 6 failures and removes the class where the answer is confidently wrong.
- Measure latency and cost per question before the next cycle — reranking has been shipped without either being known.
- Do not change the model. The generation lane is performing well on correct passages, and changing it now would make the next comparison uninterpretable.
Not recommended this cycle: query rewriting. It addresses 4 failures and adds a step to the latency budget we have not yet measured.
Notes on using this format#
Three properties make the difference between this and a dashboard nobody reads.
Score the halves separately. Every decision above follows from splitting retrieval from generation. Combine them and the only available conclusion is "it got better", which is not a plan.
Group failures by cause. Thirty-seven individual failures are a list. Six causes are a backlog, ordered by how many failures each one closes.
Change one thing per cycle. This cycle changed reranking alone, which is why the +13 points can be attributed at all. Two changes at once and the report can describe what happened but not why.