Checklist · RAG

RAG Implementation Checklist

Build a retrieval-augmented system that works, content ownership, chunking, hybrid search, permissions and measurement. Starts with the question that decides the project.

About the numbers here. Numbers here are thresholds the checklist proposes, not measurements.

Download the checklist (Word) Read the full guide

Markdown. No sign-up, no email.

System: _______________ Date: _______

1. Before writing any code#

[ ] Who owns each document set, and is responsible for it being current? If the answer is "nobody", fix that first: the most common RAG failure is a stale document retrieved perfectly
[ ] Corpus scoped: start with one team's documents, not everything
[ ] Twenty real questions written down, with known answers
[ ] Decided what happens when the answer is not in the corpus
[ ] Confirmed the knowledge is actually written down, not in people's heads
[ ] Fine-tuning ruled out, facts belong in documents, not weights

2. Chunking#

[ ] Split on structure. Sections, headings, paragraphs. Not fixed character counts
[ ] Small overlap so a thought spanning a boundary survives
[ ] Document title and heading path prepended to every chunk
[ ] Chunks large enough to contain an answer, not just to match a query
[ ] Metadata stored: owner, department, date, access level, document type

3. Retrieval#

[ ] Hybrid search: keyword AND semantic. Semantic alone fails on exact identifiers
[ ] Metadata filtering available to narrow before ranking
[ ] Embedding model and version pinned
[ ] Number of chunks retrieved tuned, a cost lever and a quality lever at once
[ ] Re-indexing process defined for when documents change

4. Generation#

[ ] Instruction to answer only from supplied passages
[ ] Explicit refusal instruction. "if the passages do not contain the answer, say so"
[ ] Supplied content clearly delimited as data, not instructions
[ ] Output length constrained
[ ] Model and version pinned

5. Sources#

[ ] Every answer shows the chunks it used
[ ] Sources are clickable and lead to the real document
[ ] Document date shown, so a reader can judge currency

This is not cosmetic. It is the mechanism by which users catch mistakes, and the difference between a system people trust and one they quietly stop using.

6. Permissions: a security requirement#

[ ] Retrieval filtered by the requesting user's permissions
[ ] Filtering happens at query time, not after generation
[ ] System does not use a privileged service account
[ ] Tested with users of different entitlements
[ ] Failure treated as a security incident

7. Measurement: before launch#

[ ] Retrieval measured separately from generation
[ ] Hit rate / recall@k recorded
[ ] Groundedness measured
[ ] Refusal accuracy measured on unanswerable questions
[ ] Over-refusal measured
[ ] Baseline recorded so later changes are comparable

If retrieval hit rate is 60%, your system's ceiling is 60%. No prompt work raises it.

8. Operations#

[ ] Re-index triggered when documents change
[ ] Retrieval confidence logged
[ ] Source click-through measured
[ ] Every reported bad answer added to the test set permanently
[ ] Document owners notified when their content produces a bad answer
[ ] Scheduled re-test: the corpus changes even when the code does not

9. Cost#

[ ] Cost per answer measured
[ ] Caching for repeated questions
[ ] Chunk count tuned against quality, not maximised

Sign-off#

	Name	Date
Built by
Content owner
Approved for users

Back to RAG