Checklist · RAG

RAG Implementation Checklist

Build a retrieval-augmented system that works — content ownership, chunking, hybrid search, permissions and measurement. Starts with the question that decides the project.

Download the checklist Read the full guide

Markdown. No sign-up, no email.

System: _______________ Date: _______

1. Before writing any code#

[ ] Who owns each document set, and is responsible for it being current? If the answer is "nobody", fix that first — the most common RAG failure is a stale document retrieved perfectly
[ ] Corpus scoped — start with one team's documents, not everything
[ ] Twenty real questions written down, with known answers
[ ] Decided what happens when the answer is not in the corpus
[ ] Confirmed the knowledge is actually written down, not in people's heads
[ ] Fine-tuning ruled out — facts belong in documents, not weights

2. Chunking#

[ ] Split on structure — sections, headings, paragraphs. Not fixed character counts
[ ] Small overlap so a thought spanning a boundary survives
[ ] Document title and heading path prepended to every chunk
[ ] Chunks large enough to contain an answer, not just to match a query
[ ] Metadata stored: owner, department, date, access level, document type

3. Retrieval#

[ ] Hybrid search — keyword AND semantic. Semantic alone fails on exact identifiers
[ ] Metadata filtering available to narrow before ranking
[ ] Embedding model and version pinned
[ ] Number of chunks retrieved tuned — a cost lever and a quality lever at once
[ ] Re-indexing process defined for when documents change

4. Generation#

[ ] Instruction to answer only from supplied passages
[ ] Explicit refusal instruction — "if the passages do not contain the answer, say so"
[ ] Supplied content clearly delimited as data, not instructions
[ ] Output length constrained
[ ] Model and version pinned

5. Sources#

[ ] Every answer shows the chunks it used
[ ] Sources are clickable and lead to the real document
[ ] Document date shown, so a reader can judge currency

This is not cosmetic. It is the mechanism by which users catch mistakes, and the difference between a system people trust and one they quietly stop using.

6. Permissions — a security requirement#

[ ] Retrieval filtered by the requesting user's permissions
[ ] Filtering happens at query time, not after generation
[ ] System does not use a privileged service account
[ ] Tested with users of different entitlements
[ ] Failure treated as a security incident

7. Measurement — before launch#

[ ] Retrieval measured separately from generation
[ ] Hit rate / recall@k recorded
[ ] Groundedness measured
[ ] Refusal accuracy measured on unanswerable questions
[ ] Over-refusal measured
[ ] Baseline recorded so later changes are comparable

🔴 If retrieval hit rate is 60%, your system's ceiling is 60%. No prompt work raises it.

8. Operations#

[ ] Re-index triggered when documents change
[ ] Retrieval confidence logged
[ ] Source click-through measured
[ ] Every reported bad answer added to the test set permanently
[ ] Document owners notified when their content produces a bad answer
[ ] Scheduled re-test — the corpus changes even when the code does not

9. Cost#

[ ] Cost per answer measured
[ ] Caching for repeated questions
[ ] Chunk count tuned against quality, not maximised

Sign-off#

	Name	Date
Built by
Content owner
Approved for users

Back to RAG