BvLogic
Diagram · RAG

RAG Architecture Diagram

The two halves of a retrieval-augmented generation system drawn as they actually run — the offline index build, the per-query retrieval path, and the generation step — with the points where answer quality is really decided.

Download the diagram Read the full guide

SVG. No sign-up, no email.

Most RAG diagrams show a single arrow from "documents" to "answer". That hides the thing that matters: a RAG system is two systems on different clocks. One builds an index on a schedule. The other answers a question in a few hundred milliseconds. They share nothing at runtime except the index, and almost every failure is a failure of one half being tuned without regard to the other.

Retrieval-augmented generation — index build and query path Index · offline: Source systems (docs, wiki,,tickets, code) → Extract & clean (strip nav,,headers,,boilerplate) → Chunk (split on,structure, not,length) → Embed (one model,,versioned) → Vector + keyword index (stores text,,vector, source,,date). Retrieve · per query: User question () → Rewrite (resolve pronouns, add,context) → Search (vector + keyword, both) → Rerank (cross-encoder, keep 3–5). Generate: Assemble prompt (passages + question +,rules) → Model () → Answer with citations (every claim traceable) → Log question, passages, answer (this is your evaluation,set). Index · offline Source systems docs, wiki, tickets, code Extract & clean strip nav, headers, boilerplate Chunk split on structure, not length Embed one model, versioned Vector + keyword index stores text, vector, source, date Retrieve · per query User question Rewrite resolve pronouns, add context history Search vector + keyword, both against the index Rerank cross-encoder, keep 3–5 Generate Assemble prompt passages + question + rules Model Answer with citations every claim traceable Log question, passages, answer this is your evaluation set top passages only Where answer quality is usually won or lost Runs on a schedule, not per request
The index is built on a schedule; the query path runs per request. They meet only at the vector index. Shaded boxes are where retrieval quality is actually decided — not at the model.

How to read it#

The index half runs on a schedule. Nothing in it happens while a user waits. That is a freedom people forget: you can afford expensive cleaning, structural chunking and quality checks here, because the cost is paid once per document, not once per question.

The query half has a latency budget. Everything between the question and the answer competes for the same few hundred milliseconds. This is why reranking is the step teams cut first and regret first — it is the most expensive box in the row and the one that most improves the answer.

They meet at exactly one place. Change the embedding model and the index becomes meaningless for the queries — but nothing errors. Retrieval simply gets quietly worse. This is the single most common silent failure in production RAG, and the reason the index stores its embedding model version alongside the vectors.

Where it actually goes wrong#

The two shaded boxes account for most of the complaints that arrive as "the AI is wrong".

Chunking decides what can ever be retrieved. A chunk that splits a table from its heading, or a procedure from its precondition, produces passages that are individually grammatical and collectively useless. Split on the document's own structure — sections, list items, table rows — and only fall back to fixed length when there is no structure.

Search and rerank decide what is actually retrieved from what could be. Pure vector search misses exact terms: part numbers, error codes, names. Pure keyword search misses paraphrase. Running both and merging is not a refinement, it is the baseline. Reranking then does the job the first pass cannot afford: reading each candidate against the question properly, and throwing away the near-misses that would otherwise fill the prompt with plausible noise.

Note what is not shaded. The model is rarely the problem. When a RAG answer is wrong, the passages that produced it are usually wrong too — which is exactly why the last box in the generation lane exists. Logging the question, the retrieved passages and the answer gives you the only evaluation set that reflects your real users, and it costs nothing to collect from day one.

Using this diagram#

Draw your own system against this one and mark the boxes you do not have. Most teams discover they have no rewrite step, no reranker, and no log — and that their retrieval is vector-only. Those four gaps, in that order, are usually a bigger improvement than any change of model.

Back to RAG