RAG Explained: How to Make AI Answer From Your Own Documents

Ask a general AI assistant "what is our refund policy for enterprise customers in their second year?" and you get a confident, well-written, completely invented answer. The model has never seen your policy. It has seen thousands of other companies' policies, so it produces something that sounds like one.

RAG fixes this. Instead of hoping the model memorised your information, you look up the relevant documents first and hand them to the model along with the question. The model's job changes from "recall this fact" to "read these three paragraphs and answer using only them" — a task it is far better at, and one you can actually verify.

That's the whole idea. The rest is engineering.

Why not just train the model on our data?#

This is the first question everyone asks, and the answer surprises people: for most business problems, fine-tuning is the wrong tool.

The deciding factor is usually that last row. If your finance team can see documents your interns cannot, a fine-tuned model has no way to enforce that — the knowledge is smeared across its weights. RAG enforces it naturally: run the search as the user, and they can only be shown what they were already allowed to read.

Fine-tuning earns its place when you need the model to consistently produce a particular shape of output — a house writing style, a rigid JSON structure, a specialist tone. Facts belong in documents.

How it actually works#

Five steps. Only two of them involve the model at all.

1. Prepare — collect the documents and split them into chunks small enough to be precise but large enough to make sense on their own.
2. Index — convert each chunk into a numerical representation (an embedding) that captures its meaning, and store it so you can search by meaning rather than by exact words.
3. Retrieve — when a question arrives, search the index and pull back the handful of chunks most likely to contain the answer.
4. Augment — build a prompt containing the question and those chunks, with an instruction to answer only from the supplied material.
5. Generate — the model writes the answer, and you show which chunks it came from.

Question ──▶ Retrieve ──▶ 3-8 relevant chunks ──┐
                                                ├──▶ Model ──▶ Answer + sources
        System instruction: "answer only from ──┘
        the passages provided; if they don't
        contain the answer, say so"

The instruction in that last box does more work than most people expect. Without it, the model happily blends supplied documents with half-remembered training data, and you're back to plausible-sounding fiction — except now it looks sourced.

Searching by meaning, not by keyword#

Traditional search matches words. If your policy says "reimbursement window" and the user asks about "refund period", keyword search finds nothing.

Embeddings solve this by placing text in a mathematical space where similar meaning means physically close. "Reimbursement window" and "refund period" land near each other; "quarterly revenue" lands far away. Search becomes "find the chunks nearest to this question."

This is genuinely powerful, and it has a specific weakness worth knowing: embeddings are bad at exact identifiers. Ask for invoice INV-2026-0043 and semantic search returns chunks about invoices generally, because all invoice text looks similar in meaning-space.

The standard fix is hybrid search — run keyword search and semantic search together, then merge the results. Keyword catches the exact identifiers, semantic catches the paraphrases. In our experience this single change resolves more retrieval complaints than any amount of tuning.

Building one that works#

Chunk on structure, not on character count. The most common beginner approach is splitting every 500 characters. This cuts sentences in half and separates a heading from the paragraph it introduces. Split on real boundaries instead — sections, headings, paragraphs — and keep a small overlap so a thought that spans a boundary survives in at least one chunk.

Keep the context with the chunk. A chunk reading "It must be approved by two directors" is useless on its own. Two is what? Prepend the document title and heading path so the chunk carries its own context: "Expenses Policy → Approvals → Over 50,000: It must be approved by two directors." This is cheap and reliably improves both retrieval and answer quality.

Store metadata you will want to filter on. Owner, department, last-reviewed date, access level, document type. Retrieval quality improves enormously when you can narrow the search before ranking — "only HR documents, only current versions" beats searching everything and hoping.

Return sources, always. Every answer should link the chunks it used. This is not a nice-to-have: it is the mechanism by which users catch mistakes, and it is the difference between a system people trust and one they quietly stop using.

Handle "I don't know" explicitly. Decide what happens when retrieval finds nothing relevant. The correct behaviour is to say so. The default behaviour, without instruction, is to answer anyway.

Why RAG systems fail#

Four causes, in the order we most often find them.

1. The documents are wrong. RAG faithfully retrieves your outdated 2023 policy and answers from it. The system worked perfectly; the content was stale. This is the most common failure and it is not a technical problem — it is a content ownership problem. Before building, ask who is responsible for each document being current. If the answer is "nobody", fix that first.

2. Retrieval returns the wrong chunks. The model can only answer from what it is given. If the right passage never surfaces, no amount of prompt tuning helps. Diagnose this by checking retrieval separately from generation — take twenty real questions, look at what came back, and judge whether a human given those passages could have answered. If not, the problem is upstream of the model.

3. Chunks are too small to be meaningful. Aggressive splitting produces fragments that match a query but contain no usable answer. Symptom: retrieval looks correct, answers are vague.

4. Nobody measures anything. The system launches, feels impressive in a demo, and slowly degrades as documents change. Without measurement you find out from a complaint.

Measuring whether it works#

Test the two halves separately, because they fail differently.

Retrieval quality — for a set of real questions, was the correct passage in the returned set? This is a yes/no you can count. If retrieval is at 60%, the ceiling on your whole system is 60%, and prompt work is wasted effort.

Answer quality — given correct passages, was the answer correct, complete, and grounded in them? Grounding is the critical one: an answer containing facts that appear in no retrieved chunk is a hallucination, regardless of whether it happens to be true.

Build the question set from real usage, not imagination. Twenty genuine questions teach you more than two hundred invented ones. Add every reported failure to the set permanently — that single habit does more for long-run quality than any tuning.

Our companion guide on Prompt Testing covers how to run these as an automated suite rather than a spreadsheet someone updates occasionally.

A realistic starting point#

You do not need a vector database, a framework, or a platform to begin.

Start with a few hundred documents that one team actually uses and that someone owns. Chunk on headings. Use hybrid search. Write twenty real questions with known answers before you write any retrieval code — they become your baseline, and they will tell you honestly whether the thing is working.

Scale the infrastructure when the volume demands it, not before. We have seen more projects stall from choosing a platform first than from outgrowing a simple one.

What this costs to run#

Three cost lines, and the surprising one is usually the second.

Embedding the documents is a one-off per document version, and it is cheap. Re-embedding when a document changes is also cheap.

Retrieval is nearly free at small scale and becomes an infrastructure cost at large scale — this is where a managed vector database starts earning its fee.

Generation is the recurring cost, and it scales with how much context you send, not just how many questions you ask. Sending twenty chunks instead of five costs roughly four times as much per question and often answers no better — beyond a certain point extra context dilutes rather than helps. Tuning how many chunks you retrieve is a cost lever and a quality lever at the same time.

FAQ#

Is RAG the same as a chatbot?#

No. A chatbot is an interface; RAG is a technique for grounding answers in your documents. You can build RAG into a chatbot, a search box, an internal API, or a report generator. The interface is a separate decision.

Does RAG stop hallucinations completely?#

No, and be sceptical of anyone who says it does. RAG dramatically reduces them by giving the model the right material and instructing it to stay within that material. It does not eliminate them — the model can still misread a passage or blend in prior knowledge. Showing sources is what makes the remaining errors catchable.

How current can the answers be?#

As current as your index. If a document is re-indexed within minutes of being edited, answers are minutes fresh. This is one of RAG's real advantages over fine-tuning, where updating a fact means retraining.

Do we need a vector database?#

Not to start. Below roughly the tens-of-thousands-of-chunks range, simpler storage is perfectly adequate and much easier to operate. Adopt a vector database when scale, filtering complexity or latency genuinely demand it — not as step one.

Can it respect our existing permissions?#

Yes, and this is a major reason to prefer RAG. Filter at retrieval time using the requesting user's permissions, so they can only be shown passages they were already entitled to read. Get this wrong and you have built a very efficient way to leak documents — treat it as a security requirement, not a feature.

How long does a first version take?#

For a scoped set of documents with a clear owner, a working prototype is usually days rather than months. Getting from "impressive demo" to "people rely on it" takes longer, and almost all of that time goes on document quality, permissions and measurement — not on the AI.

What if our documents are a mess?#

Then that is the project. RAG makes the state of your documentation visible very quickly, which is uncomfortable but useful. Start with one well-maintained corpus rather than pointing it at everything; a narrow system that is right beats a broad one that is unreliable.

What else is coming for RAG