Grounding AI in Your Data: RAG

A base model knows nothing about your handbook, your tickets, or last week's numbers. Retrieval-augmented generation fixes that without retraining: when a question arrives, you find the most relevant pieces of your own data and put them in the prompt, so the model answers from your reality. It's Level 2's context engineering, automated and at scale.

The machinery has three moving parts. You chunk your documents and turn each chunk into an embedding — a list of numbers that captures its meaning — and store those in a vector database. At query time you embed the question and search for the chunks closest in meaning. Then you put those chunks into the prompt alongside the question and ask the model to answer from them, citing which chunk each claim came from.

Chunking is the decision that quietly determines whether any of this works, and it gets far less attention than it deserves. Chunks that are too small lose the context that makes them meaningful; too large and the embedding blurs several topics together so it matches nothing well. A few hundred tokens with a modest overlap is a reasonable default, but the real rule is to split on structure — sections, headings, logical units — rather than blindly every N characters. Splitting a table down the middle, or severing a clause from the heading that gives it meaning, produces chunks that retrieve badly no matter how good your model is. Keep a title or breadcrumb on each chunk so the model knows where it came from.

Pure vector search has a well-known weakness: it finds things that are semantically similar, which is not the same as things that contain the exact term you need. Search for an error code, a product SKU or a surname and semantic similarity can sail right past the one chunk that mentions it. The standard fix is hybrid search — combine vector similarity with old-fashioned keyword search and merge the results. Then re-rank: retrieve twenty candidates cheaply, and use a re-ranking model to pick the best five to actually send. Hybrid retrieval plus re-ranking is usually a bigger quality win than upgrading the generation model.

Two failure modes decide whether the system is trustworthy. The first is retrieval quality: if the search pulls the wrong chunks, the model answers confidently from the wrong material, and no amount of prompt tuning saves it. The second is grounding discipline: you have to instruct the model to answer only from the retrieved material and to say it doesn't know when the answer isn't there. Without that instruction it falls back on training data and produces something plausible, which is the worst outcome because it looks identical to a good answer.

When RAG gives a bad answer, diagnose in that order. Look at what was retrieved before you touch the prompt. Most of the time the model was working faithfully from chunks that simply didn't contain the answer — which is a chunking or retrieval problem, not a generation one. Teams routinely spend weeks tuning prompts to fix what was a retrieval bug all along.

Always show sources. Citations let a human verify in seconds, which is the Level 2 habit made structural, and they're also your own best debugging tool — a wrong answer with visible sources tells you immediately whether retrieval or generation failed. Keep the metadata: document, section, date. "According to the 2024 policy" is a very different answer from "according to the 2019 one", and freshness is invisible unless you carry it through.

Two practical cautions. Permissions must be enforced at retrieval, not by asking the model nicely — if a user shouldn't see a document, it must never enter their context, because anything in the context can be surfaced. And your index needs a refresh story: documents change, and a system confidently quoting a policy that was replaced last quarter is worse than one that admits ignorance.

You don't have to build every piece. Vector databases like Pinecone, Weaviate, Qdrant and Chroma, frameworks like LlamaIndex and LangChain, and turnkey products like NotebookLM each handle parts of this. The concepts are identical whether you assemble them yourself or buy them — and knowing them is what lets you work out why a "chat with your docs" feature is giving bad answers instead of just shrugging at it.