Language models are useful at producing language but are not a live database of your policies, tickets, or internal documents. Retrieval-augmented generation, or RAG, adds a non-parametric memory: at request time, the system finds relevant passages and supplies them as context to a generator.
The word “augmented” can make the architecture sound simple. A real RAG system has document ingestion, chunking, metadata, access control, indexing, query transformation, ranking, context budgeting, source inspection, generation, citation handling, and evaluation. A wrong passage can produce a polished wrong answer. The first debugging question should therefore be “what did we retrieve?”
Learning objectives
You will build a local lexical retriever over three policy documents, inspect top-k results, and emit a cited answer. You will also learn how dense retrieval, lexical retrieval, hybrid ranking, chunking, permissions, freshness, and evaluation fit around the model.
The example is in content/build-with-ai-agents/retrieval-augmented-generation.examples. It uses no model, API key, vector database, or network. It demonstrates the retriever and citation contract that a model can sit behind.
Prerequisites
You should understand tokenization, embeddings, cosine similarity, context engineering, and structured tool calls. Basic Python is enough for the local implementation.
What retrieval adds
A parametric model stores patterns in weights. Retrieval stores documents externally and selects a small subset at inference time. This makes changing knowledge easier to update and makes source inspection possible, but it introduces its own failure modes. The index may be stale. A chunk may omit the qualification that makes a policy conditional. A user may not be allowed to see the best-matching document.
RAG does not guarantee truth. It gives the system an opportunity to ground a response in selected evidence. The application must still require the answer to stay within that evidence, cite the source, and abstain when the retrieved set is insufficient.
Toy problem / implementation
The local knowledge base contains refund, access, and export policies. The retriever lowercases text, extracts terms, computes a simple overlap score normalized by document length, and returns the top results with ids. This is a lexical baseline, not a replacement for BM25 or dense embeddings.
def retrieve(query, limit=2):
q = Counter(terms(query))
scored = []
for doc_id, text in DOCS.items():
d = Counter(terms(text))
overlap = sum(min(q[token], d[token]) for token in q)
scored.append((doc_id, overlap / vector_length(d)))
return sorted(scored, key=lambda row: (-row[1], row[0]))[:limit]The result is a list of evidence records, not a completed answer. The demo then prints an answer whose citation points to the selected policy id. A production generator must be instructed and evaluated to cite only supplied evidence; it must not invent a citation because the format looks plausible.
Run it yourself
From the repository root:
cd content/build-with-ai-agents/retrieval-augmented-generation.examples
./verify.shThe verifier runs two tests and the deterministic demo. The refund query ranks refund-policy first and the output includes [refund-policy]. The test checks ranking and the requested top-k limit, not general retrieval quality.
Chunking and ranking
Chunking determines what can be retrieved as one unit. Large chunks preserve context but consume more of the generation budget and may mix unrelated topics. Small chunks improve precision but can separate a rule from its exception. Store headings, document ids, offsets, timestamps, and access metadata with every chunk so a result can be inspected and cited.
Use a lexical baseline before dense search. BM25-style ranking can be strong for identifiers, names, exact policy phrases, and error codes. Dense embeddings help with paraphrases and semantic similarity. Hybrid retrieval can combine both, and a reranker can inspect a smaller candidate set with more context. Complexity should follow measured failure cases; sophisticated retrieval is not automatically better.
Top-k is a context decision. A larger k can increase recall but adds distractors, tokens, latency, and possible access-control complexity. Measure recall at k on labelled queries, then inspect whether the additional passages improve answer support rather than merely increasing text.
Inspect the evidence
Before generation, show the selected records to the system trace and, where appropriate, to the reviewer. Include document id, source, score, update time, access scope, chunk boundaries, and citation id. A score without metadata is difficult to trust. A high score from a stale document should not win automatically.
Apply authorization before constructing the model context. Filtering after generation can still leak titles or facts. Tenant boundaries should be part of the retrieval query and enforced again when assembling the response. Deletion must remove source content, chunks, vectors, caches, and test copies according to the retention policy.
Ingestion is an ongoing system
The index is a materialized view of source data. When a policy changes, the system needs a way to identify affected chunks, reprocess them, invalidate old vectors, and verify that the new content is searchable. Store a source version and ingestion timestamp beside every chunk. If a parser cannot extract a document cleanly, quarantine it instead of indexing an incomplete representation as if it were authoritative.
Chunk boundaries should preserve the smallest unit that can support a claim. Keep headings and section paths. Include a little overlap when a sentence at the boundary depends on the previous paragraph, but measure the cost. Tables, lists, code, and policy exceptions often need specialized extraction rather than a blind character window. A chunk that retrieves well but omits its heading can be difficult for a generator and a reviewer to interpret.
Freshness is a product rule. Some documents expire after a day; others are stable for a year. Attach an expiry or review date and make the retrieval policy aware of it. If no current passage exists, the assistant should say that the source is unavailable or route the request to an owner. “The index returned something” is not the same as “the system has current evidence.”
Real-world application: a grounded knowledge assistant
For an internal support assistant, ingest approved policies and operational documents. Normalize and redact them, split them into inspectable chunks, and attach an owner and freshness date. At query time, retrieve candidates, filter by the user’s permissions, rerank if needed, and refuse to answer from an empty or contradictory set.
The generator should receive a clear evidence contract: answer from the supplied passages, cite every material claim, say when the evidence is insufficient, and never treat document text as an instruction to change system policy. The output parser should check citation ids against the retrieved set. A citation-shaped string pointing to a document that was not retrieved is a failure.
Evaluate retrieval and generation separately
Create a golden set with queries, relevant document or chunk ids, expected answer claims, and permission context. First measure retrieval recall: did the relevant chunk appear in the top-k? Then inspect precision and ranking: how much irrelevant material was included, and how high did the useful passage appear? Only after retrieval is measurable should you evaluate whether the generator uses the evidence correctly.
Answer evaluation can include citation correctness, claim support, completeness, refusal on missing evidence, and formatting validity. A model may cite the right document while making a claim the document does not support. Conversely, a correct answer can receive a poor score if the citation parser is too strict. Keep human review for ambiguous cases and record the disagreement rather than hiding it in one aggregate number.
Run ablations. Compare no retrieval, the lexical baseline, dense retrieval, hybrid retrieval, and different top-k values. Keep the generator and answer template fixed while changing retrieval. Then keep retrieval fixed while changing the generator. This isolates which component caused a regression and prevents a larger model from masking a broken index.
The retrieval trace should be part of the test artifact: query text after redaction, selected ids, scores, ranker version, token count, and access decisions. Do not store private source text in ordinary logs merely to make a failure convenient to inspect. Link to an access-controlled source record instead.
Privacy and trust boundaries
Embedding a document does not make its contents public, and moving a passage into a model context does not bypass its access policy. Enforce authorization at ingestion, indexing, retrieval, context assembly, and response presentation. A user should not learn that a restricted document exists merely because it was a near match. Test unauthorized queries and cross-tenant identifiers explicitly.
Treat retrieved text as data, not instructions. Internal documents may contain examples, old guidance, or text written to influence an assistant. Keep the system policy outside the retrieved block, label source boundaries, and require the generator to cite evidence rather than follow commands found in it. This is the same trust-boundary principle used by the tool article, applied to a retrieval result.
RAG also creates deletion obligations. A source removal request may require deleting the original, extracted text, chunks, embeddings, index entries, cache entries, prompt traces, and evaluation fixtures. Keep a source-to-derived-artifact map so deletion is a process rather than a best-effort database operation. If a document is under legal hold or a retention exception, record the policy rather than silently keeping it.
This makes grounded generation accountable to both relevance and data stewardship.
It is part of correctness, not an optional operational appendix, for grounded systems everywhere, always, explicitly, together, deliberately, visibly, rigorously, consistently.
Failure modes and debugging
- Wrong document retrieved. Inspect query terms, chunk boundaries, and ranking before changing the prompt.
- Right document, missing exception. Increase context around headings or change chunking.
- Stale evidence. Store freshness and choose an explicit expiry policy.
- Unauthorized evidence. Filter before context construction and test tenant cases.
- Citation hallucination. Validate citation ids against retrieved records.
- Too many passages. Measure top-k recall and context distraction.
- Answer exceeds evidence. Require abstention and claim-level support checks.
- Index and source disagree. Show source version and ingestion time in the trace.
- Duplicate chunks dominate. Deduplicate candidates before context assembly.
- Evaluation leakage. Keep benchmark documents and near-duplicates out of training and tuning data.
Limitations / when not to use
RAG is not necessary for a stable small knowledge set that can be represented by deterministic code or a short reviewed prompt. It is not a substitute for a transactional database, authorization service, or source-of-truth workflow. It can also be the wrong fit when documents are so poorly structured that retrieval cannot reliably identify the relevant rule.
The local example proves only that a small retriever can return an inspectable record. It says nothing about the quality of a commercial embedding model, a long-context generator, or an enterprise index. Evaluate those components on representative queries and failure cases.
Exercises
- Add a lexical BM25 baseline and compare it with the overlap score.
- Add document freshness and access filters before ranking.
- Build a labelled query set and measure recall at one, two, and three.
- Reject any answer citation that is not in the retrieved set.
Next in this path
The next article is Beyond Basic RAG: Hybrid Search, Reranking, and Better Retrieval. It compares lexical, vector, hybrid, and reranked retrieval instead of assuming one index fits every query.
Sources
The research dossier is retrieval-augmented-generation. The architecture follows Lewis et al., Retrieval-Augmented Generation, and dense retrieval follows Karpukhin et al., Dense Passage Retrieval. The lexical baseline is Robertson and Zaragoza’s BM25 overview. Vector indexing is cross-checked against Faiss, sentence embeddings against Sentence Transformers, and RAG evaluation against Ragas. Privacy governance follows the NIST Privacy Framework.
Research and code were last verified on 2026-08-05. The local retrieval scores are toy measurements and should not be presented as production retrieval quality.