RAG Tutorial: How to Build a Reliable Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation (RAG) combines two separate capabilities: a retrieval system finds relevant information from a controlled collection, and a language model uses that information to produce an answer. The model does not need to memorise every document because the application supplies relevant context at query time.

A useful RAG system is more than a vector database connected to an LLM. Its quality depends on the entire pipeline: source documents must be accurate and current; content must be divided into useful passages; search must return the right evidence; prompts must constrain how that evidence is used; and the application must show uncertainty when the answer is not supported.

Before choosing a framework or model, define the task. A support assistant, internal policy search tool, research interface, and document question-answering app may all use RAG, but they require different retrieval strategies, permissions, response formats, and evaluation criteria. Treat the following as a baseline checklist for an LLM application rather than a fixed architecture.

Core RAG architecture

1. Ingestion: collect documents and record their origin, version, owner, and access rules.
2. Pre-processing: extract text, remove avoidable noise, preserve headings and important structure, and split content into retrievable chunks.
3. Embedding: convert chunks into numerical representations for semantic search.
4. Indexing: store vectors alongside the original text and useful metadata in a vector database or compatible search system.
5. Retrieval: turn a user question into a search query and select relevant passages.
6. Generation: place the selected context into a controlled prompt and ask the model to answer within the evidence provided.
7. Evaluation and monitoring: measure retrieval quality, answer quality, latency, cost, safety, and failure patterns.

Keep retrieval and generation observable as separate stages. If an answer is wrong, you need to know whether the system found poor evidence or whether the model misused good evidence.

Checklist by scenario

For a small document collection

• Start with a simple ingestion script and a clearly defined folder or source list.
• Use stable document identifiers so an updated file replaces the correct record instead of creating a duplicate.
• Chunk by meaningful structure where possible, such as headings, sections, or paragraphs, rather than splitting every document at an arbitrary character count.
• Store metadata such as title, section heading, publication date, source URL, language, and access group.
• Test a small set of representative questions before adding more documents.

This approach is often sufficient for a prototype or an internal tool. The main risk is not scale but weak source preparation. A small collection with clear structure can outperform a larger, noisier index.

For policy, support, or operational content

• Assign an owner to each source and record when it was last reviewed.
• Preserve effective dates, version numbers, and superseded status in metadata.
• Filter retrieval by the user’s permissions before context reaches the model.
• Instruct the model to distinguish between a direct answer, an inference, and insufficient evidence.
• Return citations that identify the document and relevant section, not just a generic source label.
• Add an escalation route for questions involving exceptions, sensitive decisions, or conflicting documents.

For business use, access control is part of retrieval design. Do not rely on the model to hide information after restricted documents have already been supplied in its context.

For a large or frequently changing knowledge base

• Use an incremental indexing process instead of rebuilding everything for every update.
• Hash or otherwise identify source content so unchanged documents are not processed repeatedly.
• Define deletion and replacement behaviour for removed or superseded documents.
• Consider hybrid retrieval, combining semantic search with keyword or metadata filters when exact terms, product codes, names, or legal phrases matter.
• Use a reranking stage when initial retrieval returns too many broadly related passages.
• Track retrieval latency and index freshness alongside answer metrics.

Frequent updates make freshness a product requirement. A highly relevant answer based on an old document can still be incorrect for the user’s situation.

For a conversational application

• Separate the current user question from conversation history before searching.
• Rewrite follow-up questions into standalone search queries when the meaning depends on earlier turns.
• Limit how much history is placed into the final generation prompt.
• Test ambiguous references such as “that policy”, “the previous version”, or “what about Scotland?”
• Make it clear whether the answer is based on retrieved documents, the conversation, or general model knowledge.

Conversation history can improve usability but also introduce retrieval noise. Search should focus on the user’s actual information need, not every word exchanged in the session.

What to double-check

Chunking and metadata

Chunk size should preserve enough context to make a passage understandable while remaining focused enough for search. There is no universal ideal size. Compare several strategies using real questions, and inspect the returned passages manually. Add overlap only when it helps preserve meaning across boundaries; excessive overlap can create duplicate evidence and waste context.

Metadata often determines whether retrieval is usable in production. Include fields that support filtering, citation, access control, and freshness checks. Keep the original text available so developers can inspect exactly what the model received.

Retrieval quality

Create a test set containing common questions, difficult questions, questions with exact names or identifiers, and questions that should produce “I cannot find enough information”. For each query, record which document or passage should be retrieved. This lets you evaluate search independently from the generated response.

Check for false positives as well as missed results. Returning loosely related passages may encourage a confident but unsupported answer. If search results are weak, improve source cleaning, query rewriting, metadata filters, hybrid search, or reranking before changing the generation prompt.

Prompt construction

A reliable generation prompt should define the assistant’s task, identify the retrieved context, explain how to handle missing evidence, and specify the required answer format. For example, instruct the model to answer only from the supplied context for factual claims, cite the supporting source, and state when the context does not resolve the question.

Keep instructions separate from retrieved text. Treat documents as data, not as instructions. This reduces the risk that text inside a document changes the application’s intended behaviour. For structured outputs, validate the model’s response in application code and handle invalid output safely.

Citations and user experience

Citations should be traceable. A user should be able to identify the source, section, and version behind an answer. Avoid presenting citations as proof if the cited passage does not actually support the claim. When evidence conflicts, show the conflict or explain which source was prioritised and why.

Design for uncertainty. A useful response may be a short explanation that no reliable match was found, followed by a request for clarification or a link to a human support route.

Evaluation and operations

Measure at least four dimensions: whether relevant evidence was retrieved, whether the answer is supported by that evidence, whether the response follows the required format, and whether the system meets acceptable latency and cost targets. Add tests for permissions, prompt injection, malformed documents, empty results, duplicate chunks, and service failures.

For a broader production checklist, see LLM App Development Checklist: From Prototype to Production. You can also use the guidance in How to Measure AI Chatbot Performance when turning evaluation results into an ongoing reporting process.

Common mistakes

• Indexing raw files without inspection: headers, navigation, repeated footers, tables, and OCR errors can pollute search results.
• Assuming semantic search solves exact matching: identifiers, part numbers, names, and quoted phrases may need keyword search or filters.
• Sending too much context: a large prompt can dilute the relevant evidence and increase processing overhead.
• Using retrieval as a guarantee of truth: retrieved content may be outdated, incomplete, duplicated, or wrong.
• Skipping access controls: document permissions must be enforced before retrieval results are passed to the model.
• Testing only easy questions: production failures often appear with ambiguity, missing information, conflicting sources, or follow-up questions.
• Updating the index without a rollback plan: keep versions or an audit trail so a faulty ingestion run can be identified and reversed.
• Hiding uncertainty: forcing an answer when evidence is absent creates a worse user experience than a clear limitation.

For practical techniques to reduce unsupported answers, see How to Reduce Hallucinations in LLM Apps. If your RAG app handles regulated, personal, or sensitive information, review the relevant governance requirements for your operating environment rather than treating this technical checklist as legal advice.

When to revisit

Review the RAG pipeline before seasonal planning cycles, major content releases, changes to business processes, or migrations between models and search services. Revisit it whenever users report an answer that cites the wrong version, misses a known document, exposes information from the wrong access group, or gives a confident response to an unanswerable question.

Use this maintenance routine:

1. Review source ownership, freshness, duplicates, and retired documents.
2. Run the evaluation set and compare retrieval and answer failures separately.
3. Inspect a sample of citations for traceability and factual support.
4. Check permission filters and adversarial inputs after application or schema changes.
5. Review latency, token usage, error rates, and user feedback.
6. Record changes to chunking, embeddings, prompts, retrieval settings, and model versions.
7. Roll out changes gradually and retain a way to compare the new pipeline with the previous one.

Start with a narrow use case, measurable questions, and a dependable source collection. Then expand only when the retrieval evidence, response behaviour, and maintenance process are strong enough to support the next scenario.