How to Build a RAG Pipeline: Chunking, Embeddings, Retrieval, and Re-Ranking Explained

Overview

This article gives you a working mental model for how to build a RAG pipeline and where each decision matters.

At a high level, a RAG system does four things:

1. Turns source content into retrieval-ready chunks.
2. Converts those chunks into embeddings so similarity search is possible.
3. Retrieves a small candidate set for a user query.
4. Re-ranks and packages the best context for the generation model.

That sounds simple, but most production issues appear in the seams between those steps. A weak chunking policy can hide relevant facts. A good embedding model with poor metadata can still return noisy results. Fast retrieval without re-ranking often produces context that is adjacent to the answer rather than directly useful. And even a strong retrieval stack can fail if prompts, citations, or evaluation are treated as an afterthought.

A useful way to think about RAG is as a search system feeding a generation system. The search side is responsible for recall and relevance. The generation side is responsible for synthesis, formatting, and user experience. When teams blur those responsibilities, debugging becomes difficult. When they separate them clearly, failures become easier to trace.

Before building anything, decide what your system is meant to answer. Internal policy search, product documentation chat, research assistants, support knowledge bases, and contract review helpers all look like “RAG” on a diagram, but they have different constraints. Some need exact citations. Some need freshness. Some need long-context summarisation. Some need permission-aware retrieval. Your architecture should reflect that reality, not a generic demo.

For readers building broader LLM products, it also helps to keep framework choices separate from design choices. Whether you use a custom stack or a framework, the core design questions remain the same. If you are comparing orchestration options, see Best Open-Source LLM Frameworks Compared: LangChain vs LlamaIndex vs Haystack vs DSPy.

Step-by-step workflow

This section gives you a process you can follow, test, and improve over time.

1. Define the retrieval task before choosing tools

Start with queries, not infrastructure. Collect 20 to 50 realistic questions users will ask. Then note what a good answer requires:

• A single exact passage?
• Multiple passages from one document?
• Cross-document synthesis?
• Time-sensitive or version-specific content?
• Strict citation requirements?

This step helps you avoid a common mistake: selecting a vector database and embedding model before understanding the retrieval shape of the problem.

2. Clean and structure your source content

RAG quality depends heavily on source hygiene. Remove obvious duplicates, broken markup, navigation junk, repeated footers, and irrelevant boilerplate. Preserve meaningful structure such as headings, section titles, document IDs, publication dates, authors, and access labels. These fields become useful metadata later.

In many pipelines, raw text alone is not enough. For example, if two chunks contain similar wording but one is obsolete, metadata like version or effective date may matter more than semantic similarity. Build for that early.

3. Choose a chunking strategy that matches how answers are found

Chunking is not just a token-count exercise. It determines the retrieval unit your system can actually return.

Common chunking approaches include:

• Fixed-size chunking: Simple and predictable. Good for early baselines, but can split meaning across boundaries.
• Sliding-window chunking: Adds overlap to reduce boundary loss. Usually better than rigid fixed chunks for prose-heavy documents.
• Structure-aware chunking: Splits by headings, paragraphs, tables, list items, or document sections. Often a strong default for manuals, policies, and documentation.
• Semantic chunking: Attempts to keep coherent ideas together. Can work well, but is more complex and should be validated rather than assumed.

In practice, start with structure-aware chunking if the source has good headings; otherwise use fixed-size chunks with overlap as a baseline. Then test. If answers often require local detail, smaller chunks may improve precision. If answers depend on surrounding context, larger chunks or parent-child retrieval may work better.

Useful chunk metadata often includes document title, section heading, URL or file path, version, updated date, chunk index, and parent document ID. This makes later filtering, attribution, and debugging much easier.

4. Generate embeddings for chunks and queries

Embeddings convert text into vectors that support semantic search. The important point is consistency: if your chunk embeddings and query embeddings are not aligned to the same embedding space and retrieval design, quality will degrade.

When selecting an embedding model, focus on practical questions:

• Does it work well on your document type?
• Does it support the languages you need?
• Is latency acceptable for indexing and querying?
• Can you re-embed the corpus when models change?

Do not assume the newest or largest option is always best. If your documents are short, technical, and repetitive, indexing quality may depend more on chunk design and metadata than on chasing small embedding improvements.

Also decide how you will handle updates. If documents change frequently, you need an incremental indexing process rather than periodic full rebuilds. A stable document ID plus chunk-level hashing can help identify what actually needs re-embedding.

5. Store vectors with metadata and plan for filters

Your vector store is not just a bucket for embeddings. It is part of retrieval logic. Store enough metadata to support constraints such as product, region, document type, publish status, access permissions, or date ranges. This is especially important in enterprise and internal knowledge settings.

A practical record often includes:

• Chunk text
• Embedding vector
• Document ID
• Chunk ID
• Title and section
• Source link
• Version or freshness marker
• Security or tenant labels

Many retrieval problems that look like embedding failures are really metadata failures. If a user asks about one product line and the system returns another, semantic similarity may be working perfectly while filtering is missing.

6. Retrieve candidates with a simple baseline first

Begin with top-k vector retrieval. Keep it plain enough that you can observe what is happening. A baseline system teaches you more than a highly layered stack you cannot diagnose.

At this stage, inspect retrieved chunks for representative queries and ask:

• Are the right documents present at all?
• Are relevant passages buried below irrelevant ones?
• Are chunks too narrow or too broad?
• Do near-duplicate chunks crowd out diversity?

If recall is poor, revisit chunking, metadata, and indexing before adding complexity. Re-ranking cannot rescue documents that never enter the candidate set.

7. Add hybrid retrieval when pure semantic search misses exact terms

Many real-world applications benefit from hybrid retrieval, which combines vector search with keyword or lexical retrieval. This matters when user queries include product names, error codes, legal phrases, internal abbreviations, or exact field labels. Dense retrieval is good at meaning; lexical retrieval is often better at exact matching.

A durable pattern is to pull a candidate set from both methods, merge results, de-duplicate, and then send the combined set to a re-ranker. This often improves robustness without requiring a fully custom search engine.

8. Use re-ranking to improve the final context set

Re-ranking is the step that sorts candidate passages by likely relevance to the actual query. In many pipelines, this is where answer quality visibly improves.

Why re-ranking matters: the first retrieval stage is usually optimised for speed and recall, not perfect ordering. That means useful passages may be present but not ranked highly enough to reach the final prompt. A cross-encoder or comparable re-ranking model can compare the query and each candidate more directly, often producing a better final list.

Re-ranking is especially helpful when:

• Your documents contain many similar sections.
• Answers depend on subtle wording differences.
• Hybrid retrieval produces a broad but noisy candidate set.
• You want fewer, more precise chunks in the prompt.

Keep the pipeline disciplined: retrieve a wider candidate pool, re-rank it, then pass only the top results to the generator. This usually gives better prompt context than simply increasing top-k and hoping for the best.

9. Build the answer prompt around evidence, not guesswork

Once you have the final retrieved passages, your generation prompt should make the model's job narrow and explicit. Tell it what context is authoritative, when to cite sources, what to do if evidence is missing, and how to format the answer.

A strong RAG prompt often includes:

• The user question
• The retrieved passages
• Source labels for each passage
• Instructions to answer only from provided context when appropriate
• Rules for uncertainty, such as saying the answer is not supported by the retrieved text

If you need structured outputs for downstream systems, use a schema-first approach. For that, see JSON Prompting Guide: How to Get Structured Output Reliably From LLMs.

10. Return citations and traceability by default

Even when users do not explicitly ask for citations, traceability improves trust and debugging. At minimum, return the source title or section for each claim-bearing answer. In internal tools, a direct link back to the original document is often more valuable than a polished paragraph.

Design the interface so that users can inspect evidence quickly. This reduces overreliance on fluent responses and helps teams catch retrieval drift sooner.

Tools and handoffs

This section explains how the pieces fit together so you can assign ownership and swap components without redesigning the whole system.

A clean RAG stack usually has five handoffs:

1. Ingestion: Files, pages, tickets, policies, or database records enter the system.
2. Preparation: Content is cleaned, normalised, chunked, and enriched with metadata.
3. Indexing: Embeddings are generated and stored in a retrievable index.
4. Retrieval and ranking: Query-time logic collects, filters, and re-orders candidates.
5. Generation and response: The LLM answers using the selected context and returns citations.

Different teams often own different parts. Data engineering may handle ingestion. Platform or backend teams may own indexing and APIs. Search or ML teams may tune retrieval and re-ranking. Product teams may define prompt behaviour and UX. If these handoffs are unclear, issues are likely to bounce between teams with no clear root cause.

For example, “the bot answered incorrectly” may actually mean one of several things:

• The relevant document was never ingested.
• The chunk boundary split the answer in half.
• The embedding model grouped similar but wrong passages together.
• The retriever did not apply the right metadata filter.
• The re-ranker preferred background material over the direct answer.
• The generation prompt ignored evidence ordering.

This is why observability matters. Log the query, retrieved candidates, re-ranked candidates, chosen prompt context, final answer, and cited sources. Without that chain, RAG debugging becomes guesswork.

Frameworks can speed up prototyping, but avoid hiding core decisions behind defaults you do not understand. If you need help choosing a stack, the framework comparison linked earlier is a useful starting point. If pricing and throughput matter for generation or embedding calls, LLM API Pricing Comparison: OpenAI vs Anthropic vs Google vs Mistral can help frame trade-offs without treating cost as the only variable.

Security handoffs matter too. If your system accepts untrusted documents or open web content, retrieval can carry harmful instructions into the prompt. Treat prompt injection as part of architecture, not just prompt wording. See Prompt Injection Prevention Checklist for Chatbots, Agents, and RAG Systems for a focused treatment of that risk.

Quality checks

This section gives you the checks that make a RAG tutorial useful in practice rather than just implementable.

Evaluate each layer separately before judging the whole app.

Retrieval checks

• Hit rate: For known-answer queries, does the right document appear in the candidate set?
• Ranking quality: Does the best evidence appear near the top?
• Chunk usefulness: Are returned chunks self-contained enough to answer the question?
• Filter correctness: Are version, product, tenant, or access boundaries respected?

Generation checks

• Groundedness: Does the answer stay within retrieved evidence?
• Citation fidelity: Do cited passages actually support the claims made?
• Completeness: Does the answer combine multiple pieces of evidence when needed?
• Refusal behaviour: Does the system say it lacks support when retrieval is weak?

Create a small labelled evaluation set early. It does not need to be large to be useful. What matters is that it covers common queries, edge cases, ambiguous wording, and known failure modes. Production-ready teams usually revisit that set regularly as the corpus and user behaviour change.

For a fuller treatment of metrics and failure analysis, link your build process to RAG Evaluation Framework: Metrics, Test Sets, and Failure Analysis for Production Apps. If your application is customer-facing or content-sensitive, the broader lens in AI Output Evaluation Rubric for Marketing Teams: Accuracy, Brand Voice, and Risk is also useful, even outside marketing contexts.

One practical debugging rule: change one variable at a time. If you alter chunk size, embedding model, retrieval method, and prompt simultaneously, you will not know what improved or degraded the result. Keep a baseline. Version your indexing pipeline. Save representative queries. Treat RAG tuning like search and systems engineering, not prompt folklore.

When to revisit

This section tells you when a RAG pipeline should be updated and what to check first.

You should revisit your pipeline when any of the following changes:

• Your source corpus changes shape: New document types, table-heavy content, multilingual content, or faster update cycles often require a new chunking or metadata strategy.
• User queries change: If the product starts serving support, compliance, and research queries together, one retrieval policy may no longer fit all.
• Your models change: New embedding or re-ranking models can improve quality, but they may also require re-indexing and fresh evaluation.
• Latency or cost becomes a problem: Re-ranking and larger candidate pools improve quality, but not for free. Revisit top-k, hybrid settings, and caching before assuming the architecture is wrong.
• Failures become systematic: Repeated missed citations, stale answers, or wrong-document retrieval usually mean a pipeline issue, not isolated prompt weakness.
• Security posture changes: New data sources, external uploads, or agent-style actions should trigger a review of retrieval safety and prompt injection controls.

A practical refresh checklist looks like this:

1. Review recent failed queries and group them by failure type.
2. Check whether the right documents are present in the retrieved candidate set.
3. If not, inspect ingestion, metadata, and chunking before touching prompts.
4. If they are present but ranked poorly, test hybrid retrieval or stronger re-ranking.
5. If evidence is present and ranked well but answers are still weak, refine prompt instructions and output format.
6. Re-run your evaluation set before and after every material change.

The most durable RAG systems are not the ones with the most components. They are the ones whose owners can explain, inspect, and update each stage with confidence. If you build your pipeline around clear responsibilities—chunking for retrieval units, embeddings for semantic matching, retrieval for recall, re-ranking for precision, and prompting for grounded synthesis—you will have a system that remains understandable as tooling evolves.

If you are implementing this in a live product, make your next step concrete: choose 25 real user questions, build a simple baseline with chunking plus top-k retrieval, inspect failures manually, then add hybrid retrieval or re-ranking only where the evidence shows a need. That process will teach you more than any architecture diagram, and it will give you a RAG foundation you can return to whenever tools or models change.