A retrieval-augmented generation system is not finished when its first index completes. This recurring checklist helps you keep a RAG knowledge base accurate by tracking source ownership, document freshness, duplicates, permissions, indexing health, retrieval quality, and obsolete content. Use it after content changes, during monthly checks, and as a deeper quarterly knowledge-base audit.
Overview
RAG connects an LLM to an external collection of documents so that responses can be grounded in retrieved context. The quality of those answers depends on more than the model and the prompt. If the underlying collection contains an old policy, duplicate pages, incomplete permissions metadata, or failed indexing jobs, the application may retrieve information that is technically available but no longer trustworthy.
RAG knowledge base maintenance is therefore a data and operations responsibility, not just a vector database task. A reliable process should answer five questions:
- Which sources are included, and who owns each one?
- When was each source last checked or changed?
- Did updates, deletions, or permission changes reach the index?
- Are retrieval results relevant, current, and appropriately scoped?
- Can an operator explain, correct, and verify a poor answer?
Keep a simple maintenance record alongside the application. Record the source URL or system identifier, document title, owner, access classification, last successful crawl, content version or modified date, chunking and embedding configuration, index status, and the date of the next review. This inventory makes failures easier to investigate and prevents important sources from becoming invisible after launch.
If you are still designing the application, the RAG tutorial on reliable citations provides useful implementation context. Maintenance should extend the same grounding and citation principles into an ongoing operational routine.
What to track
1. Source ownership and scope
Every source should have a named business or technical owner. The owner does not necessarily operate the pipeline, but should be able to confirm whether the content is accurate, replace an obsolete document, and answer questions about its intended audience. Also record whether the source is authoritative, supplementary, or temporary.
Useful source inventory fields include:
| Field | What to record |
|---|---|
| Source | System, site, repository, file set, or page pattern |
| Owner | Person or team responsible for accuracy |
| Scope | Products, departments, regions, or users covered |
| Freshness rule | Expected review interval or event that triggers an update |
| Access class | Who may retrieve the content |
| Last indexed | Time and status of the latest successful indexing run |
| Next review | Scheduled manual or automated check |
2. Freshness, changes, and deletions
Track both the source's modification date and the date it entered the index. These are different signals. A page can change without a successful indexing job, or an index can be rebuilt from a stale export. For websites and repositories, test how the pipeline detects changed pages, renamed files, removed files, and redirects. Deletion handling deserves particular attention: removing a document from the source does not automatically remove its chunks from every index.
Do not rely only on a global “last updated” timestamp. Review a sample of high-value documents and confirm that the indexed text reflects the current source. Where documents have formal versions, store the version identifier in metadata and, where appropriate, expose it in citations or internal diagnostics.
3. Duplicates, conflicts, and metadata
Duplicate content can crowd retrieval results and make conflicting instructions harder to resolve. Look for repeated documents, copied pages, old exports, and multiple versions of the same policy. A practical duplicate check can compare canonical source identifiers, hashes of normalised text, titles, and near-duplicate similarity.
Review metadata as carefully as the text itself. Filters for department, product, region, publication status, document type, and access permissions must be populated consistently. Missing or incorrectly inherited metadata can cause a response to use a document that the user should not see, or prevent a relevant document from being retrieved.
4. Retrieval and answer behaviour
Maintain a small evaluation set of representative questions. Include common requests, questions about recently changed documents, deliberately ambiguous queries, permission-sensitive requests, and questions whose answer should be “not found”. For each test, record expected sources, acceptable answer points, citation requirements, and whether escalation is needed.
Monitor retrieval separately from generation. Inspect the top results, similarity or relevance scores where available, applied filters, and the final context passed to the model. If the right document is absent from context, changing the prompt will not fix the underlying retrieval problem. For prompt changes and repeatable evaluation, see the prompt evaluation framework.
Cadence and checkpoints
Use event-driven checks for urgent changes and a recurring schedule for routine control. The exact interval should reflect how quickly the source changes and the consequences of an incorrect answer.
| Cadence | Checkpoint |
|---|---|
| Every indexing run | Check job completion, document counts, failed items, deletion handling, and unexpected changes in chunk totals. |
| Weekly or operationally appropriate | Review failed crawls, ingestion warnings, permission errors, latency anomalies, and a small sample of recent updates. |
| Monthly | Run representative retrieval tests, inspect high-value sources, check duplicate growth, and confirm source owners are still correct. |
| Quarterly | Audit the full source inventory, review obsolete collections, reassess chunking and embedding choices, test access boundaries, and examine unresolved incidents. |
| After a major change | Re-index affected collections, rerun regression questions, compare retrieval results, and retain a record of the change and outcome. |
Re-indexing checklist
- Identify the changed source set and confirm its owner.
- Take note of the current index version or deployment state.
- Confirm the ingestion job can detect additions, edits, renames, and deletions.
- Validate text extraction, encoding, metadata, and access fields before embedding.
- Run the smallest safe re-index scope first, unless a full rebuild is required.
- Check document and chunk counts against reasonable expectations.
- Run retrieval tests for changed content and known neighbouring content.
- Compare citations and answer behaviour with the previous version.
- Record the index version, configuration, failures, and approval to release.
Embedding models, vector database features, provider limits, and indexing APIs can change over time. Treat these as configuration dependencies: review them during technical maintenance rather than assuming an unchanged pipeline will behave identically forever. A provider or model move should follow a controlled migration process; the LLM API migration checklist covers related release-control practices.
How to interpret changes
Not every metric change indicates a defect. A rise in chunk count may reflect legitimate content growth, while a fall may indicate a deletion or an extraction failure. Interpret signals together and compare them with the source system.
When answers are stale
First, identify the exact answer and the source the user expected. Check the source's current text, its indexed copy, the ingestion timestamp, and the retrieved chunks. If the current source was never indexed, investigate change detection or job failure. If the indexed copy is current but the wrong chunk is retrieved, inspect chunk boundaries, metadata filters, query rewriting, and ranking. If the right context is present but the answer ignores it, evaluate the prompt, context length, and model behaviour.
When answers conflict
Find all retrieved documents that make competing claims. Check publication dates, version metadata, ownership, and whether an older copy remains searchable. Prefer a single authoritative source where possible. Otherwise, make the conflict explicit in the application: ask for clarification, cite both versions with dates, or route the case to a human reviewer. A human-in-the-loop workflow is useful for high-impact or unresolved cases.
When retrieval quality declines
Compare recent results with a fixed test set rather than relying on isolated user reports. Segment failures by source, query type, language, metadata filter, and document format. Possible causes include collection growth, duplicated content, a changed embedding configuration, poor extraction, overly broad queries, or an altered ranking step. Correct one variable at a time where practical, then rerun the same tests.
When to revisit
Use this checklist at least monthly for active knowledge bases and conduct a deeper audit quarterly, adjusting the interval to the speed and risk of content change. Revisit immediately after a policy release, product update, repository migration, permission-model change, embedding or vector-store change, model change, or incident involving an incorrect answer.
At each review, do not only confirm that jobs are green. Select a few important user questions, inspect the retrieved context, verify citations against the source, and confirm that removed or restricted content cannot be returned. Record findings, owners, deadlines, and the next review date.
A practical next step is to create three artefacts: a source inventory, a freshness and indexing log, and a small retrieval regression set. Start with the sources that matter most, establish a baseline, and expand coverage as failures reveal new cases. This turns RAG knowledge base maintenance from an occasional emergency into a repeatable part of reliable AI development.
