# RAG in Production: The Boring Parts That Matter

URL: https://donedeploy.com/rag-in-production/
Published: 2026-09-16

Retrieval augmented generation (RAG) is simple to demonstrate. Load some documents into a vector database, retrieve the closest chunks, put them in a prompt, and the model answers questions about your data. The demo usually works. Production is where it gets hard, and the hard parts are not the model. They are ingestion, freshness, evaluation and permissions.

## Ingestion is a data pipeline

Your sources are PDFs, wiki pages, tickets, spreadsheets and emails, and each has its own mess: headers and footers repeated on every page, tables flattened into nonsense, scanned images with no text layer. Treat ingestion as a proper pipeline:

- Use a parser suited to each format, and check its output by eye on a sample before trusting it.
- Keep source metadata with every chunk: document ID, title, URL, section, last modified date and owner.
- Make ingestion idempotent, so running it twice produces the same index rather than duplicates.
- Log what failed to parse. Silent failures become missing answers that nobody can explain.

## Chunking deserves thought

How you split documents affects retrieval more than most model choices. A few principles hold up well:

- Split on structure (headings, sections, list items) before splitting on length.
- Keep chunks large enough to carry meaning and small enough to stay on one topic. Test a few sizes against your questions.
- Add a little overlap, or prepend the document and section title to each chunk, so context is not lost at boundaries.
- Keep tables together where possible, or convert them into a form the model can read.

## Retrieval is more than vectors

Pure vector search misses exact terms such as product codes, error messages and names. Hybrid search, combining keyword search (BM25) with embeddings, is a strong default. A reranking step on the top results often improves relevance further. PostgreSQL with pgvector, OpenSearch, Elasticsearch, Qdrant and Weaviate all support some form of this, so you rarely need a separate system just for vectors.

## Keep the index fresh

Stale answers destroy trust quickly. Decide how fresh each source must be, then build to that:

1. Incremental updates triggered by change events or a modified timestamp, not full rebuilds every night.
2. Deletions handled explicitly. A document removed from the source must leave the index.
3. Versioned embeddings. If you change the embedding model, re-embed everything into a new index and switch over, because vectors from different models are not comparable.

## Access control at retrieval time

This is the part most often skipped, and the most dangerous. If a user cannot open a document in the source system, the RAG system must not quote it to them.

- Store permissions (groups, users, tenants) as metadata on each chunk.
- Filter by the requesting user’s permissions inside the retrieval query, before anything reaches the model.
- Sync permission changes as carefully as content changes.

Asking the model to “only use documents the user may see” is not access control. Filtering has to happen in code.

## Evaluate retrieval and answers separately

When an answer is wrong, you need to know whether retrieval found the wrong chunks or the model misused the right ones. Build a test set of real questions with the documents that should answer them, then measure:

Stage What to check Retrieval Did the right chunks appear in the top results? Generation Is the answer supported by the retrieved text? Does it cite sources? End to end Would a domain expert accept the answer?

Tools such as Ragas and other open evaluation frameworks can automate parts of this, including LLM-as-judge checks, but keep humans reviewing a sample. Rerun the evaluation whenever you change chunking, embeddings, retrieval settings, prompts or the model.

## Show your sources

Return citations with every answer and link back to the original document. It lets users verify answers, it exposes retrieval problems quickly, and it builds the trust that keeps people using the tool.

DoneDeploy builds the ingestion pipelines, search infrastructure and evaluation around RAG systems, with access control designed in from the start. If you have a RAG prototype that needs to become dependable, [talk to us](https://donedeploy.com/lets-talk/).