RAG architecture: best practices

Retrieval-Augmented Generation (RAG) has become the go-to pattern for building LLM applications that need access to up-to-date or domain-specific information. Rather than relying entirely on the model's memory, you fetch relevant information at query time.

Why RAG?

Large language models are impressive, but they have concrete limitations. Their knowledge stops at the training cutoff. They can generate plausible but wrong information (hallucinations). And they know nothing about your internal data.

RAG solves this by retrieving relevant information from your own sources before generating a response. The model stops guessing: it works from facts.

Core components

Document processing

Transform your documents into searchable chunks. Fixed-size chunking (256-512 tokens with 10-20% overlap) is a good starting point, but semantic chunking gives better results: it splits at natural text boundaries (paragraphs, sections) rather than at arbitrary token counts. Store metadata (source, date, section): it enables effective filtering at retrieval time.

For embeddings, models like BGE-M3 (multi-retrieval: dense + sparse + multi-vector) or Qwen3-Embedding offer strong quality across languages. BGE-large and E5 remain solid production choices for single-language use cases. Version your embeddings: when you switch models, everything needs re-indexing.

Vector database

The choice depends on your constraints:

If you already use PostgreSQL, pgvector lets your vectors live alongside your relational data without adding infrastructure. Qdrant is a fast on-premise option for teams that want a dedicated vector database with full control. Weaviate offers open-source hybrid search with a good balance between control and features. For prototyping, Chroma is the simplest embedded option. And if you don't want to host anything, Pinecone is managed and scales automatically.

Retrieval strategy

Semantic search alone isn't enough. Combine it with keyword search (hybrid search). Add reranking with cross-encoders to improve relevance. Avoid redundant chunks by favoring result diversity. And use metadata to filter based on your business logic.

Advanced patterns

Iterative retrieval

For complex queries, a single retrieval pass isn't enough. You start by retrieving broad context, then decompose the query into sub-questions, retrieve targeted information for each, and synthesize the results. More expensive in API calls, but significantly more accurate.

Self-querying

Rather than using the user's raw question as a search query, let the LLM generate a structured query. It can extract metadata filters, reformulate the question, and produce a more relevant search.

HyDE (hypothetical document embeddings)

A counterintuitive technique: you ask the LLM to generate a hypothetical answer to the question, then use that answer as the search query. The hypothetical answer's embedding is often closer to the actual relevant documents than the original question's embedding.

Common challenges

Relevance vs coverage

Top search results can all be similar. Use Maximum Marginal Relevance (MMR) to force diversity in retrieved results.

Long context

Too many chunks exceed the model's context window. Summarize less relevant chunks, use hierarchical retrieval, or implement context compression.

Data freshness

Information in the vector database becomes stale. Set up incremental updates, filter by date, and schedule periodic full re-indexing.

Cost control

Embeddings and LLM calls add up. Semantic caching (caching responses for semantically similar queries, not just identical ones) drastically cuts costs and latency. Batch your embedding generation and use smaller models when the task allows it. An upstream classifier can also bypass RAG entirely on simple questions the LLM already knows how to handle.

In production

What to monitor

Four metrics to track: retrieval precision (are the chunks relevant?), answer quality (via user feedback), end-to-end latency, and cost per query (embeddings + LLM).

Testing

Build a test suite with cases that validate both the relevance of retrieved sources and the content of generated answers. Frameworks like RAGAS automate this by measuring faithfulness (does the answer stick to the sources?), context precision, and answer relevance. Integrate these metrics into your CI: without automated evaluation, you're flying blind.

Optimization

Cache embeddings for static documents and results for frequent queries. Parallelize retrieval and generation where possible, and group similar queries to reduce redundant work. Every millisecond matters when your users are waiting for a response.

Beyond basic RAG

RAG is evolving fast. GraphRAG adds entity relationships for multi-hop queries (useful when your data has strong relational structure). Agentic RAG goes further: the agent decides when and how to retrieve, critiques its own results, and re-runs the search if needed. Single-pass "basic" RAG is starting to show its limits on complex cases.

On the security side, think about PII redaction before indexing and guards against prompt injection in source documents. Your chunks are uncontrolled content injected into the prompt: treat them like user input.

Ready to retrieve?

Detective searching

If you're building AI agents that use RAG, my article on AI agents covers the fundamentals. And for infrastructure, I detail the options in self-hosting AI.