RAG: what actually works in production

RAG (retrieval-augmented generation) has become the default pattern whenever an LLM needs to know something beyond what it learned during training. Instead of betting everything on the model's memory, you fetch the information at query time.

Why RAG

LLMs have a training cutoff. They invent facts with total confidence (hallucinations). And they know nothing about your internal data. RAG fixes all three by pulling relevant information from your own sources before generating a response. The model stops guessing, it works from facts.

Processing your documents

Document processing

You need to break documents into searchable chunks. Fixed-size chunking (256-512 tokens, 10-20% overlap) is a decent starting point. Semantic chunking does better: it cuts at natural text boundaries, paragraphs or sections, instead of an arbitrary token count. Store metadata, source, date, section. It's what lets you filter at retrieval time.

For embeddings, models like BGE-M3 (dense + sparse + multi-vector) or Qwen3-Embedding offer strong quality on multilingual projects. BGE-large and E5 hold up fine for single-language work. One thing not to skip: version your embeddings. Switching models means re-indexing everything.

Vector database

It depends on your constraints.

• pgvector: a Postgres extension. If you're already on Postgres, your vectors live next to your relational data, no new piece of infrastructure
• Qdrant: fast, on-premise. For anyone who wants a dedicated vector database with full control
• Weaviate: open source, hybrid search built in. Good balance of control and features
• Chroma: simple, embedded. For prototyping fast
• Pinecone: managed, scales on its own. The choice if you don't want to host anything

Retrieval strategy

Semantic search alone misses results that are obvious keyword matches. Combine it with lexical search (hybrid). Reranking with cross-encoders then fixes the ordering. For diversity, drop redundant chunks. And metadata filters on your business logic.

Advanced patterns

Iterative retrieval

For a complex query, one retrieval pass isn't enough. You pull broad context first, break the question into sub-questions, retrieve for each, then synthesize. More calls, but noticeably more accurate.

Self-querying

Rather than sending the user's raw question as the search query, you let the LLM build the query. It can pull out metadata filters, reformulate, and the search lands closer to the mark.

HyDE, or searching with an answer that doesn't exist

Counterintuitive, but it works: ask the LLM for a hypothetical answer to the question, then search with that answer's embedding instead of the question's. The hypothetical answer's embedding often lands closer to the real relevant documents than the question ever would.

The problems that show up everywhere

Relevance versus coverage

The best search results all look alike. Maximum Marginal Relevance (MMR) forces diversity into what gets retrieved.

Context running too long

Too many chunks blow past the model's context window. Summarize the secondary chunks, move to hierarchical retrieval, or compress the context.

Data going stale

The vector database ages. Incremental updates, date filtering, a scheduled full re-index: you need all three.

Costs creeping up

Embeddings and LLM calls add up fast. Semantic caching, caching responses for semantically similar queries and not just identical ones, cuts cost and latency in one move. Batch your embedding generation. Use a smaller model when the task allows it. An upstream classifier can also skip RAG entirely on simple questions the LLM already knows how to answer.

In production

What to monitor

Four metrics worth tracking: retrieval precision (are the chunks relevant?), answer quality (user feedback), end-to-end latency, cost per query.

Testing

For testing, you need a suite that checks both the retrieved sources and the generated content. RAGAS automates this by measuring faithfulness (does the answer stick to the sources?), context precision, and answer relevance. Skip that in your CI and you're flying blind.

Optimization

On optimization: cache embeddings for static documents and results for frequent queries. Parallelize retrieval and generation. Batch similar queries together. Every millisecond matters to a user who's waiting.

Beyond basic RAG

RAG moves fast. GraphRAG adds entity relationships for multi-hop queries, useful when your data has real relational structure. Agentic RAG goes further: the agent decides on its own when and how to retrieve, critiques its own results, reruns the search if it needs to. Single-pass basic RAG is starting to show its limits on complex cases.

On security: think about PII redaction before indexing, and guards against prompt injection in source documents. Your chunks are uncontrolled content injected into the prompt. Treat them like user input.

Ready to retrieve?

Detective searching

If you're building AI agents that use RAG, my article on AI agents covers the fundamentals. And for infrastructure, I go through the options in self-hosting AI.