RAG

Retrieval-Augmented Generation (RAG) is an architecture that combines an information retrieval system with a language model to ground responses in specific source documents. Instead of relying solely on knowledge encoded in its parameters during training, the model receives relevant passages fetched from an external corpus at inference time, then generates a response grounded in that retrieved evidence. RAG directly addresses the two most critical limitations of standalone language models: hallucination (generating plausible but incorrect information) and knowledge cutoff (lacking information that post-dates training data).

The concept was introduced by Meta AI researchers Patrick Lewis et al. in a 2020 paper, but it became the dominant enterprise AI architecture in 2023-2024 as organizations discovered that simply prompting a foundation model was insufficient for production use cases requiring factual accuracy over proprietary data. Today, RAG is the default architecture for enterprise AI deployments — Microsoft estimates that over 80% of enterprise GPT-4 deployments use some form of RAG.

How a RAG pipeline works

A standard RAG pipeline operates in two phases: an offline indexing phase and an online query phase.

Offline indexing

Before any queries are served, the source documents must be prepared:

1. Document ingestion. Source documents — PDFs, web pages, database exports, wikis, support articles, legal filings — are collected and normalized into plain text. This seemingly simple step is a major source of quality issues: PDF parsing loses table structure, OCR introduces errors, and HTML cleaning can strip important formatting context.

2. Chunking. Documents are split into smaller passages, typically 200-1000 tokens each. Chunking strategy has an outsized impact on retrieval quality. Too small and chunks lack context; too large and they dilute relevance and consume context window space. Common approaches include fixed-size chunking with overlap, semantic chunking (splitting at paragraph or section boundaries), and recursive chunking (splitting hierarchically).

3. Embedding. Each chunk is converted to a dense vector representation (typically 768-3072 dimensions) using an embedding model. Popular choices include OpenAI’s text-embedding-3-large (3072 dimensions), Cohere’s embed-v3, Google’s text-embedding-004, and open-source models like BGE, E5, and GTE. The embedding model determines how well semantic similarity between queries and documents is captured.

4. Indexing. Vectors are stored in a vector database — Pinecone, Weaviate, Qdrant, Chroma, Milvus, or pgvector — with metadata (source URL, document title, section, timestamp). The database builds an approximate nearest neighbor (ANN) index for fast retrieval, typically using HNSW (Hierarchical Navigable Small World) graphs.

Online query phase

When a user submits a query:

1. Query embedding. The query is embedded using the same embedding model used during indexing.

2. Retrieval. The vector database returns the top-k most similar chunks (typically k=5-20) based on cosine similarity or dot product between the query vector and indexed vectors. Retrieval latency is typically 10-100ms even for millions of documents.

3. Context assembly. Retrieved chunks are assembled into a prompt alongside the user’s question and system instructions. The prompt typically instructs the model to answer based only on the provided context and to indicate when the context is insufficient.

4. Generation. The language model generates a response grounded in the retrieved passages. With citation-capable models like Anthropic’s Claude, specific claims can be attributed to specific source chunks.

RAG architecture variants

The basic pattern described above is “naive RAG.” Production systems have evolved significantly:

Advanced RAG adds pre-retrieval and post-retrieval processing stages. Pre-retrieval enhancements include query rewriting (using an LLM to reformulate the user’s question for better retrieval), query decomposition (splitting complex questions into sub-questions), and HyDE — Hypothetical Document Embedding, where the LLM generates a hypothetical answer that is then used as the retrieval query, often improving recall. Post-retrieval enhancements include reranking (using a cross-encoder model like Cohere Rerank or BGE-reranker to re-score retrieved chunks for relevance), contextual compression (summarizing retrieved chunks to fit more information in the context window), and filtering (removing low-relevance or duplicate chunks).

Modular RAG treats each component as an interchangeable module, allowing different retrieval strategies for different query types. A system might use vector search for semantic questions, keyword search (BM25) for specific term lookups, SQL queries for structured data questions, and API calls for real-time data. The router — typically an LLM classifier — selects the appropriate retrieval strategy for each query.

Agentic RAG gives an AI agent control over the retrieval process. Instead of a fixed retrieve-then-generate pipeline, the agent decides when to retrieve, what to search for, whether the retrieved results are sufficient, and whether to search again with a different query. This handles complex questions that require multiple retrieval steps — for example, “Compare Company A’s revenue growth to Company B’s” requires retrieving financial data for both companies, potentially from different sources.

Graph RAG augments vector retrieval with knowledge graph traversal. Microsoft Research’s GraphRAG system, published in 2024, builds a knowledge graph from the source corpus and uses graph-based retrieval to answer questions that require synthesizing information across multiple documents — something vector similarity search handles poorly. Graph RAG excels at questions like “What are the main themes across this document collection?” that require global understanding rather than local passage retrieval.

RAG vs. alternatives: when to use what

Approach	Best for	Limitations	Cost
RAG	Private/current data, factual accuracy, auditability	Retrieval quality bottleneck, added latency	Medium (embedding + vector DB + retrieval)
Long context (stuff everything in)	Small corpora (<100K tokens), high-value documents	Token costs, attention degradation at scale	High (massive context = massive token cost)
Fine-tuning	Teaching style/format/behavior, domain specialization	No citations, stale knowledge, expensive to update	High (training cost), then low (inference)
Pre-training	Foundational knowledge acquisition	Extremely expensive, no citation, slow to update	Very high
Tool use / API calls	Structured data, real-time information	Requires existing APIs, structured schemas	Low-medium

The key insight is that RAG and long context are not mutually exclusive. The most effective systems use RAG to select relevant passages from large corpora, then pass those passages into a long context window alongside the query. This combines the precision of retrieval with the reasoning capability of large contexts.

The retrieval quality problem

RAG systems live or die on retrieval quality. If the right documents are not retrieved, the model either hallucinates an answer from parametric knowledge or provides an unhelpful response. The three main failure modes:

Embedding model mismatch. General-purpose embedding models may not capture domain-specific semantic similarity. The word “cell” means different things in biology, electrical engineering, and telecommunications. If the embedding model does not understand the domain, semantically relevant documents may not have similar vector representations. Domain-adapted embedding models and fine-tuning on domain data significantly improve retrieval quality.

Chunking artifacts. Poor chunking splits critical information across chunk boundaries. A paragraph explaining a policy might be split so that the condition is in one chunk and the consequence in another — and only one gets retrieved. Overlap-based chunking, parent-child retrieval (retrieving the parent document when a child chunk matches), and document-level context headers all mitigate this problem.

Query-document vocabulary gap. Users often express questions differently than documents express answers. A user asking “How do I get my money back?” may not retrieve a document titled “Refund Policy and Procedures” if the embedding model does not bridge this vocabulary gap. Query expansion, synonym injection, and hybrid search (combining vector similarity with keyword matching) address this.

Measuring RAG performance

Production RAG systems require systematic evaluation across multiple dimensions:

Retrieval metrics measure whether the right documents are found:

• Recall@k. Of all relevant documents in the corpus, what fraction appears in the top-k retrieved results? Target: 85-95%.
• Precision@k. Of the top-k retrieved documents, what fraction is actually relevant? Target: 70-90%.
• Mean Reciprocal Rank (MRR). How high does the first relevant document appear in the results?

Generation metrics measure whether the response is accurate and faithful:

• Faithfulness. Does the response only make claims supported by the retrieved context? Measured by NLI (natural language inference) models or LLM-based judges.
• Answer relevance. Does the response actually address the user’s question?
• Hallucination rate. What percentage of claims in the response are not supported by any retrieved source?

Evaluation frameworks like RAGAS (Retrieval-Augmented Generation Assessment), TruLens, and LangSmith provide automated measurement of these metrics. Production systems typically target under 5% hallucination rate and over 85% retrieval recall.

Real-world RAG deployments at scale

Microsoft Copilot. The largest RAG deployment in the world runs across Microsoft 365 — Outlook, Word, Excel, PowerPoint, Teams. Azure AI Search indexes each organization’s documents, emails, and messages. When a user asks Copilot a question, it retrieves relevant content from the organization’s Microsoft Graph, injects it into context with GPT-4, and generates a grounded response. Microsoft reports that Copilot processes over 1 billion RAG queries per month across enterprise customers.

Notion AI. Notion’s AI features use RAG over each workspace’s pages, databases, and documents. When a user asks a question, the system retrieves relevant Notion pages using a custom embedding and retrieval pipeline, then generates an answer grounded in the workspace’s content. This is a canonical example of “private knowledge RAG” — the same model architecture works across millions of workspaces, each with completely different content.

Perplexity. Perplexity’s search engine is fundamentally a RAG system over the internet. Each query triggers web search, the results are retrieved and indexed in real-time, and the language model generates a cited answer from the retrieved pages. Perplexity processes over 100 million queries per month, each involving real-time RAG over web content.

Stripe Docs AI. Stripe’s documentation AI assistant uses RAG over Stripe’s developer documentation, API references, and integration guides. When a developer asks a question, the system retrieves relevant documentation passages and generates an answer with links to source pages. Stripe reports that the RAG-based assistant resolves 30-40% of developer questions without requiring human support.

Cost engineering for RAG

RAG introduces costs at every stage of the pipeline. Understanding the cost structure is essential for production deployment:

Embedding costs. Converting documents to vectors costs $0.02-0.13 per million tokens depending on the embedding model. For a 10-million-document corpus averaging 500 tokens per document, initial embedding costs $100-650. Re-embedding is needed when switching embedding models or when documents change.

Vector database costs. Managed vector databases charge based on storage and query volume. Pinecone’s standard tier costs approximately $70/month per million vectors. Self-hosted alternatives (Qdrant, Milvus, pgvector) reduce per-unit costs but add operational overhead.

Retrieval compute. Each query requires an embedding call ($0.00002-0.00013 per query) plus a vector search (negligible at low-medium volume, significant at millions of queries per day).

Generation costs. The retrieved context adds tokens to every API call. If RAG adds an average of 2,000 tokens of context to each query, and the system handles 1 million queries per month using Claude Sonnet at $3/million input tokens, the RAG context alone costs $6,000/month in additional input tokens.

Total cost example. A medium-scale enterprise RAG system processing 500,000 queries per month over a 1-million-document corpus might cost: $2,000/month for the vector database, $50/month for embedding new/updated documents, $3,000-5,000/month in additional LLM token costs for retrieved context, totaling $5,000-7,000/month in RAG-specific costs on top of base LLM expenses.

Common RAG pitfalls and solutions

Pitfall: Treating RAG as a solved problem. Many teams assume that plugging a vector database into an LLM pipeline produces reliable results. In practice, a naive RAG implementation often retrieves irrelevant documents and produces worse results than an ungrounded model that at least draws on its training knowledge. Solution: invest in retrieval evaluation, chunking optimization, and reranking before optimizing the generation stage.

Pitfall: Ignoring document preprocessing. The quality ceiling of RAG is set by document preparation. Tables that parse as gibberish, PDFs with broken text extraction, and HTML with boilerplate noise all poison the retrieval index. Solution: invest in high-quality document processing using tools like Unstructured, LlamaParse, or custom extraction pipelines. Budget 30-50% of RAG development time for data preparation.

Pitfall: Embedding model one-size-fits-all. General-purpose embedding models underperform on specialized domains. A medical RAG system using a general embedding model will retrieve documents that are semantically similar in general English but not medically relevant. Solution: evaluate domain-specific embedding models, or fine-tune a general model on domain data using contrastive learning.

Pitfall: Fixed chunk sizes. Using a single chunk size for all documents loses information. A legal contract needs larger chunks to preserve clause context. A FAQ document works better with small, question-answer chunks. Solution: use adaptive chunking based on document structure, or implement hierarchical retrieval that considers both small chunks and their parent sections.

The future of RAG

RAG continues evolving rapidly. Several trends are shaping its trajectory:

Longer context windows are not killing RAG. When Gemini 2.5 Pro launched with a 1-million-token context, many predicted RAG’s obsolescence. In practice, stuffing millions of tokens into context is expensive ($1+ per query at million-token scale), slow (initial prompt processing takes seconds), and often counterproductive (models struggle to attend equally to information across very long contexts). RAG remains more cost-effective and often more accurate for corpora larger than a few hundred pages.

Multimodal RAG retrieves and reasons over images, tables, charts, and diagrams in addition to text. Vision-language models like GPT-4o and Claude can process images in context, enabling RAG systems that retrieve relevant figures and charts alongside text passages. This is critical for technical documentation, scientific papers, and financial reports where visual content carries essential information.

Real-time RAG continuously updates the retrieval index as source data changes, enabling responses that reflect the latest information within seconds or minutes of a change. This is replacing the batch-update approach (re-indexing nightly or weekly) for time-sensitive applications like customer support and news.

Frequently asked questions

What is the difference between RAG and fine-tuning? RAG provides the model with external information at query time without changing the model itself. Fine-tuning modifies the model’s weights to encode new knowledge or behavior. RAG is better for factual accuracy (because claims can be cited), frequently changing data, and large knowledge bases. Fine-tuning is better for teaching models new formats, styles, or domain-specific behaviors. Many production systems use both: a fine-tuned model that is also augmented with RAG.

How many documents can a RAG system handle? There is no practical upper limit on corpus size. Vector databases like Pinecone and Milvus handle billions of vectors. The key constraint is retrieval quality, not quantity — as the corpus grows, the chance of retrieving irrelevant documents increases, requiring better reranking, filtering, and query strategies. Production systems routinely handle 1-100 million documents.

Is RAG still relevant with million-token context windows? Yes. Even with Gemini’s 1-million-token context, RAG is more cost-effective for large corpora (processing 1M tokens costs roughly $1 per query vs. $0.01-0.05 for RAG), and retrieval often outperforms brute-force long context for finding specific information in large document collections. The optimal approach combines RAG retrieval with moderate context windows (32K-200K tokens).

What is the biggest mistake teams make with RAG? Underinvesting in retrieval quality. Most teams spend 80% of their effort on the generation side (prompt engineering, model selection) and 20% on retrieval, when the ratio should be reversed. If the wrong documents are retrieved, no amount of prompt engineering will produce a correct answer. Start by measuring retrieval recall and precision, then optimize chunking, embedding, and reranking before touching the generation prompt.

How do you evaluate whether a RAG system is working? Use a combination of retrieval metrics (recall@k, precision@k) and generation metrics (faithfulness, hallucination rate, answer relevance). Build an evaluation dataset of 100-500 question-answer pairs with known source documents. Run automated evaluation using frameworks like RAGAS or manual evaluation with domain experts. Target under 5% hallucination rate and over 85% retrieval recall for production readiness.