RAG (Retrieval-Augmented Generation)
Combine LLMs with external knowledge bases for grounded responses.
Last updated: July 2026
A user asks a question in plain language.
Retrieval-Augmented Generation (RAG) is an AI framework that improves the accuracy and relevance of large language model (LLM) outputs by connecting the model to an external knowledge base at query time, before generating a response. Instead of relying solely on what an LLM 'memorized' during training, RAG retrieves fresh, relevant context from external documents, databases, or APIs and injects that context into the LLM's prompt. The LLM then generates its answer grounded in that retrieved information. RAG was formally introduced in a 2020 paper by Lewis et al. at Meta AI.
“Without RAG, an LLM is like a brilliant consultant who only knows what they read before 2023 and refuses to check their notes. With RAG, that consultant pauses, pulls the relevant memo from the filing cabinet, and answers based on current facts.”
| Component | What it does | Example |
|---|---|---|
Knowledge Base / Document Store | The source of truth: a collection of documents, PDFs, web pages, or database records the LLM should draw from | A legal tech company indexes 10,000 case law PDFs as raw input before any processing |
Chunking | Splits large documents into smaller text segments (256–1024 tokens, with overlap) before embedding and storage | A 50-page employee handbook is split into ~200 chunks of 500 tokens each, with 50-token overlaps to preserve boundary context |
Embedding Model | Converts text chunks and queries into dense numerical vectors representing semantic meaning | 'How do I cancel my order?' and 'What is the cancellation process?' produce nearly identical vectors and match the same query |
Vector Database | Stores chunk embeddings and enables fast similarity search, returning the top-K most semantically similar chunks for a given query | Like a library that groups books by how related they are in meaning. A query surfaces the 5 most relevant passages regardless of exact wording |
Retriever | Takes the user's query, embeds it, queries the vector database, and returns the top-K most relevant chunks using vector, keyword, or hybrid search | A user asks about drug side effects; the retriever returns the 3 most relevant passages from clinical documentation before the LLM sees the question |
Prompt Augmentation | Injects retrieved chunks into the LLM's prompt alongside the original query, with instructions to answer only from the provided context | System: Answer ONLY based on the context below. Context: [Chunk 1: return policy...] [Chunk 2: final sale items...] User: Can I return a sale item? |
LLM Generator | Reads the augmented prompt (query + retrieved context) and generates a coherent, grounded response | GPT-5.5 or Claude Sonnet 4.6 reads the augmented prompt and responds: 'Items marked Final Sale cannot be returned, regardless of when they were purchased' |
Full assembled example
User query: 'What is our refund policy for orders over $200?' → Retriever fetches the top-2 chunks from refund-policy.pdf → Chunks injected into prompt with instruction 'Answer ONLY from context below' → LLM generates: 'Per Section 3.2 of our refund policy, orders over $200 are eligible for a full refund within 30 days of purchase with proof of purchase.' [Source: refund-policy.pdf, p.3]
Core Principles
- 1Retrieval before generation - The LLM should never generate without first attempting retrieval. If no relevant context is found, the system should return 'I don't know' rather than guess.
- 2Chunk quality determines answer quality - A RAG system for a cooking website returned full recipe pages as chunks, and answers became diffuse. Switching to ingredient-section-level chunks (200 tokens) immediately improved precision.
- 3The LLM is the synthesizer, not the source - If a retrieved chunk says Q3 revenue is $4.2M, the LLM should cite that figure from context, not recall a different number from its training on public financial data.
- 4Transparency and traceability - A legal research tool surfaces the exact paragraph and document name alongside every answer, so a lawyer can verify the source before acting on it.
- 5Retrieval and generation can be evaluated independently - Use DCG/nDCG metrics for retrieval quality and LLM-as-a-judge for generation quality. Tools like RAGAS (a framework for scoring the quality of RAG answers) cover both. Measure each failure mode separately.
Without RAG vs. with RAG
Before
User asks: 'What is our refund policy for orders over $200?' LLM replies with a generic, plausible-sounding but potentially incorrect refund policy it hallucinated from training data.
After
The system retrieves the company's actual refund-policy-v3.pdf, feeds the relevant paragraph to the LLM, and the LLM answers accurately, citing the actual policy section.
Naive / Basic RAG
The simplest RAG pipeline. Documents are chunked at fixed sizes, embedded, stored in a vector DB, and the top-K semantically similar chunks are retrieved using cosine similarity and passed directly to the LLM.
Prompt
A startup builds a Q&A bot over 50-page product documentation. They chunk by 512 tokens, embed with a sentence transformer, store in Pinecone, and retrieve the top-3 chunks per query.
Output
Accurate answers grounded in product documentation with low infrastructure overhead
Advanced / Modular RAG
An improved pipeline with query rewriting, re-ranking retrieved results with a cross-encoder, metadata filtering, hybrid search (vector + keyword), and multi-step retrieval.
Prompt
An HR chatbot rewrites 'what is the pto policy' to 'paid time off and vacation leave policy,' runs hybrid search (BM25 + vector), then re-ranks top-10 with a cross-encoder to select the best 3.
Output
Higher-precision answers from large document corpora with reduced irrelevant context in the LLM prompt
Graph RAG
Documents are parsed into a knowledge graph where entities and relationships become nodes and edges. Retrieval traverses graph paths, enabling multi-hop reasoning across related facts.
Prompt
A pharmaceutical company stores drug interactions as a knowledge graph. A query traverses: Drug A → metabolized by CYP3A4 → Drug B also inhibits CYP3A4 → flag potential interaction.
Output
Multi-hop answers connecting related entities that flat vector search would miss
Agentic RAG
The LLM acts as an agent that decides when to retrieve, what queries to issue, how many retrieval rounds to perform, and whether to use other tools. The LLM controls the retrieval loop dynamically.
Prompt
A financial analyst assistant receives 'Compare Q3 2024 earnings of our top 5 clients.' The agent issues 5 separate retrieval queries, synthesizes all results, and generates a comparative summary in one turn.
Output
Synthesized multi-document analysis generated through dynamic, self-directed retrieval
Self-RAG (Self-Reflective RAG)
The LLM is fine-tuned to generate special 'reflection tokens' during generation, indicating whether it needs to retrieve, whether retrieved passages are relevant, and whether its output is supported by evidence.
Prompt
Mid-response, the model flags that its draft answer contains a factual claim not supported by retrieved context, triggers a second retrieval, fetches a more relevant chunk, and revises the sentence on its own.
Output
Self-audited responses that trigger additional retrieval when the model detects unsupported claims
Best Practices
- 1Optimize chunk size and overlap for your content type - A legal team using contract documents sets chunk size to 600 tokens with 100-token overlap, ensuring clause headers and body text always appear together in at least one chunk.
- 2Use hybrid search (vector + keyword) - A support bot uses hybrid search: vector search finds conceptually relevant docs about connection errors, while BM25 ensures the specific error code ERR_CONN_TIMEOUT_4023 is still surfaced.
- 3Add metadata filters to retrieval - A financial compliance chatbot filters retrieved chunks to date >= 2024-01-01 and jurisdiction = 'US', preventing the LLM from citing superseded regulations or EU-only rules.
- 4Re-rank retrieved results - Vector search returns 10 chunks; a cross-encoder scores them by true relevance and selects the top 3, preserving the LLM's context window for actual content rather than padding.
- 5Keep the knowledge base clean and structured - An enterprise pipeline strips PDF watermarks, normalizes whitespace, removes table-of-contents pages, and adds a 2-sentence summary to each chunk before embedding. This improved retrieval precision by ~20%.
- 6Evaluate retrieval and generation separately - A team discovers generation quality scores are high but recall@5 is only 0.62. The right chunk is missing 38% of the time, so they focus on the retriever, not the LLM, saving weeks of misdirected effort.
- 7Design prompts to enforce grounded responses - Explicitly instruct the LLM: 'Answer ONLY from the context. If the answer is not present, say I don't have enough information.' Without this, LLMs blend retrieved facts with parametric memory.
Common Mistakes
| Mistake | Real-world scenario | Fix |
|---|---|---|
| Chunks too large | A RAG bot over a 200-page manual uses 3,000-token chunks. The LLM receives bloated prompts with irrelevant surrounding text, diluting the answer. | Use 256–512 token chunks with ~10% overlap. Test with your specific document type. |
| No chunk overlap | Consecutive 512-token chunks cut sentences in half at the boundary. Queries about content near a boundary return incomplete, contextless fragments. | Add 50–100 token overlap between consecutive chunks to preserve boundary context. |
| Embedding everything without preprocessing | A product catalog with HTML tags, nav menus, cookie notices, and duplicate sidebar text is embedded as-is. The vector space is polluted with noise. | Clean and normalize documents before embedding: strip boilerplate, normalize whitespace, remove duplicates. |
| Using vector search only | A support bot fails to retrieve a chunk containing error code 'ERR_504_TIMEOUT' because the query 'connection dropped' has low cosine similarity to the exact string. | Implement hybrid search: vector similarity + BM25/keyword search with weighted fusion or re-ranking. |
| Ignoring retrieval evaluation | A team tunes the LLM for better generation but retrieval recall@5 is 0.55. The right chunk is missing nearly half the time. Better LLM prompts don't help if the context is wrong. | Evaluate retrieval separately using metrics like recall@K and nDCG. Fix the retriever before tuning the generator. |
| No 'I don't know' instruction | The LLM blends retrieved info with training-time knowledge, confidently providing an answer that mixes real policy with hallucinated details. | Explicitly instruct the LLM: 'Answer only from the context. If not found, say I don't know.' |
| Indexing duplicate content | The same policy document exists in 3 versions (v1, v2, v3). All three are indexed. Retrieval surfaces conflicting information from different versions. | Deduplicate, version-control, and date-filter your knowledge base. Only index the canonical, current version. |
| Ignoring context window limits | Retrieving 20 chunks × 500 tokens each = 10,000 tokens. This exceeds many models' effective context windows, causing 'lost in the middle' degradation. | Retrieve top-K then hard-limit the context passed to the LLM to fit within 40–60% of the model's context window. |
Use Cases
| Use case | Real-world example | Best approach |
|---|---|---|
| Customer support chatbot | A SaaS company's support bot answers billing and technical questions using indexed help docs and changelogs | Basic or Advanced RAG over structured documentation; hybrid search; strict grounding prompt |
| Legal research assistant | Lawyers query a database of 100,000+ case law documents and statutes for relevant precedents | Advanced RAG with re-ranking, metadata filtering by jurisdiction and date; Graph RAG for related cases |
| HR policy Q&A | Employees ask 'How many PTO days do I have?' against the company's HR handbook and employee agreements | Basic RAG with strong metadata filters (employee type, country) and strict 'only from context' prompting |
| Medical literature search | A clinical research tool surfaces relevant trial data when doctors query patient symptom profiles | Advanced RAG with domain-specific embedding models; Graph RAG for drug-interaction relationships |
| Financial document analysis | Analysts query earnings reports, 10-Ks, and analyst notes for comparative data across companies and quarters | Agentic RAG for multi-document synthesis; table-aware chunking; metadata filter by company and period |
| Code documentation helper | A developer tool answers questions about a codebase's internal APIs by indexing code files and docstrings | Basic RAG with code-aware chunking (function/class level); hybrid search on function names + semantic meaning |
| Compliance & regulatory Q&A | A financial firm's compliance bot answers questions about current regulations and internal policies | Advanced RAG with strict date-version filtering; full source citation in every response; audit-log retrieval |
| Internal knowledge management | A company's 'second brain' bot surfaces relevant Confluence pages, Slack threads, and Notion docs | Agentic RAG with multi-source retrieval; freshness-weighted ranking; personal context filters |
Basic RAG for Customer Support
Weak prompt
Ask the LLM about the return policy without retrieval. [LLM answers from parametric memory, risking hallucination]
Strong prompt
User Query: 'Can I return a product I bought 6 weeks ago if it's defective?' Retrieved Chunks (top-2 by vector similarity): [Chunk A]: 'All items may be returned within 30 days for a full refund. Items returned between 31–60 days receive store credit only.' [Chunk B]: 'Defective items are eligible for return or replacement within 90 days of purchase regardless of standard return window, provided a defect is documented.' Prompt to LLM: System: Answer using ONLY the context below. Cite source if possible. If the answer is not in the context, say 'I don't have that information.' Context: [Chunk A] [Chunk B] User: Can I return a product I bought 6 weeks ago if it's defective?
Output
Yes - defective items can be returned or replaced within 90 days of purchase, regardless of the standard 30-day window. Since 6 weeks (42 days) is within that 90-day window, your defective item qualifies. Please contact customer support to document the defect. [Source: return-policy.pdf, p.2]
Advanced RAG with Query Rewriting and Re-ranking
Weak prompt
Run a simple vector search on the vague query 'pto stuff' and pass top-1 result to the LLM.
Strong prompt
Step 1 - Query Rewriting: 'pto stuff' → 'paid time off policy, vacation accrual, sick leave, PTO balance' Step 2 - Hybrid Retrieval: Vector search top-5 + BM25 top-5 → merged to 8 unique chunks Step 3 - Re-ranking: Cross-encoder scores all 8 chunks; top-3 selected: [Score 0.94]: 'Employees accrue 1.5 PTO days per month, totaling 18 days per year...' [Score 0.89]: 'PTO balances can be viewed in the HR portal under Time & Attendance...' [Score 0.81]: 'Unused PTO of up to 5 days may be carried over to the next calendar year...' Step 4 - Prompt Augmentation & LLM Generation
Output
Here's a summary of our PTO policy: - Accrual: You earn 1.5 PTO days per month, totaling 18 days per year. - Viewing Your Balance: Check your current PTO balance in the HR portal under 'Time & Attendance.' - Carryover: Up to 5 unused PTO days can roll over to the next calendar year. If you have a more specific question (e.g., sick leave, parental leave, or requesting time off), feel free to ask!
Agentic RAG for Multi-Document Research
Weak prompt
Run a single retrieval query for 'Q3 2024 earnings' and hope the top chunk covers all three clients.
Strong prompt
User Query: 'Compare Q3 2024 revenue for Acme Corp, Beta Inc, and Gamma Ltd.' Agent Step 1 - Query Decomposition: Sub-query 1: 'Acme Corp Q3 2024 revenue' Sub-query 2: 'Beta Inc Q3 2024 revenue' Sub-query 3: 'Gamma Ltd Q3 2024 revenue' Agent Step 2 - Parallel Retrieval (3 separate queries): Result 1: [Acme_Q3_2024_report.pdf] 'Revenue for Q3 2024: $14.2M, up 8% YoY' Result 2: [Beta_Q3_report.pdf] 'Total Q3 revenue: $9.8M, down 3% from Q3 2023' Result 3: [Gamma_earnings.pdf] 'Gamma Q3 2024 net revenue: $21.5M, +15% YoY' Agent Step 3 - Synthesis Prompt: Synthesize into a comparison table using ONLY retrieved context.
Output
Q3 2024 Revenue Comparison: | Client | Q3 2024 Revenue | YoY Change | Source | |-----------|-----------------|------------|-------------------------| | Acme Corp | $14.2M | +8% | Acme_Q3_2024_report.pdf | | Beta Inc | $9.8M | -3% | Beta_Q3_report.pdf | | Gamma Ltd | $21.5M | +15% | Gamma_earnings.pdf | Gamma Ltd leads in both absolute revenue and growth rate. Beta Inc is the only client showing a revenue decline for the quarter.
Tools & Platforms
| Tool | Best for | Link |
|---|---|---|
| LangChain | Building full RAG pipelines in Python with extensive retriever and LLM integrations; rapid prototyping | https://www.langchain.com |
| LlamaIndex | Document-centric RAG with advanced indexing strategies; great for complex document hierarchies | https://www.llamaindex.ai |
| Pinecone | Managed vector database for production-scale semantic search with a simple API | https://www.pinecone.io |
| Weaviate | Open-source vector DB with built-in hybrid search (BM25 + vector) and schema-based filtering | https://weaviate.io |
| Chroma | Lightweight open-source vector DB ideal for local development and small-scale RAG | https://www.trychroma.com |
| Qdrant | High-performance vector DB with payload-based filtering, built in Rust for speed | https://qdrant.tech |
| FAISS (Meta) | High-performance open-source library for approximate nearest-neighbor search on CPU and GPU | https://github.com/facebookresearch/faiss |
| Haystack (deepset) | End-to-end NLP framework with first-class RAG pipeline support and modular components | https://haystack.deepset.ai |
| RAGAS | Evaluation framework specifically for RAG: scores faithfulness, answer relevancy, and context recall | https://ragas.io |
| Azure AI Search | Enterprise-grade hybrid search (vector + keyword + semantic) with tight Azure/OpenAI integration | https://azure.microsoft.com/en-us/products/ai-services/ai-search |
AWS - What is RAG?
A clear overview of RAG architecture and use cases
IBM - What is RAG?
A plain-language explanation of RAG with enterprise use case context
LangChain RAG Tutorial
Step-by-step tutorial for building a RAG pipeline with LangChain in Python
RAGAS Documentation
Evaluation framework documentation for scoring RAG faithfulness, relevancy, and recall
Meta AI - Original RAG Paper (Lewis et al., 2020)
The foundational academic paper introducing the RAG framework
Frequently asked questions
What is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation (RAG) is an AI framework that enhances the accuracy and relevance of responses from large language models (LLMs) by integrating external knowledge bases during query processing. It retrieves relevant information from documents or databases and incorporates it into the LLM's prompt, allowing the model to generate answers grounded in current data rather than relying solely on pre-trained knowledge.
How does RAG improve the performance of large language models?
RAG improves the performance of large language models by connecting them to external knowledge bases, allowing the models to retrieve and use up-to-date information to generate responses. This process involves retrieving relevant chunks of data from a document store, embedding them into the model's prompt, and ensuring the generated answer is based on the most relevant and current context available.
What are the core components of a RAG system?
A RAG system comprises several key components: a knowledge base or document store for storing information, a chunking process to divide documents into manageable text segments, an embedding model to convert text into numerical vectors, a vector database for storing these vectors, a retriever to find relevant chunks, prompt augmentation to integrate these chunks into the LLM's prompt, and the LLM generator to produce the final response.
What are the best practices for implementing a RAG system?
Best practices for implementing a RAG system include optimizing chunk size and overlap to ensure context is preserved, using hybrid search methods to enhance retrieval accuracy, adding metadata filters to refine search results, re-ranking retrieved results for relevance, maintaining a clean and structured knowledge base, and designing prompts to enforce grounded responses from the LLM.
What are common mistakes to avoid in RAG systems?
Common mistakes in RAG systems include using chunks that are too large, which can dilute the relevance of the response, failing to overlap chunks, which can lead to incomplete context, embedding documents without preprocessing, which introduces noise, relying solely on vector search, which can miss specific queries, and neglecting retrieval evaluation, which can result in poor retrieval accuracy.