Why RAG? The fundamental problem with generic LLMs
Large language models (GPT-4, Claude, Llama, Mistral) have knowledge frozen at their training cutoff and know nothing about your internal documents, your knowledge base or your specific business data. Ask GPT-4 about your company's internal procedures, your API specs, or the latest regulations in your industry — it will confidently make something up. That is the hallucination problem, compounded by the training cutoff limit.
Adapting a model through fine-tuning is expensive (GPU time, labeled data, infrastructure), slow (days to weeks), and rigid (a single new document requires a new training run). RAG solves this by augmenting the LLM's prompt with relevant document passages retrieved in real time. The LLM no longer answers from memory — it answers based on the documents supplied in its immediate context. The result: up-to-date, sourceable, controllable answers tailored to your domain.
RAG vs fine-tuning vs long context: when to use what?
Three approaches let you specialize an LLM on specific data. Long-context prompt engineering means loading every document straight into the prompt — possible with models offering 128K or 1M token windows (Gemini 1.5 Pro, Claude 3), but expensive in tokens and latency, and it degrades performance once the context gets too long (the LLM 'forgets' information buried in the middle of the context).
Fine-tuning changes the model's weights to permanently adapt its style, behavior or knowledge. Ideal for adapting tone (legal, medical, conversational), enforcing a specific response format, or speeding up inference on repetitive tasks. Poorly suited to data that changes often: a single new document requires a new training cycle. RAG is generally the best-suited approach for business applications with dynamic knowledge bases: technical documentation, FAQs, regulatory databases, internal wikis. Combining RAG and fine-tuning is sometimes used together: fine-tuning adapts behavior, RAG supplies the specific knowledge.
Use RAG when data changes often, when the source must be cited explicitly, and when the knowledge scope is broad and evolving. Use fine-tuning when the response style needs to change permanently, when the task is highly repetitive and well-defined, or when inference latency is critical (a model fine-tuned on a narrow domain can be smaller and faster than a generalist LLM plus RAG). The two are not mutually exclusive.
The RAG pipeline: indexing, retrieval and generation
A RAG pipeline is made of two independent phases: the indexing phase (offline, run once or periodically) which prepares and stores documents, and the query phase (online, real time) which retrieves the relevant passages and generates the answer.
Indexing phase: load, chunk, embed, index
Indexing has four steps. (1) Loading: ingesting source documents (PDF via PyMuPDF or pdfplumber, Markdown, HTML, databases, Confluence, Notion, Google Drive, emails). Connectors (LlamaIndex Readers, LangChain Document Loaders) handle most common formats. (2) Chunking: splitting content into appropriately sized passages (see the dedicated section). (3) Embedding: each chunk is turned into a numeric vector via an embedding model (OpenAI text-embedding-3-small, Cohere embed-v3, or open-source sentence-transformers models). (4) Indexing into a vector database along with associated metadata (source, date, section, author) that will enable filtering at retrieval time.
The quality of indexing sets a ceiling on the final quality of the RAG system. Poorly loaded documents (imperfect OCR on scanned PDFs, HTML full of navigation noise), badly split chunks (too short, too long, cutting an idea mid-thought), or an embedding model unsuited to the language or domain irreversibly degrade downstream retrieval.
Query phase: retrieve, rerank, generate
The query phase has three steps. (1) The user's question is embedded with exactly the same embedding model used during indexing — a different model would produce vectors in an incompatible space. (2) Similarity search in the vector database: the k nearest vectors (typically k=5 to 20 depending on context) are retrieved via cosine similarity, dot product, or euclidean distance. (3) The matching chunks are injected into the prompt with clear instructions: 'Answer the following question using only the documents below. If the answer is not in the documents, say so explicitly. Documents: [chunks]. Question: [question]'. The LLM then generates a response grounded in the supplied documents.
An optional (but strongly recommended) reranking stage sits between retrieval and generation: the initial k candidates are re-scored by a more precise cross-encoder, and only the top-m (m < k) are injected into the prompt. This avoids sending loosely relevant passages that pollute the context.
Hybrid search: combining semantics and keywords
Pure vector search (dense retrieval) excels at semantic similarity — it finds conceptually close passages even with different wording. But it performs worse for queries on specific terms: proper nouns, version numbers, product codes, rare technical terms. Keyword search (sparse retrieval — BM25 or TF-IDF) is excellent for exactly these cases but does not understand semantics.
Hybrid search combines both: you compute a dense relevance score and a sparse (BM25) score, merge them via Reciprocal Rank Fusion (RRF) or a weighted combination, and retrieve the passages ranked highest across both dimensions. Pinecone, Weaviate and Qdrant support hybrid search natively. Elasticsearch and OpenSearch also offer this combination. It is the recommended approach in production: the precision gain on mixed queries (semantic plus specific terms) is generally significant.
Embeddings: the semantic representation of text
An embedding is a numeric representation (a vector of 384 to 3072 dimensions depending on the model) that captures the semantic meaning of a text in a continuous vector space. Two semantically close texts have close vectors in this space, even if they do not use the same words — and that is what makes semantic search possible.
This property emerges from training the model on millions of pairs of similar and dissimilar texts. 'Car' and 'automobile' sit close together in embedding space. 'Financial bank' and 'river bank' sit far apart — contextual disambiguation is captured. Keyword search (BM25, TF-IDF) does not capture this semantics: it finds documents containing the exact same terms, not conceptually equivalent documents.
Choosing an embedding model
OpenAI's text-embedding-3-small (1536 dimensions) and text-embedding-3-large (3072 dimensions) offer an excellent quality/price ratio and are the most widely used choices in production applications. Cohere Embed v3 is particularly strong on multilingual use cases and supports query vs document modes (query and document vectors are generated with different instructions to maximize retrieval precision).
Open-source Hugging Face models (sentence-transformers) enable on-premise deployment with no dependency on an external API. For French and multilingual use cases: paraphrase-multilingual-mpnet-base-v2, multilingual-e5-large (Microsoft), or CamemBERT-based models for French specifically. Hugging Face's MTEB Leaderboard (Massive Text Embedding Benchmark) is the reference for comparing model performance by language and task type.
Similarity metrics: cosine, dot product, euclidean
The similarity metric determines how you measure 'closeness' between two vectors. Cosine similarity measures the angle between two vectors — it is insensitive to magnitude and excels at comparing texts of different lengths. It is the default metric for text embeddings. Dot product measures both magnitude and direction — it is equivalent to cosine similarity when vectors are normalized (norm = 1), which is often the case for modern models. Euclidean distance measures the geometric distance between two points — less used for text since it depends on vector magnitude.
In practice: for OpenAI models and most commercial models, vectors are normalized and all three metrics produce similar results. Check the documentation of the model you choose for the recommended metric and configure your vector database accordingly.
The MTEB (Massive Text Embedding Benchmark) from Hugging Face evaluates embedding models on 56 tasks across 112 languages (retrieval, classification, clustering, semantic similarity). It is the reference for choosing a model based on target language and task type. Rankings vary significantly: a model excellent in English can be mediocre in French.
Muennighoff et al. - MTEB: Massive Text Embedding Benchmark, 2023Vector databases: indexes, tools and choices
A vector database stores vectors and enables large-scale similarity search (millions to billions of vectors) with millisecond response times, using specialized indexes that trade exact search for approximate but extremely fast search.
HNSW vs IVF indexes: how they work
HNSW (Hierarchical Navigable Small World) is the most popular vector index for RAG applications. It builds a hierarchical graph where nearby nodes are connected at multiple levels — search navigates from the coarsest level to the finest, quickly eliminating distant vectors. HNSW offers an excellent precision/speed trade-off and supports incremental insertions without rebuilding the index. Pinecone, Weaviate, Qdrant and pgvector all use HNSW.
IVF (Inverted File Index) divides the vector space into clusters (Voronoi cells) and assigns each vector to its nearest cluster at indexing time. Search only scans the n cells closest to the query vector. IVF is very efficient at massive scale (billions of vectors) but less precise than HNSW at smaller scale, and does not handle incremental insertions well (needs periodic reindexing). FAISS (Facebook AI Similarity Search) is the reference library for IVF at very large scale.
Managed solutions: Pinecone, Weaviate Cloud, Qdrant Cloud
Pinecone is the most widely used managed vector service in production: a simple API, automatic scalability, metadata filtering, native hybrid search support (dense + sparse). Its limitation is cost at scale and the lack of self-hosting. Weaviate is open-source with a managed cloud offering: it supports multi-tenancy (important for SaaS applications with multiple customers), a GraphQL API, and built-in embedding modules (you can delegate embedding to Weaviate). Qdrant is an open-source Rust alternative, very strong on ANN benchmarks and particularly memory-efficient.
pgvector and Chroma: integrations and prototyping
pgvector is a PostgreSQL extension that adds a vector column type and similarity operators (cosine, dot product, L2). Its major advantage is architectural: no new infrastructure — embeddings and relational data coexist in the same PostgreSQL database, enabling hybrid SQL queries that combine metadata filters and vector search. Supabase, Neon and Timescale offer pgvector natively. Its limit is performance beyond a few million vectors — pgvector's HNSW index remains less optimized than dedicated vector databases.
Chroma is the lightest and simplest open-source vector database to set up: a Python API in a few lines, on-disk or in-memory persistence, ideal for prototyping and low-volume applications. LanceDB is a newer alternative (the Lance format, built for Lakehouses) that stores vectors as Lance files on disk or S3 and delivers very good serverless performance.
Chunking and reranking: the two quality levers of RAG
The quality of a RAG system depends as much on chunking and reranking as on the generation LLM itself. Poor chunking produces incoherent passages that irreversibly degrade the final answer, no matter how good the generation model is.
Chunking strategies: fixed-size, semantic, hierarchical
Fixed-size chunking (500-1000 tokens with 10-20% overlap) is the simplest strategy: text is split into fixed-size windows with overlap to avoid cutting sentences. Simple to implement, functional on homogeneous text, but it can cut reasoning mid-thought. The overlap makes it possible to retrieve the cut passages regardless of which window contains them.
Semantic chunking detects the text's natural boundaries (paragraph endings, topic changes measured by cosine distance between consecutive sentences) and splits at those boundaries — chunks respect the document's logical coherence. Parent-child chunking (or small-to-big retrieval) indexes fine-grained chunks (children) but injects broader chunks (parents) that contain the retrieved children into the prompt — you get the precision of small chunks with the full context of larger ones. Propositional chunking splits the document into atomic propositions (one idea per chunk) — very precise for factual documents.
Reranking: cross-encoders for fine-grained relevance
Vector search retrieves the k passages semantically closest to the query vector, but semantic proximity in embedding space does not guarantee specific relevance for answering the question. A passage about 'electric cars' can be semantically close to a question about 'autonomous vehicle batteries' without actually answering the question.
Reranking passes the top-k candidates through a cross-encoder — a more powerful model that takes the (question, passage) pair as input and produces a direct relevance score. Cross-encoders see both texts simultaneously and capture fine-grained interactions between question and passage, unlike bi-encoders (embedding models) which encode both separately. Cohere Rerank, Jina Reranker v2 and sentence-transformers cross-encoders (ms-marco-MiniLM-L-6-v2, bge-reranker-v2-m3) are the most widely used solutions. The cost is justified: reranking only applies to the pre-selected k candidates (typically 10 to 20), not the entire database.
RAG reduces but does not eliminate hallucinations. The LLM can still invent details not present in the supplied documents, create incorrect syntheses across multiple passages, or ignore relevant passages that were retrieved. Guardrails are necessary: prompts with an explicit 'answer only from the supplied documents' instruction, mandatory source citation, confidence scoring, and automated evaluation with RAGAS (Faithfulness, Answer Relevancy, Context Precision, Context Recall).
Advanced RAG: GraphRAG, Self-RAG, HyDE and Agentic RAG
Basic RAG (naive RAG) has well-documented limitations: vector search misses relevant passages when the question is ambiguous or phrased very differently from the document text, injected context can be insufficient for multi-hop questions (which require several linked passages), and the LLM does not verify the relevance of what it receives. Advanced RAG patterns address these limitations.
Query transformation: HyDE, decomposition and reformulation
HyDE (Hypothetical Document Embeddings) inverts retrieval logic: instead of embedding the question and searching for close passages, you first ask the LLM to generate a hypothetical document that would answer the question, then embed that hypothetical document to search for the closest real passages. The hypothetical document has a style and vocabulary similar to the documents in your base, which improves retrieval precision on questions phrased very differently from the corpus.
Query decomposition splits complex multi-hop questions into simple sub-questions, retrieves passages for each sub-question independently, and then combines the answers. Step-back prompting asks the LLM to reformulate the question into a more general one before retrieval — useful for very specific questions that don't match direct passages. These techniques are available in LangChain and LlamaIndex.
GraphRAG: retrieval over knowledge graphs
GraphRAG (popularized by Microsoft Research in 2024) replaces the vector database with a knowledge graph automatically extracted from documents via an LLM. Entities (people, concepts, products, events) and their relationships are extracted and stored in a graph, and retrieval traverses this graph to find information related to the question — instead of doing vector similarity over raw passages.
The advantage is significant for questions that require synthesizing information scattered across a document: 'What are all the partners mentioned across the last 3 quarters' reports?' is a hard question for vector RAG (you would need to retrieve and aggregate passages across 3 periods) but a natural one for GraphRAG (you traverse the 'Partner' entity graph). The limitation is the cost of building the graph (expensive LLM extraction) and maintaining it as documents change.
Self-RAG and Agentic RAG
Self-RAG (Asai et al. 2023) is an LLM fine-tuned to dynamically decide whether it needs to retrieve documents for each sub-question, and to self-evaluate the relevance and faithfulness of its own answer via special tokens (Retrieve, IsRel, IsSup, IsUse). It is more precise and cheaper than systematic RAG because it only retrieves documents when needed.
Agentic RAG turns the pipeline into an agentic loop: an LLM orchestrator dynamically decides which tools to use (vector search, web search, SQL execution, third-party APIs), can re-run retrieval if the first pass was insufficient, and iterates until it reaches a satisfactory answer. LangGraph, LlamaIndex Workflows and AutoGen are the most widely used frameworks for implementing Agentic RAG. This is the approach behind the most sophisticated RAG systems in production (enterprise AI assistants, autonomous research agents).
The paper 'From Local to Global: A Graph RAG Approach to Query-Focused Summarization' (Edge et al., Microsoft Research, 2024) formalized the GraphRAG approach and demonstrated significant gains on questions requiring global synthesis of a corpus (global sensemaking). Microsoft's open-source implementation is available on GitHub as microsoft/graphrag.
Edge et al. - From Local to Global: A Graph RAG Approach, Microsoft Research, 2024Anchoring RAG concepts with spaced repetition
RAG blends NLP concepts (embeddings, similarity search, cross-encoders), infrastructure concepts (vector databases, HNSW/IVF indexes, latency) and system architecture concepts (pipeline, hybrid search, agentic orchestration). The sheer number of components and advanced patterns makes passive re-reading an inefficient way to learn — flashcards are particularly well suited to maintaining a clear picture of each building block and how they interact.
Memia's 'RAG and LLM Architectures', 'Embeddings and Vector Databases' and 'Generative AI and LLM' decks cover definitions, subtle distinctions (HNSW vs IVF, cosine vs dot product, RAG vs fine-tuning) and the system architecture questions most frequently asked in ML Engineer and Data Engineer interviews.
The most frequent interview concepts: (1) RAG vs fine-tuning vs long context (with the trade-offs). (2) The 4 steps of the RAG pipeline (indexing, embedding, retrieval, generation). (3) HNSW vs IVF: when to use each. (4) Dense + sparse hybrid search: why and how. (5) Cross-encoder reranker vs bi-encoder. (6) HyDE: inverting the query to improve retrieval. (7) RAGAS: the 4 evaluation metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall).
Explore the Data & AI cluster
Frequently asked questions about RAG and augmented generation
What is RAG (Retrieval Augmented Generation)?
RAG is an architecture that augments an LLM's answers by supplying it with relevant document passages retrieved in real time. Rather than answering from memory (with the risk of hallucination and stale data), the LLM bases its answer on the documents injected into its prompt. This lets you query an LLM about specific internal data without costly retraining.
What is the difference between RAG and fine-tuning?
RAG adds context at inference time — documents are retrieved and injected into the prompt on every request. It is flexible, and data can be updated without retraining. Fine-tuning permanently changes the model's weights to adapt its behavior — expensive, rigid, well suited to style or very stable knowledge. For data that changes often: RAG. To permanently adapt the model's behavior: fine-tuning.
What is an embedding and why is it useful for RAG?
An embedding is a numeric vector representation of a text that captures its semantic meaning. Two semantically close texts have close vectors in embedding space, even with different words. RAG uses embeddings to retrieve passages conceptually close to the question — something exact keyword search cannot do.
What are the main vector databases?
Pinecone (managed service, simple API, native hybrid search), Weaviate (open-source plus cloud, multi-tenancy, GraphQL), Qdrant (high-performance open-source in Rust), pgvector (PostgreSQL extension, ideal if you're already on Postgres/Supabase), Chroma (lightweight, open-source, ideal for prototyping) and LanceDB (Lance format, serverless, good for Lakehouses). The choice depends on volume, existing infrastructure and filtering needs.
What is hybrid search in a RAG system?
Hybrid search combines vector search (dense retrieval — semantic, captures synonyms and meaning) with keyword search (sparse retrieval — BM25, excellent for specific terms, proper nouns, product codes). Scores are merged via Reciprocal Rank Fusion (RRF). In production, hybrid search consistently outperforms pure vector search on business queries that mix semantics with specific terms.
What is chunking and how do you choose a strategy?
Chunking is splitting documents into appropriately sized passages before indexing. Fixed-size chunking (500-1000 tokens with overlap) is simple and functional. Semantic chunking respects natural boundaries (paragraphs, topic changes). Parent-child chunking indexes fine-grained chunks but injects broader chunks into the prompt — combining retrieval precision with contextual richness. The optimal size depends on content type and the LLM's context window.
What is reranking in a RAG pipeline?
Reranking is a refinement step after initial retrieval. Vector search retrieves the k semantically closest candidates. A cross-encoder (a more precise model that sees the question and passage pair simultaneously) re-scores each candidate for relevance specific to the question. Only the re-ranked top-m passages are sent to the LLM. Cohere Rerank, Jina Reranker and sentence-transformers cross-encoders are the most widely used solutions.
Does RAG eliminate LLM hallucinations?
No, RAG reduces but does not eliminate hallucinations. The LLM can still invent details not present in the supplied documents, create incorrect syntheses, or ignore relevant retrieved passages. Additional measures are needed: prompts instructing 'answer only from the supplied documents', mandatory source citation, and automated evaluation with RAGAS (Faithfulness, Answer Relevancy, Context Precision, Context Recall).
What is HyDE (Hypothetical Document Embeddings)?
HyDE is a technique that inverts retrieval logic: instead of embedding the question and searching for close passages, you first ask the LLM to generate a hypothetical document that would answer the question, then embed that hypothetical document to retrieve the closest real passages. The hypothetical document has a style close to the corpus documents, which improves retrieval precision when the question is phrased very differently.
What is GraphRAG?
GraphRAG (Microsoft Research, 2024) replaces the vector database with a knowledge graph automatically extracted from documents via an LLM. Entities and their relationships are stored in a graph, and retrieval traverses this graph to find related information. The advantage shows up for questions requiring a synthesis of multiple linked entities ('all partners mentioned across 3 reports') — hard in vector RAG but natural in GraphRAG.