What is RAG?

This post establishes the core mental model for Retrieval-Augmented Generation, the single most common architecture for putting large language models to work on real, specific data. The headline question — what is RAG? — deserves a clean answer before we argue why it matters or open up the mechanics, because almost every later confusion traces back to a fuzzy mental model here.

The goal of post one is to make RAG concrete and unmysterious: a model that, before answering, reads relevant text you fetched for it. Retrieve, then generate. Hold that two-step shape and the rest of the day builds cleanly on top of it.

The crucial phrase in this definition is 'before the model answers.' RAG does not change the model at all — no weights are touched, no training happens. Instead, at the moment of answering, you fetch text that's likely relevant to the question from some external source and prepend it to the prompt. The model then generates its response with that text sitting right in front of it.

This reframes what the model is doing. Without RAG, the model recalls from frozen memory. With RAG, it reads supplied material and synthesizes an answer from it. That shift — from recalling to reading — is the entire value proposition, because reading current, specific text is far more reliable than recalling fuzzy training memories.

This slide separates the two kinds of knowledge a RAG system juggles, and the distinction is foundational. Parametric knowledge is everything compressed into the model's weights during pretraining — vast, implicit, and frozen at the training cutoff. The model 'knows' it but can't update it and often can't quote it precisely. Non-parametric knowledge is the text you supply at query time: explicit, exact, and as fresh as your last index update.

RAG is best understood as bolting a non-parametric memory onto a parametric model. The weights provide language fluency and general reasoning; the retrieved text provides the specific facts. Keeping these two sources of knowledge distinct in your head explains why RAG can answer questions about data the model was never trained on.

The flow diagram captures the canonical RAG loop in four nodes. A question comes in. The retrieve step finds the documents most relevant to that question. The augment step combines those documents with the original question into a single enriched prompt. The generate step hands that prompt to the model, which produces an answer grounded in the retrieved text.

The word 'augment' is the A in RAG and the easiest to overlook. You're not replacing the model's reasoning — you're augmenting the prompt with evidence. The model still does the reasoning and writing; it just does so with the right facts in view. This four-step shape is the skeleton every RAG system shares, from a weekend prototype to a production stack.

Embeddings are the mechanism that makes retrieval by meaning possible, and they're worth understanding even at a conceptual level. An embedding model maps a piece of text to a vector — a list of numbers, often hundreds or thousands of them — positioned so that texts with similar meaning land near each other. 'Refund policy' and 'money-back window' end up close even though they share no words.

This is why RAG retrieval is semantic, not keyword-based. You embed the question into the same space, then look for the stored chunks whose vectors sit nearest. Proximity in this space approximates similarity in meaning, so the nearest neighbors are the chunks most likely to answer the question. The quality of these embeddings directly caps how good your retrieval can be.

A vector store is the database that makes the retrieve step fast at scale. It holds the embedding vector for every chunk of your documents, alongside the original text and any metadata. Its core operation is nearest-neighbor search: given a query vector, return the stored vectors closest to it, in milliseconds, even across millions of entries.

Think of it as a search engine indexed by meaning rather than keywords. Tools like Chroma, FAISS, Pinecone, and pgvector all play this role. Without a vector store you'd have to compare the query against every chunk by brute force, which doesn't scale. The store is the practical machinery that turns the elegant idea of semantic retrieval into something that runs in production.

This slide names the most important structural fact about RAG: it has two distinct lifecycles that happen at different times. Indexing is the offline phase — you do it once (and re-do it when documents change): split each document into chunks, embed every chunk, and store the vectors. It can take minutes or hours and nobody is waiting on it.

Querying is the online phase — it happens on every single request and the user is waiting: embed the incoming question, fetch the closest chunks, build the prompt, and generate. A huge fraction of RAG bugs come from conflating these phases — for example, changing the embedding model for queries but forgetting to re-index the documents. Keep the two halves mentally separate and the system stays coherent.

This comparison addresses the question everyone asks early: why not just fine-tune the model on my data instead? The two approaches solve overlapping problems through completely different means. RAG adds knowledge at query time by retrieving text; you update it by editing documents and re-indexing, it can cite the exact source it used, and it needs no training run. Fine-tuning bakes patterns into the weights through a training process; you update it by retraining, it has no built-in notion of citations, and it requires data and compute.

The practical rule of thumb: use RAG when the problem is knowledge — facts the model needs to look up. Use fine-tuning when the problem is behavior — a consistent style, format, or skill. They're not mutually exclusive; mature systems often fine-tune for behavior and use RAG for knowledge. But for 'make the model know my data,' RAG is almost always the cheaper, faster, more auditable answer.

This snippet strips RAG down to its irreducible core so the pattern stops being abstract. A query comes in. We embed it and search the vector store for the four closest chunks. We join those chunks into a context block. We build a prompt that instructs the model to use only that context, then call the model and print the answer. That's the whole architecture in eight lines.

The point isn't that production systems are this short — they add chunking strategy, re-ranking, citations, and evaluation. The point is that none of that changes the fundamental shape: search for relevant text, put it in the prompt, generate. If you understand these eight lines, you understand RAG; everything else is refinement on top of this skeleton.

Naming what RAG is not is as clarifying as defining what it is. RAG is not fine-tuning — no weights change, so the model itself is exactly the same before and after. It is not persistent memory — the retrieved text lives only in this one prompt and vanishes when the request ends; nothing carries to the next call unless you retrieve it again. It is not merely a search engine — search returns documents, while RAG uses those documents to generate a synthesized, natural-language answer.

Most importantly, RAG is not a guarantee of truth. The answer is only as good as the text you retrieved. If retrieval surfaces the wrong chunk, or misses the right one, the model will faithfully answer from bad evidence. This is the antidote to magical thinking: RAG grounds the model in retrieved text, but the quality of that retrieval is on you.

The summary compresses the whole post into four lines you can recall under pressure. RAG means retrieve relevant text, then generate an answer from it. It grounds responses in your data rather than relying solely on training memory. It's built on two pieces — embeddings to capture meaning and a vector store to search by it. And you update its knowledge by editing documents, never by retraining the model.

Hold these four facts and RAG stops being a buzzword and becomes a clear architecture. They're the foundation the next four posts build on, from why retrieval became the default for grounded LLM features to how to build and debug one in code.

That wraps the conceptual foundation. You now have a clean mental model: RAG as a two-step retrieve-then-generate loop that bolts a non-parametric memory — embeddings in a vector store — onto a frozen, parametric model so it can answer from fresh, private, specific data.

The next post shifts from definitions to stakes — why this architecture quietly became the default way to ship LLM features on real data, how it closes the knowledge-cutoff and private-data gaps, and why grounding with citations is what turns a demo into something a business can trust.