The RAG Pipeline

This post sets up the single idea the rest of the day builds on: a large language model is frozen at training time, and RAG is how you give it access to knowledge it never saw. The cover frames RAG not as a clever prompt but as a system you assemble around the model.

The reason this distinction matters is that people often treat 'the AI' as the thing that knows or doesn't know an answer. In a RAG system, the knowing is done by your retrieval layer; the model's job is to read and reason over what that layer hands it. Internalize that split early and every later stage makes sense.

Retrieval-Augmented Generation is two verbs in order. First you retrieve: search your own corpus for the passages most likely to contain the answer. Then you generate: hand those passages to the model as context and ask it to answer using them. The model's parametric memory is still in play, but it's now anchored to text you supplied.

The practical effect is that the model stops being the source of truth and becomes a fluent interface to a source of truth you control. When the answer is wrong, you can ask a sharper question: did retrieval fail to find the right passage, or did the model misread a passage it was given? Those are two different bugs with two different fixes.

The core problem RAG solves is the gap between what a model learned and what your users actually ask about. A frontier model trained last year knows nothing about your internal wiki, your customer's account, or a policy you changed this morning. No amount of clever prompting conjures facts that were never in the weights.

RAG closes that gap at query time. Instead of trying to push knowledge into the model through expensive retraining, you keep the knowledge in a searchable store and pull the relevant slice into the prompt for each question. The model stays general; your data stays current and external.

This comparison is the decision most teams get wrong first. RAG and fine-tuning solve different problems. RAG injects knowledge — facts, documents, things that change — at the moment of the query, and you update it by editing documents. Fine-tuning changes behavior — tone, format, how the model responds — and you update it by retraining.

The rule of thumb: use RAG when the answer depends on information that lives in documents and changes over time, and use fine-tuning when you want to shape how the model talks or follows a format. Many real systems use both — fine-tune for style, RAG for facts — but conflating them leads to retraining when you should have re-indexed.

The pipeline diagram is the map for the whole day. A query comes in, the retriever finds relevant chunks, those chunks are folded into the prompt (the 'augment' step), and the model generates an answer. Four stages, each with knobs.

Seeing it as a flow rather than a single call is the key mental shift. It tells you where to look when something breaks: a wrong answer could originate at retrieval (wrong chunks found), at augmentation (chunks found but poorly assembled), or at generation (good context, bad reading). Post 3 expands each of these stages in detail.

Here we make explicit why RAG is described as a pipeline. There's an offline half that runs once — ingest documents, chunk them, embed the chunks, store the vectors — and an online half that runs per request — embed the question, search, assemble, generate. Each stage has independent choices: chunk size, embedding model, index type, top-k, prompt template.

The consequence is that RAG quality is a product of every stage, not just the model. A brilliant LLM on top of bad chunking still produces bad answers. Treating RAG as a pipeline tells you that improving results often means tuning a stage that has nothing to do with the model at all.

This is the vocabulary you'll meet in every RAG tutorial and tool, so it's worth pinning down once. A chunk is a small slice of a document. An embedding is a vector — a list of numbers — that places that chunk in a space where similar meanings are close together. A vector store is a database you search by closeness rather than by exact keywords.

The retriever is the component that, given a question, returns the most similar chunks. Context is the retrieved text you paste into the prompt. Once these five words are concrete, the rest of the pipeline reads as plain plumbing connecting them together.

This snippet collapses the entire idea into six lines so the shape is unmistakable. retriever.search finds the most relevant chunks for the question; you join their text into a single context string; you build a prompt that explicitly tells the model to use only that context; and you generate.

Everything in the rest of the day is an elaboration of these six lines. Post 3 expands what search really does under the hood, and post 4 implements each piece — chunking, embedding, similarity search — in runnable detail. Keep this skeleton in mind as the thing all that detail hangs on.

The flow diagram answers the question users implicitly ask: where did this answer come from? The ground truth is your documents. The retriever selects the relevant parts. The LLM writes the prose. The output is a reply that is, ideally, traceable back to specific source text.

This traceability is one of RAG's biggest practical wins and the subject of post 2. Because the answer is assembled from identifiable chunks, you can show users the sources, let reviewers verify, and debug wrong answers by inspecting which chunks were retrieved rather than guessing at the model's opaque memory.

It's worth being clear about what RAG does not do, because the misconceptions cause real bugs. RAG does not teach the model your data permanently — nothing is written to the weights; the context is gone after the request. It is not a fix for messy source documents; garbage in still produces garbage out. It is not merely keyword search dressed up; semantic retrieval matches meaning, not exact words.

Most importantly, RAG grounds answers but does not verify them. If your documents are wrong, or retrieval surfaces the wrong passage, the model will faithfully answer from bad input. RAG narrows the model's room to hallucinate; it does not guarantee truth. Post 5 is the full catalogue of how this goes wrong.

The summary is the four sentences to remember if you forget everything else. The LLM is the reasoning engine; RAG's job is to put the right facts in front of it. You retrieve relevant chunks, then generate. It is a multi-stage pipeline you own and tune. And you update its knowledge by updating documents, not by retraining.

If those four hold in your head, the rest of the day is detail. The next post explains why this architecture matters so much in practice — the specific, expensive problems it solves that a bigger or better model simply cannot.

The cover for the concept post and this cta bracket the introduction. We've established what RAG is and the vocabulary around it. The next post turns to motivation: why teams adopt RAG instead of just reaching for a stronger model or a fine-tune.

The teaser points at the central argument of post 2 — that most 'the AI is wrong' problems are really 'the AI didn't have the facts' problems, and that retrieval fixes a whole cluster of issues at once.