The RAG Pipeline

The cover signals that this is the mechanics post — the engine, not the pitch. The framing line is that RAG is a chain of stages and a weak link anywhere degrades the final answer, which is the lens to read the whole post through.

The practical payoff stated up front is that most quality problems originate in chunking and retrieval, well before the LLM is invoked. That tells the reader where to focus attention as the stages unfold.

This compare slide splits RAG into its two halves, which is the single most clarifying way to think about it. The offline half runs once (or whenever documents change): load, chunk, embed, store. The online half runs on every query: embed the question, search, rerank, build the prompt, generate.

The separation matters operationally. Offline work is batch, can be slow, and is amortized across all future queries. Online work is latency-sensitive and runs per request. Knowing which half a step lives in tells you where you can spend time (indexing) and where you must be fast (querying).

Stage one is loading and chunking, and the headline claim is that chunking drives retrieval quality more than any other single choice. You extract raw text from PDFs, HTML, or documents, then split it into pieces of a few hundred tokens with a little overlap between neighbors.

The tension is precision versus completeness. Chunks that are too large bury the relevant sentence among unrelated text, blurring similarity scores. Chunks that are too small strip away the context needed to make sense of a fact. Overlap helps ensure an idea split across a boundary still appears intact in at least one chunk. Post 5 covers the failure modes of getting this wrong.

Stage two is embedding: each chunk is passed through an embedding model that outputs a vector — a fixed-length list of numbers positioned so that texts with similar meaning land near each other in the space. This is what makes semantic search possible.

The crucial detail is that the very same model must later embed the query, so the question and its answer chunks map into the same space and end up close even when they share no exact words. 'How do I get my money back?' should land near a chunk about 'refund policy.' Using one model to index and another to query is a classic, silent bug — see post 5.

Stage three stores the vectors in a vector database alongside their original text and any metadata (source id, permissions, timestamps). The database builds an index — HNSW is the common choice — so that finding the nearest vectors is fast even across millions of chunks, instead of comparing against every vector one by one.

This store is your searchable knowledge base. The metadata stored next to each vector is what later enables features like access-control filtering and source citation. The index type and its parameters trade off recall against speed, but for most applications the defaults of a mature vector DB are fine to start.

This pipeline diagram is the full end-to-end view in one strip: chunk, embed, store, search, rerank, generate. The first three are offline; the last three are online. It's the visual spine of the post and worth memorizing.

Reading it left to right also reads as a debugging order. If answers are bad, walk the stages: did chunking produce sensible pieces, did embedding place them well, did search surface the right ones, did reranking order them correctly, and only finally — did generation read them well. The earlier you find the break, the cheaper the fix.

Stage four is retrieval. At query time you embed the question and ask the store for the k chunks whose vectors are nearest, usually by cosine similarity. k is a dial with a real tradeoff: too small and you miss the chunk that held the answer; too large and you flood the prompt with marginally relevant noise that distracts the model.

A typical starting range is k between 3 and 8, then tuned against your own evaluation set. A common pattern, set up by the next slide, is to retrieve a larger candidate set, rerank it, and then keep only the best few for the prompt — getting both recall and precision.

Stage five, reranking, is optional but often the highest-leverage upgrade. Vector search is fast but approximate: it compares the query and chunk embeddings independently, which can miss nuance. A reranker is a cross-encoder that looks at the query and a chunk together and scores their relevance directly, which is more accurate but too slow to run over the whole corpus.

The standard pattern is two-stage retrieval: use fast vector search to pull, say, 20 candidates, then run the slower, sharper reranker over just those 20 and keep the top 4 for the prompt. You get the recall of broad retrieval and the precision of a strong relevance model, without paying the reranker's cost over millions of chunks.

This snippet shows the offline indexing half in real code. An embedding model encodes the text of every chunk into vectors, and those vectors — with their chunk payloads — are added to the store. This runs once when documents are ingested, and again only when they change.

The key things to notice: split_documents produces chunks with a size and overlap (the choices from stage one), encode turns text into vectors (stage two), and store.add persists them with their payloads (stage three). Post 4 implements split_documents and the encoding concretely; here it's the shape that matters.

This snippet shows the online query half — the per-request path. It embeds the question, retrieves a broad top-20 from the store, reranks and trims to the best k, joins their text into a context block, builds a prompt, and generates. It's stages four, five, and six in one function.

Notice the two-stage retrieval baked in: top_k=20 then rerank(...)[:k]. That's the recall-then-precision pattern from the reranking slide. Everything that determines answer quality has already happened by the time llm.generate is called — which reinforces the post's thesis that the LLM is the last and often least of your concerns.

This tips slide states the post's core lesson plainly: quality is decided long before the model runs. Chunking decides what is even possible to retrieve. The embedding model defines what counts as 'similar.' Top-k and reranking decide which chunks actually reach the prompt. The LLM only ever sees what survived all of those stages.

The operational takeaway is to debug in pipeline order, earliest stage first. When an answer is wrong, inspect the retrieved chunks before you touch the prompt or swap the model. Most of the time you'll find the answer was never in the context to begin with — a retrieval or chunking problem, not a generation one.

The cta closes the mechanics post and points to the hands-on build. Having walked all six stages conceptually, the reader is ready to see them implemented as runnable code.

The teaser promises a full end-to-end build, which is exactly what post 4 delivers — chunking, embedding, similarity search, prompt assembly, and the wired-together ask() call.