What is RAG?

Post three is the engine room. Having defined RAG and argued why it matters, we now open up the actual machinery: the steps that turn your documents into a searchable index and turn an incoming question into a grounded answer. This is the most technical of the conceptual posts, and it's where the behavior of a RAG system stops being mysterious.

The single most important insight here is structural: a RAG system is two separate pipelines, not one. Indexing and querying run at different times, for different reasons, and most RAG bugs come from forgetting they're distinct. Get that division clear and the rest of the mechanics fall into place.

This slide names the two-pipeline structure that organizes everything else. Indexing is the offline pipeline: load your documents, chunk them, embed each chunk, and store the vectors. It runs once per document and again whenever a document changes — nobody is waiting on it, so it can be slow and thorough. Querying is the online pipeline: embed the incoming question, search the store, optionally re-rank the candidates, assemble the prompt, and generate. It runs on every request with a user waiting.

The only thing the two pipelines share is the embedding model — and that shared dependency is exactly where things break if you're careless. Indexing produces vectors with one model; querying must produce its query vector with the same model, or 'nearest neighbor' is meaningless. Holding the two pipelines and their one shared link in your head is the key to reasoning about any RAG system.

The pipeline diagram traces the indexing half end to end. Raw documents are loaded from wherever they live — files, a database, a CMS. They're chunked into smaller pieces. Each chunk is embedded into a vector by the embedding model. And the vectors, along with the original text and metadata, are stored in the vector database.

This pipeline is offline and idempotent: you can re-run it whenever content changes without affecting live queries. Its output is the searchable index that the querying pipeline depends on entirely. A weak link anywhere here — poor chunking, a mediocre embedding model — caps the quality of every answer the system will ever give, no matter how good the prompt is at query time. Indexing is where retrieval quality is born.

Chunking is the unglamorous step that quietly determines retrieval quality. Whole documents are usually too large to embed into a single meaningful vector and too large to drop into a prompt, so you split them into chunks — commonly a few hundred tokens each, with a small overlap between consecutive chunks. Each chunk becomes one independently retrievable unit.

The size is a genuine tradeoff. Chunks that are too large dilute the embedding — a single vector tries to represent too many ideas, so retrieval gets imprecise and you waste prompt space on irrelevant sentences. Chunks that are too small lose the surrounding context needed to make sense of a fact. The overlap exists so that an answer sitting near a boundary isn't sliced in half. Good chunking — ideally on natural boundaries like paragraphs or sections — is one of the highest-leverage things you can tune.

This slide explains the heart of semantic retrieval. During indexing, each chunk is embedded into a vector that encodes its meaning. At query time, the question is embedded by the exact same model into the same vector space. Similarity between two vectors is typically measured by cosine distance — essentially the angle between them — where a smaller angle means more similar meaning.

Because the embedding model places semantically related text near each other, the chunks whose vectors are closest to the query vector are the ones most likely to be relevant — even if they share no exact keywords with the question. This is the crucial advantage over old keyword search: 'how do I get my money back' can retrieve a chunk titled 'refund policy' without any shared words. The whole system's intelligence about relevance lives in this embedding-and-distance step.

The second pipeline diagram traces the querying half, which runs on every request. The incoming question is embedded by the same model used for indexing. The vector store is searched for the top-k nearest chunks. Those candidates are optionally re-ranked to sharpen their order. And finally the best chunks are assembled into a prompt and handed to the model to generate a grounded answer.

Unlike indexing, every stage here is latency-sensitive because the user is waiting. This is why the search step uses approximate nearest-neighbor algorithms rather than exact brute force, and why re-ranking is optional — it adds accuracy at the cost of time. The art of a fast, accurate RAG system is largely about tuning this pipeline: how many candidates to fetch, whether to re-rank, and how to pack the prompt.

Top-k retrieval is the step where you decide how much evidence to give the model, and k is one of the most consequential knobs in the whole system. The vector store returns the k chunks most similar to the query — often somewhere between three and eight. That number directly trades coverage against noise.

Set k too low and the chunk containing the answer might not make the cut, so the model answers from incomplete evidence or refuses. Set k too high and you flood the prompt with marginally-relevant text: this costs more tokens, adds latency, and — per the lost-in-the-middle effect — can actually make the model fixate on the wrong passage. There's no universally correct k; the right value depends on your chunk size, document structure, and questions, and the only way to find it is to measure against real queries.

Re-ranking is the optional second pass that fixes the cheap first pass's mistakes. Vector search is fast but approximate — it ranks chunks by embedding similarity alone, which is a coarse signal. A re-ranker, typically a cross-encoder model, takes the query and each candidate chunk together and scores how well that specific chunk answers that specific query, then keeps only the best few.

The difference matters because the embedding used for initial retrieval compresses each chunk into a vector independently of the query, losing nuance. A cross-encoder reads them jointly, so it catches relevance the bi-encoder missed. The standard pattern is to retrieve a generous candidate set — say the top 20 — cheaply by vector search, then re-rank down to the top 4 precisely. You get the speed of approximate search and the precision of joint scoring, which often produces a noticeable jump in answer quality.

This snippet shows the indexing pipeline as real, runnable code, making the offline half concrete. We load a sentence-transformer embedding model and create a Chroma collection. The chunk function splits text into overlapping windows — note the step is size minus overlap, so consecutive chunks share content and no answer gets cleanly severed at a boundary. Then we loop over the chunks, embedding each one and adding it to the store with an id.

This is the literal 'load, chunk, embed, store' pipeline from the diagram, in a dozen lines. In production you'd add smarter boundary-aware splitting, metadata, and batched embedding for speed, but the shape is exactly this. Run it once and you've built the searchable index that every future query will lean on. Notice it touches the embedding model and the store but never the LLM — indexing is entirely upstream of generation.

This snippet shows the querying pipeline as code, the online half that runs per request. The answer function embeds the question with the same model used during indexing — that consistency is non-negotiable. It queries the store for the k nearest chunks, joins them into a context block, and builds a prompt that includes the context and an instruction to answer using only that context. Then it calls the LLM and returns the grounded answer.

This is the 'embed query, search, assemble, generate' pipeline made real. Compare it to the indexing snippet and the two-pipeline structure becomes obvious — they share only model.encode, the embedding step. Everything else is independent. The 'answer using only the context' line is the small but critical instruction that keeps the model grounded; drop it and the model drifts back to guessing from its weights, which the final post examines in depth.

These are the knobs that actually determine RAG quality, gathered in one place. Chunk size and overlap govern the precision-versus-context tradeoff in retrieval. The embedding model sets the ceiling on how well retrieval understands meaning — a better embedder lifts everything downstream. k controls the coverage-versus-noise balance in how much evidence reaches the model. A re-ranker buys precision on the shortlist. And the prompt template enforces grounding and the refusal behavior.

The reason to know this list is leverage. When a RAG system underperforms, these are the dials to turn, roughly in order of impact: most teams should fix chunking and the embedding model before fiddling with prompts, because a great prompt can't rescue bad retrieval. Knowing which knob addresses which symptom is what makes debugging systematic rather than guesswork — exactly the skill the final post sharpens.

That's the machinery. You now understand the two-pipeline structure, why and how documents are chunked, how embeddings and cosine similarity power semantic retrieval, what top-k controls, how re-ranking sharpens results, and how the grounded prompt is assembled — all the way to runnable indexing and querying code.

The next post makes every bit of this tangible: a complete, copy-pasteable RAG pipeline that loads a document, indexes it, retrieves the right chunks for a question, and returns a grounded, cited answer you can run and inspect yourself.