What is RAG?

Post four is the hands-on lab. The previous posts built the theory; now we run real, copy-pasteable code to build a working RAG pipeline end to end. Everything here uses sentence-transformers for embeddings, Chroma as the vector store, and an OpenAI chat model for generation — a stack that runs on a laptop and mirrors what production systems actually use.

We'll load and chunk a document, embed and store the chunks, retrieve the top matches for a question, assemble a grounded prompt, generate a cited answer, and finally return structured citations. By the end, the two-pipeline architecture from post three will be something you've built with your own hands rather than just read about.

This first block handles the start of the indexing pipeline: install the dependencies, load a document, and chunk it. We read a plain text file — here a company handbook — and split it with the same overlapping-window function from the previous post. The step size is the chunk size minus the overlap, so consecutive chunks share content and a fact sitting near a boundary appears intact in at least one chunk.

Printing the chunk count grounds the abstraction in a real number and is a useful sanity check: if a 50-page handbook produces three chunks or three thousand, something is off with your sizing. In a real system you'd load from many sources and split on paragraph or heading boundaries rather than raw character offsets, but this simple version is enough to make the whole pipeline run, and the rest of the lab builds directly on these chunks.

This block completes indexing: embed every chunk and store it. We load the all-MiniLM-L6-v2 embedding model — small, fast, and a sensible default for learning — and create a Chroma collection to hold the vectors. The add call stores three parallel things for each chunk: a stable id, the original document text, and the embedding vector produced by encoding that text.

Storing the original text alongside the vector is what lets us return real content later instead of just numbers, and the ids are what make citation possible downstream. Printing the collection count confirms everything landed. This is the entire offline half of RAG complete in a handful of lines — the searchable index now exists, and crucially this code touched only the embedder and the store, never the LLM. Indexing is upstream of generation.

This block opens the querying pipeline with the retrieve step. The function embeds the incoming question with the same embedder used for indexing — reusing the exact same model object is what guarantees the query vector lives in the same space as the stored vectors. It queries Chroma for the k nearest chunks, then zips each returned chunk's text together with its id so we can cite the source later.

The loop printing each id and a text preview is there so you can see retrieval actually working — eyeball whether the chunks coming back are genuinely relevant to the question before you ever involve the LLM. This habit is invaluable: a huge fraction of RAG debugging is just looking at what retrieval returned, because if the right chunk isn't here, no prompt downstream can fix it. Carrying the ids through from this step is what makes the citations in later slides possible.

This block assembles the grounded prompt and generates the answer — the payoff of the whole pipeline. We call retrieve to get the top chunks, then build a context block where each chunk is prefixed with its id in brackets, like [chunk-7]. That bracketed id is the trick that makes citation work: the model can reference it directly in its answer. The system message enforces the three grounding rules from post two — answer only from the context, cite the ids you use, and admit when something is unknown.

We send the system instruction and the context-plus-question as a chat completion and return the model's reply. This is where retrieval and generation finally meet. Notice how thin the LLM-specific code is — most of the work was in indexing and retrieval. The model is doing the easy part: reading supplied text and writing a grounded answer. The hard part, getting the right text in front of it, already happened upstream.

The pipeline diagram frames what the code in this post actually does, compressed into four stages. A document is chunked. The chunks are embedded and stored in the vector database. At query time, the top-k relevant chunks are retrieved. And the model generates a cited answer from them.

This four-stage view is the skeleton of essentially every RAG system, from this laptop demo to a production stack serving thousands of queries. The implementations differ enormously in sophistication — better chunking, re-ranking, hybrid search, caching — but the stages are the same. Holding this picture in mind helps you locate any problem in the right place: a wrong answer is a retrieval-stage issue or a generation-stage issue, and knowing which saves you from debugging blindly.

This block demonstrates the two behaviors that prove your RAG system actually works. The first question has its answer in the documents, and the model returns it with a citation — '20 paid vacation days per year [chunk-7]' — so the claim is traceable to a specific source chunk. The second question asks for something the documents don't contain, and instead of fabricating an answer, the model refuses: 'I don't know based on the provided documents.'

That refusal is the single most important behavior to verify, because it's the difference between a grounded system and a confident liar. A RAG system that always produces a fluent answer is suspicious — it should decline when the evidence isn't there. If yours never refuses, the grounding instruction from the previous slide isn't doing its job, and you're back to the model guessing from its weights, which the final post treats as a top mistake.

This final block wraps the answer with structured citations, turning a free-text response into something a UI can render and a user can verify. The answer_with_sources function calls the existing answer function and pairs the generated text with the list of chunk ids that retrieval surfaced, returning both in a dictionary. A frontend can now show the answer with clickable source links back to the original chunks.

Returning sources as structured data, separate from the prose, is what makes RAG auditable in practice — exactly the trust feature from post two. Note this version returns all retrieved chunk ids; a more refined system would parse which ids the model actually cited in its text and return only those, so the displayed sources match the claims. Either way, carrying ids through the whole pipeline from retrieval to response is what makes verifiable answers possible at all.

These four takeaways distill what the code demonstrated. Indexing and querying are genuinely separate functions in the code, mirroring the two-pipeline architecture — the index is built once, queries run against it repeatedly. The same embedder object encodes both documents and queries, which is non-negotiable for retrieval to work. Chunk ids are threaded through every step so the final answer can cite its sources. And the system prompt is what enforces grounding and the refusal path.

If you ran the snippets, you saw all four directly: you built the index separately from the query function, reused one embedder, watched ids flow through to citations, and saw the model refuse a question it couldn't ground. That hands-on confirmation is worth far more than taking the architecture on faith, and it's the foundation for diagnosing the failures in the final post.

These are the easiest ways to break a RAG pipeline in code, each a preview of the deeper treatment in the final post. Using different embedding models for indexing and querying puts the vectors in incompatible spaces, so 'nearest' returns noise. Forgetting overlap when chunking can split an answer across two chunks so neither retrieves cleanly. Setting k so large that the assembled context overflows the model's window truncates your own evidence. And omitting the refusal instruction lets the model abandon the context and guess freely from its weights.

Each of these is easy to fall into and easy to prevent once you've built the pipeline by hand and seen where the pieces connect. That's the real value of writing this code at least once: the failure modes stop being abstract warnings and become things you understand mechanically, which makes them far easier to spot and fix.

That's the lab complete. You've installed the stack, loaded and chunked a document, embedded and stored the chunks, retrieved the top-k relevant pieces, assembled a grounded prompt, generated a cited answer, verified the refusal behavior, and returned structured citations — a full RAG pipeline, built and run.

The final post steps back to the production view: the recurring RAG mistakes that quietly wreck retrieval quality — bad chunking, mismatched embedders, the wrong k, missing grounding instructions, and never evaluating retrieval — along with the concrete fix for each, so the system you just built actually holds up in the real world.