What is RAG?

Post five collects the failure modes. After the concept, the stakes, the mechanism, and the code, this is the practical payoff: the specific, repeatable mistakes that turn RAG from a feature into a source of confidently wrong answers. The unifying theme is liberating once you internalize it — almost none of these failures are the language model's fault. They're retrieval failures, and retrieval is something you control.

Learning these from a carousel is dramatically cheaper than learning them from a production incident where your bot cited a policy that doesn't exist, or quietly answered from stale training memory while everyone assumed it was grounded.

The first and most common mistake is chunking that splits the answer. If a chunk boundary falls in the middle of the sentence or paragraph that contains the answer, the fact ends up smeared across two chunks. Neither chunk embeds as a clean representation of the answer, so neither retrieves strongly for the question, and the model never sees the complete fact even though it's right there in your documents.

The fix has three parts. Use a sensible chunk size matched to your content. Add an overlap between consecutive chunks so a fact near a boundary survives intact in at least one of them. And, most importantly, split on natural boundaries — paragraphs, sections, headings — rather than blind character counts that don't respect the structure of the text. Good chunking is upstream of everything; get it wrong and no downstream tuning can compensate.

Mistake two is mismatched embedding models, and it's insidious because the system still runs — it just returns garbage. Embeddings only carry meaning relative to the model that produced them. Documents embedded with one model and queries embedded with another live in different vector spaces, so the geometric notion of 'nearest' becomes meaningless and retrieval returns essentially random chunks.

The classic way this happens isn't using two obviously different models on purpose — it's subtle drift. You upgrade the embedding model and re-index the documents but forget to update the query path, or vice versa. Or two parts of the codebase each instantiate their own model. The result is a retrieval system that quietly degrades to noise. The discipline is to treat the embedding model as a single shared dependency used identically at index time and query time, and to re-index everything whenever you change it.

This fix operationalizes the single-embedder discipline. We define one module-level EMBEDDER and a single embed function that everything calls — there is one place where text becomes a vector, and it's used at both index time and query time. No part of the system instantiates its own model, which eliminates the accidental-drift failure from the previous slide.

The pattern matters more than it looks. A single source of truth for embedding means that when you do deliberately upgrade the model, there's exactly one line to change, and it's immediately obvious that you must re-index all documents to match. Scattering model instantiation across the codebase is how teams end up with index and query vectors silently out of sync. Centralizing it makes the constraint — same model, both sides — structurally enforced rather than something you have to remember.

Mistake three is choosing the wrong k, the number of chunks you retrieve. It's a balance with a failure mode on each side. Too small a k and the chunk holding the answer can fall just outside the cut, so the model answers from incomplete evidence or refuses despite the answer existing in your store. Too large a k and you flood the prompt with marginally-relevant chunks, which raises token cost and latency and — through the lost-in-the-middle effect — can lead the model to fixate on the wrong passage and produce a worse answer than a tighter retrieval would.

There's no universal right value. A reasonable starting point is around k=4, but the correct k depends on your chunk size, how your documents are structured, and the nature of your questions. The only honest way to set it is empirically: tune k against a set of real questions and measure which value retrieves the right text most reliably, which is exactly what the evaluation slide enables.

The decision tree gives you a triage procedure for the most common symptom: a wrong RAG answer. The first and most important question is whether the right chunk was even retrieved. If not, the problem is upstream — fix your chunking, your embeddings, or your k, because no prompt can save you when the evidence never reached the model. If the right chunk was retrieved, ask whether the prompt forced grounding. If it didn't, the model likely ignored the context and guessed from its weights — add the grounding and refusal instruction.

Only when the right chunk was retrieved and the prompt did force grounding, and the answer is still wrong, should you suspect a genuine model or reasoning limitation. This ordering is the whole point: it sends you to the cheap, fixable retrieval causes first, instead of blaming the LLM and prematurely reaching for a bigger model when the real bug is in your pipeline. Most RAG failures resolve at the very first branch.

Mistake four is shipping without a grounding or refusal instruction. If you simply paste the retrieved context into the prompt and ask the question, nothing stops the model from blending the context with its parametric memory — or ignoring the context entirely and answering from its weights. When the context doesn't contain the answer, the model does what models do: it produces a fluent, confident guess. That silently undoes the entire point of RAG, because now you have an ungrounded answer wearing the costume of a retrieved one.

The fix is two explicit rules in the system prompt: answer using only the provided context, and say 'I don't know based on the documents' when the context lacks the answer. The refusal path is the critical half — a RAG system that can't say 'I don't know' will fabricate, and a system that fabricates is worse than no system because it's confidently wrong in a way users trust. Pin both rules and verify the refusal actually fires on out-of-scope questions.

Mistake five is the meta-mistake that hides all the others: never evaluating retrieval. Teams pour energy into prompt wording and model choice while treating retrieval as a black box that 'probably works.' But since most RAG failures are retrieval failures, this is exactly backwards — you're polishing the part that's rarely the problem while ignoring the part that usually is.

The fix is a lightweight evaluation set: a handful of representative questions, each paired with the id of the chunk that actually contains the answer (the 'gold' chunk). Then you measure recall — for each question, did the gold chunk appear in the top-k retrieved? This single number tells you whether your retrieval is even capable of supporting good answers. You cannot improve what you don't measure, and recall@k is the cheapest, most diagnostic metric in RAG. Build this before you tune anything else.

This snippet makes retrieval evaluation concrete. The recall_at_k function takes an eval set of question-and-gold-chunk pairs, runs retrieval for each question, and checks whether the gold chunk id appears in the retrieved ids. It returns the fraction of questions for which retrieval surfaced the right chunk — recall@k. The eval set is just a list of tuples you author by hand from real questions you expect users to ask.

This is deliberately simple, and that's the point: a dozen lines give you an objective signal that turns RAG tuning from guesswork into measurement. Now when you change chunk size, swap the embedding model, or adjust k, you can see whether recall went up or down instead of eyeballing a few answers and hoping. Production systems extend this with larger eval sets and answer-quality metrics on top, but recall@k is the foundation, and it directly tells you whether the retrieval half of your system is doing its job.

The summary table pairs each mistake with its fix for quick recall. Chunks that split the answer are fixed by overlap and smart boundaries. Two different embedders are fixed by using one embedder everywhere. A badly-chosen k is fixed by tuning it against real queries. A model free to guess is fixed by forcing grounding and refusal. And never measuring retrieval is fixed by tracking recall@k.

The through-line across all five is that these are engineering problems with engineering solutions, sitting almost entirely in the retrieval pipeline rather than in the model. Internalizing this mistake-to-fix mapping means that when a RAG answer goes wrong in production, you reach for the right diagnosis quickly — check retrieval first — instead of flailing or blaming the LLM for a bug that lives in your own chunking, embeddings, or prompt.

This final checklist is the operational distillation of the entire day. Chunk on natural boundaries with overlap so answers stay intact and retrievable. Embed documents and queries with the same model so the vector space is coherent. Tune k against real questions rather than dumping everything into the prompt. Instruct the model to answer only from the context and to refuse when the answer isn't there. And measure recall@k before you ever blame the model for a wrong answer.

Five practices, each cheap to adopt, each preventing a class of failure that routinely surprises teams shipping RAG. Pin this list to your RAG pipeline and you've eliminated most retrieval-related incidents before they happen — the difference between a demo that impresses and a system that holds up in front of real users with real questions.

That completes Day 67. You can now reason about RAG the way an engineer does: as a two-pipeline architecture that grounds a frozen model in retrieved text, with quality determined by chunking, embeddings, retrieval, and grounding — and failures diagnosed by checking retrieval before blaming the model.

From concept to stakes to mechanism to code to mistakes, you've seen the full picture. This is the foundation that makes everything about production LLM applications make sense, because so much of shipping useful AI on real data is, at bottom, the art of retrieving the right text and getting the model to actually use it.