The RAG Pipeline

The cover names the trap that makes RAG bugs so insidious: a demo that works on three documents can fall apart on real data, and the model still answers confidently the whole way down. Confidence masks the failure.

The framing is that this post is a debugging checklist for the situation where RAG 'works' — it runs, it returns fluent answers — but the answers are wrong. That's a different and harder problem than a crash, because nothing signals that anything is broken.

Mistake one is bad chunk size, the most common root cause of poor retrieval. Oversized chunks dilute the signal: the one relevant sentence is averaged in with paragraphs of unrelated text, so the chunk's embedding doesn't sit close to the query and similarity scoring blurs. Undersized chunks lose the context that makes a fact meaningful — a bare number with no subject.

The fix is to aim for coherent passages of a few hundred tokens with modest overlap, then tune against real queries. Because chunking is the offline stage that decides what can ever be retrieved (post 3), getting it wrong caps the quality of everything downstream no matter how good your model is.

Mistake two is mismatched embedding models, a silent and total failure. Embeddings only have meaning relative to the model that produced them; vectors from two different models live in incompatible spaces. If you index your chunks with one model and embed queries with another, similarity scores become essentially random and retrieval returns noise.

This bug often creeps in during upgrades — someone bumps the embedding model for indexing but the query path still uses the old one, or vice versa. The fix is to pin the embedding model name and version in config and assert that indexing and querying use the same one. When retrieval suddenly returns garbage after a deploy, check this first.

Mistake three is trusting top-k blindly. Vector search is a ranking, not a filter: it always returns the k nearest chunks, even when nothing in your corpus is actually relevant. Ask a question your data can't answer and you still get the k 'least irrelevant' chunks — and a model told to answer from context will dutifully answer from junk.

The fix is a similarity threshold. Read the actual similarity scores and drop any hit below a cutoff you've tuned. If everything falls below the threshold, retrieval returns empty, which should trigger the 'I don't know' path. Without a threshold, retrieval can never say 'I found nothing,' and that gap is where a lot of confident wrong answers come from.

This bars diagram visualizes the 'lost in the middle' effect, a well-documented behavior of long-context models. They use information at the start and end of the context window far more reliably than information buried in the middle. The bars show high utilization at the start and end and a deep dip in the middle.

The implication for RAG is direct: stuffing many chunks into the prompt doesn't linearly improve accuracy, because a critical chunk landing in the middle of a long context may be effectively skimmed over. This is the evidence behind the next slide's advice to retrieve broadly but pass only a few, well-ordered chunks.

Mistake four follows from the diagram: stuffing the context with chunks in the belief that more information means more accuracy. Because of the lost-in-the-middle effect, a critical chunk buried at position 9 of 15 can be effectively ignored, even though it's right there in the prompt.

The fix is the retrieve-broadly, rerank, then trim pattern from post 3: pull a wide candidate set, use a reranker to push the truly relevant chunks to the top, and pass only the best few to the model. Fewer, better-ordered chunks beat many unranked ones — both for accuracy and for cost, since you're sending fewer tokens.

Mistake five is shipping without an escape hatch. If the system prompt doesn't explicitly tell the model to say 'I don't know' when the answer isn't in the context, the model defaults to its trained behavior of producing a fluent, plausible completion — which means it fills the gap with a guess. Combined with retrieved chunks and citations, that guess looks authoritative, producing confident, cited-looking nonsense.

The fix costs one sentence in the system prompt: instruct the model to answer only from the context and to reply 'I don't know' otherwise. This is the single cheapest, highest-impact reliability change in a RAG system, and it should be present from the very first version, not added after the first embarrassing wrong answer.

This snippet implements the threshold fix from mistake three. It scores every chunk against the query, keeps only those at or above min_score, sorts the survivors, and returns the top k. The key behavior is in the comment: if nothing clears the threshold, the function returns an empty list — which is the signal that retrieval found nothing relevant.

The min_score value (0.35 here) is illustrative and must be tuned against your own data and embedding model; the right cutoff varies. The point is that retrieval now has a way to say 'I found nothing,' which is the precondition for an honest 'I don't know' instead of an answer assembled from irrelevant chunks.

This snippet wires the empty-retrieval case to an explicit refusal, implementing both the threshold and the escape hatch together. If retrieve() returns no chunks, the code short-circuits and returns 'I don't have that information in my sources' before the model is ever called — saving a wasted LLM call and guaranteeing no fabrication.

The SYSTEM string then enforces the same rule at the generation level for the case where chunks were found but may not actually contain the answer. Belt and suspenders: the code-level check handles total misses, and the prompt-level instruction handles partial ones. Together they close most of the confident-wrong-answer surface area.

This compare slide maps symptoms to their usual causes, which is how you actually debug a RAG system in practice — you observe a symptom, then check the likely cause. A confident wrong answer usually means no grounding rule. An answer that ignores the document points to a chunk that's too big or lost in the middle. An irrelevant citation suggests a retrieval miss with no threshold. Missing obvious facts points back to bad chunking or the wrong embedder.

The meta-lesson is that the symptom is almost never 'the LLM is dumb.' It's a pipeline-stage problem — retrieval, chunking, prompting — and this table is a quick lookup to jump from what you see to where to look. It ties together every mistake in the post.

The tips slide gives the debugging procedure in order, which is itself the most important habit. First, inspect the retrieved chunks before blaming the LLM — most of the time the answer simply wasn't in the context. Log similarity scores and add a threshold so misses surface. Fix chunking first, since it's the usual culprit and caps everything downstream. Keep context tight by reranking then trimming. And always ship an 'I don't know' path.

Following this order saves enormous time, because the instinct to swap models or rewrite prompts targets the last stage of the pipeline when the bug almost always lives earlier. Debug in pipeline order, earliest stage first, and you'll find most issues fast.

The cta closes both the post and the day. The reader now has the full arc: what RAG is, why it matters, how the pipeline works, how to build it, and how it fails. That's a complete, practical understanding of the RAG pipeline.

The teaser notes that a new category begins tomorrow, signaling the end of this topic and inviting the reader to follow along for what comes next in the 100 Days series.