Evaluating RAG Quality

This cover frames the entire day. RAG systems demo beautifully because a curated question against a curated corpus almost always works. The trap is mistaking that demo for evidence of quality. The moment the system meets real, messy traffic, the absence of measurement becomes the central problem: you cannot tell whether a wrong answer came from bad retrieval, bad generation, or both.

The goal of this post is to install one mental model before any metric: RAG is two systems, and evaluation must address both. Everything else in the day builds on that split.

The core correction here is cultural, not technical. Teams routinely judge RAG by reading a handful of outputs and deciding they 'look smart.' That is sampling bias plus confirmation bias: you read the answers you expected to read, on the questions you thought to ask. Evaluation replaces that with a fixed, repeatable measurement you can run on every change.

The practical payoff is comparability. Once you have numbers tied to a versioned dataset, an improvement and a regression become distinguishable events rather than feelings. That is the entire point of evaluation: turning 'it feels better' into 'context recall went from 0.78 to 0.86 on the same 120 questions.'

This is the most important idea in the day. A RAG answer is produced by two distinct components with independent failure modes. The retriever can surface the wrong chunks; the generator can ignore the right chunks. Treating the pipeline as one black box tells you only that the final answer was bad, never which stage caused it.

The 'one coat' metaphor matters because the fix differs entirely by stage. A retrieval failure is fixed in chunking, embeddings, the index, or k. A generation failure is fixed in the prompt, the model, or grounding instructions. Diagnose the wrong stage and you burn weeks tuning a component that was never broken.

The pipeline diagram lays out where quality is created and lost: a query enters, retrieval fetches chunks, generation writes from them, and a judge scores both the answer and the sources it used. Each arrow is a place a measurement can attach.

Reading it left to right also explains why order matters in debugging. If the 'retrieve' stage hands the generator no relevant chunks, no amount of prompt tuning at the 'generate' stage will save the answer. Conversely, perfect retrieval still fails if generation ignores the evidence. The judge stage exists precisely to attribute blame correctly.

These four questions are the backbone of every serious RAG evaluation suite. Recall and precision describe the retrieved context: did we get the needed evidence, and was the rest mostly relevant rather than noise. Faithfulness and answer relevance describe the generated answer: is it grounded in that evidence, and does it actually respond to the user.

Keeping all four distinct is what makes diagnosis possible. High recall with low faithfulness means the model is ignoring good context. High faithfulness with low recall means the model is honestly summarizing context that simply lacks the answer. Each combination points to a different fix.

This comparison sorts the metric zoo into two columns so you never confuse a retrieval problem for a generation problem. The left column — context recall, context precision, hit rate, MRR, nDCG — judges only the retriever and the ranking. The right column — faithfulness, answer relevance, correctness, citation accuracy — judges only the generated text.

The discipline of keeping these in separate columns on your dashboard pays off constantly. When a number moves, the column it lives in immediately narrows the search space for the cause. Blending them into one headline figure destroys exactly the information you need to act.

This slide makes the cost of a single blended score concrete. A 70% end-to-end accuracy is useless as a diagnostic because two completely different systems produce identical headline numbers: one where retrieval misses 30% of evidence but generation is flawless, and one where retrieval is perfect but generation ignores context 30% of the time.

The lever is different in each case. Splitting the score is what reveals it. This is why mature teams almost never report a single RAG quality number without the component scores beside it — the component scores are where the decisions actually get made.

The code shows that an evaluation example is richer than a question-answer pair. It carries the ground truth, the exact chunks retrieval surfaced (including the deliberate noise chunk about shipping), and the generated answer. From this one structured row you can compute every metric in the day.

The deliberate noise line matters. Real retrieved context is never pure; it contains near-misses and off-topic passages. Including them in your eval rows is what lets precision and faithfulness metrics do their job — measuring whether the system stays grounded despite distractors, not just on a sanitized ideal.

This slide names the two families of metric you will mix in any real suite. Reference-based metrics compare the answer to a gold answer you authored in advance; they are precise and unambiguous but expensive, because every example needs a human-written truth. Reference-free metrics use an LLM judge to assess grounding and relevance with no gold answer; they scale cheaply but introduce judge noise.

The practical recipe is to use reference-based metrics on a smaller, high-quality core set and reference-free metrics for broad coverage on large, unlabeled traffic samples. Each compensates for the other's main weakness.

The recap compresses the whole post into a model you can recall under pressure. RAG evaluation equals retrieval quality plus generation quality, measured separately. Recall and precision judge retrieval; faithfulness and relevance judge the answer. A single blended score hides which half broke.

Carry this into the rest of the day. The 'why' post argues the stakes of doing this, the 'how' post computes each metric, the code post gives you a runnable harness, and the mistakes post shows how these same metrics get faked into a green dashboard that lies.

The CTA points forward to the stakes. Knowing what evaluation measures is necessary but not motivating on its own; the next post makes the case for why skipping it is dangerous — specifically how unmeasured RAG ships confident hallucinations straight to production.

Save this one as the conceptual anchor for the series. Every later post assumes you already hold the two-systems model introduced here.