Evaluating RAG Quality

This cover delivers the post's contrarian thesis: an all-green eval dashboard should raise suspicion, not relief. Green metrics you don't fully understand manufacture false confidence, which is more dangerous than no metrics, because it stops you from looking closer at exactly the moment you should.

The post is a field guide to the specific, repeatable ways RAG evaluations lie — small datasets, fluency-not-grounding metrics, blind judge trust, leakage, and blended scores — and the concrete fix for each. These are the mistakes that turn a quality system into theater.

The small-dataset mistake is first because it's the most common and the most seductive. Ten questions feel like a test but are statistically meaningless; a 20% swing on ten rows is two examples flipping, well within noise. You'll celebrate improvements and miss regressions because the set can't resolve them.

The fix is a curated set of 50 to 200 examples that deliberately spans common queries, known edge cases, and every past failure. The 'grow it on every escaped bug' habit is what keeps it representative over time — your eval set should accrete scar tissue from real incidents, not stay frozen at launch.

This mistake targets a whole class of legacy metrics. BLEU, ROUGE, and raw embedding similarity to a reference reward answers that resemble the gold text, which is not the same as answers grounded in the retrieved evidence. A fluent hallucination that happens to use similar vocabulary scores beautifully.

The fix is non-negotiable: include a faithfulness metric that ties each claim back to the retrieved context. Surface similarity can be a supplementary signal, but it must never be the primary one, because the failure mode you most need to catch — confident, well-phrased fabrication — is precisely what similarity metrics reward.

Blind faith in the LLM judge is the trap that grows as judges become standard. Judges are themselves models with biases: they drift with temperature, systematically prefer longer or more confident answers, and disagree with themselves across runs. Treating their verdicts as ground truth bakes those biases into your quality metric.

The fix is to validate the judge like any model. Label a few hundred examples by hand, measure the judge's agreement with humans, and only trust it where agreement is high. Pin the model and temperature for reproducibility. A calibrated judge is a tool; an unaudited judge is a confident, biased oracle you've mistaken for truth.

The mindmap organizes the failure space into three buckets — Data, Metric, and Judge — so the mistakes are memorable as categories rather than a flat list. Data problems are too-small sets, answer leakage, and missing edge cases. Metric problems are confusing fluency with grounding and collapsing to one blended score. Judge problems are being uncalibrated, length-biased, and high-variance.

This taxonomy is useful diagnostically. When an eval result smells wrong, walk the three branches in order: is the data representative and clean, is the metric measuring grounding, is the judge trustworthy? Most broken evaluations fail on at least one branch, and the branch tells you where to look.

Data leakage is the mistake that produces the most flattering and most worthless numbers. If the gold answer text sits inside the retrieved context, recall and faithfulness are trivially perfect for reasons that won't hold in production. The subtler version is tuning your retriever on the same questions you evaluate on — the classic train-on-test sin.

The fix imports a discipline from general ML: keep a held-out test set the system never trains or tunes on. Your evaluation must simulate production conditions, where the answer is not conveniently pre-placed in the context. Leakage makes the dashboard green and the launch a surprise.

The single-number mistake is the same error the concept post warned against, now framed as a reporting sin. A lone 'RAG score' tells you motion without direction. You see it dropped; you have no idea whether retrieval or generation caused it, so you can't act.

The fix is to keep retrieval and generation as separate columns permanently. The diagnostic power is in the combination: faithfulness down with recall steady implicates the prompt or model; recall down implicates chunking or the index. Blend them and you destroy exactly the signal that would have told you what to fix.

The compare diagram puts a fragile evaluation beside a trustworthy one, trait by trait, so you can audit your own setup. Fragile: ten cherry-picked questions, similarity to a reference, one un-audited judge, a single blended score. Trustworthy: a hundred-plus curated held-out rows, claim-level faithfulness, a calibrated and fixed judge, and retrieval and generation reported separately.

Use this as a checklist. If your evaluation matches the left column on even one row, that's a thread to pull. The right column isn't aspirational perfectionism; it's the minimum bar for an evaluation whose green is actually meaningful.

This mistake is the one teams discover last and feel worst about. A well-built RAG system must be able to say 'I don't know' when retrieval genuinely fails to surface the answer. But if your evaluation only rewards producing an answer, you're implicitly training the system to bluff — to fabricate something rather than abstain.

The fix is to score abstention as correct when retrieval truly failed. This aligns the metric with the behavior you actually want: confident answers when grounded, honest refusals when not. An eval that punishes 'I don't know' optimizes directly for confident wrongness, which is the exact failure mode RAG was supposed to reduce.

The code operationalizes two of the most important guards. The first asserts no answer leakage — the gold answer must not appear verbatim in the retrieved context — turning a subtle data sin into a hard, automated check. The second measures judge stability by scoring the same input five times and asserting the spread stays small, catching a noisy judge before you trust its verdicts.

These two assertions belong in your harness permanently. Leakage and judge variance are the failures most likely to make a broken evaluation look healthy, and both are cheap to test for. A few lines here save you from a dashboard that's confidently, quantitatively wrong.

The tips consolidate the post into a buildable checklist: a hundred-plus held-out rows, faithfulness over surface similarity, a judge calibrated against humans, no answer leakage, retrieval and generation reported apart, and credit for a correct 'I don't know.' Each line is the direct antidote to one of the six mistakes.

The meta-lesson is that a trustworthy evaluation takes deliberate engineering — it does not emerge by default from any library. The library gives you metrics; this checklist gives you metrics you can believe. The gap between those two is where most RAG evaluations quietly fail.

The CTA closes the day and points to the next building block in the series. You've now seen what RAG evaluation is, why it matters, how the metrics compute, how to build the harness, and how the whole thing gets faked into a misleading green dashboard. That's a complete, defensible foundation for measuring RAG quality.

Save the day as a set. The concept anchors the model, the stakes motivate it, the mechanics let you build it, the code makes it real, and these mistakes keep you honest about whether your numbers mean anything at all.