Evaluating RAG Quality

This cover sets expectations: the post is a working harness, not a lecture. You'll build evaluation twice — once from scratch so the mechanics are transparent, then with Ragas so you have the production-grade version — because seeing the naive implementation first is what makes the library legible rather than magical.

The through-line is the same loop every time: assemble rows of (question, contexts, answer, ground_truth), score retrieval and generation, aggregate, and re-run on every change. By the end you have a regression test for answer quality.

The eval dataset is the foundation everything else stands on, and this slide stresses curation over quantity-for-its-own-sake. Each row is a question paired with the gold answer you authored. The comment is the real advice: source 50 to 200 of these from actual production queries plus deliberate edge cases, and version the set in git so it evolves with the system.

Treat this dataset as a first-class asset. It is your definition of 'good' for this system. Every time a bug escapes to production, the fix isn't just the code change — it's adding that failing case to the eval set so the bug can never silently return.

This slide captures the one structural requirement of RAG evaluation that distinguishes it from plain LLM evaluation: you must capture the retrieved contexts, not just the final answer. run_rag returns both the chunks and the answer, because without the chunks you cannot score retrieval or check grounding.

The prompt deliberately instructs 'answer ONLY from context.' That constraint is what makes faithfulness measurable — you're explicitly asking the model to stay grounded, so an ungrounded answer is a real violation rather than expected behavior. Capturing context is the single most common thing teams forget when they bolt evaluation onto an existing pipeline.

The from-scratch retrieval scorer demystifies context recall. It lowercases the retrieved text, pulls the meaningful words from the ground truth (skipping short stopword-like tokens), and counts how many appear in the context. Average across the suite and you have a recall number you fully understand because you wrote every line of it.

This is intentionally crude — substring matching, not semantic matching — and that's the teaching value. You can see exactly what it rewards and where it would fail (synonyms, paraphrase). When you later switch to an embedding- or LLM-based matcher, you'll know precisely which limitation you're buying your way out of.

The faithfulness function shows the LLM-judge pattern in its simplest honest form: hand the judge the context and the answer, ask whether every statement in the answer is supported, and demand a numeric score in JSON. The escaped quotes in the prompt are there because the judge must return parseable JSON, not prose.

In production you'd decompose the answer into atomic claims first, as the 'how' post described, and score each — this whole-answer version is the readable starting point. The non-negotiable detail is using a separate, pinned judge model at temperature zero, so the score is a property of the answer and not of the judge's mood that run.

This slide swaps the hand-rolled metrics for Ragas, the standard library for LLM-judged RAG evaluation. The payoff is that the four headline metrics — faithfulness, answer relevancy, context precision, context recall — come calibrated and battle-tested out of the box, and the dataset format is exactly the four-key rows you've been building.

The lesson of building it from scratch first is now obvious: Ragas isn't a black box, it's a polished version of the loops you already wrote. You should still understand each metric, because the library reports numbers and you remain responsible for interpreting them and catching when a metric is misleading you.

The cycle diagram captures the rhythm of evaluation as an ongoing practice, not a one-time audit. Build the eval set, run RAG capturing contexts, score retrieval and generation, compare against the baseline — then back to the top on the next change. The loop never closes; it runs on every meaningful edit.

The 'compare vs baseline' node is the one that turns metrics into decisions. A score in isolation is trivia; a score relative to the last known-good version is a verdict on whether your change helped or hurt. Storing baselines per version is what makes the comparison possible.

This is where evaluation earns its keep: a CI gate. THRESHOLDS encodes your minimum acceptable faithfulness and recall; gate raises a failure if any metric falls below its floor. Wired into CI, a pull request that quietly drops recall can no longer merge — the build goes red.

The critical design choice is failing the build, not just logging the numbers. Logged metrics get ignored; failed builds get fixed. This single step is what converts evaluation from a dashboard people glance at into a guardrail that actually prevents regressions from shipping.

The pipeline diagram summarizes the harness as four stages: dataset, run, metrics, gate. It's the same shape as a unit-test pipeline — fixtures, execution, assertions, pass/fail — which is exactly the mental model to adopt. RAG evaluation is testing, just with fuzzy, model-scored assertions instead of exact equality.

Framing it this way helps you integrate it with how your team already works. The eval set lives in the repo, the harness runs in CI, and a regression is a red build with a diagnosable cause, because the metrics stage reported retrieval and generation separately.

The tips translate the harness into operational discipline. Seed the eval set from real queries so you measure reality, not your imagination. Pin the judge model and use temperature zero so scores are reproducible. Track scores per version so comparisons are honest. Fail CI rather than merely logging, so regressions actually get blocked. And grow the suite with every bug, so it gets stronger over time.

Each tip closes a common failure of homegrown evaluations. Together they're the difference between a harness that looks impressive once and one that keeps a production RAG system honest for years.

The CTA bridges to the failure modes. You now have a working harness, which means you're now exposed to the subtler danger: a harness that runs cleanly and reports numbers you shouldn't trust. The next post is the field guide to those traps.

Save this post as your implementation reference. When you sit down to instrument a real pipeline, these six steps are the skeleton you'll fill in with your own retriever, model, and thresholds.