Evaluating RAG Quality

This cover reframes the metric zoo as something simple: every RAG metric is counting overlaps among four objects — the question, the retrieved chunks, the answer, and the ground truth. Once you see that, the formulas stop being intimidating and start being obvious.

The post walks the two metric families in turn. Retrieval metrics relate the chunks to the truth and to relevance; generation metrics relate the answer to the chunks and to the question. Holding the four objects in mind makes every definition that follows fall into place.

Context recall answers the question that retrieval exists to satisfy: did we actually fetch the evidence needed to answer? Operationally, you decompose the ground-truth answer into the statements it requires, then check what fraction of those statements appear in the retrieved chunks. Recall is supported statements over required statements.

The key insight is that recall is an upper bound on the whole system. If the evidence isn't in the context, no generator, however good, can ground an answer in it. That's why low recall is the first thing to rule out when answers are wrong — it caps everything downstream.

Context precision is recall's counterweight. It asks: of everything you retrieved, how much was actually relevant? It penalizes the lazy strategy of cranking k sky-high to guarantee the answer is somewhere in the pile. That works for recall and quietly sabotages generation.

The reason precision matters is attention dilution. A context window stuffed with mostly-irrelevant chunks makes the model work harder to find the signal and increases the chance it latches onto a distracting passage. High recall with low precision is a haystack-with-needle problem, and it directly drives down faithfulness.

The compare diagram captures the recall-precision tension that governs all of retrieval tuning. Pushing recall up — raising k, broadening the search — guarantees you found the evidence but drags in noise. Pushing precision up — reranking, lowering k, tighter filters — gives a clean context but risks dropping an edge-case fact.

The right operating point depends on your generator and your domain. A strong model tolerates more noise, favoring recall; a weaker or stricter model wants clean context, favoring precision. The metrics let you find that point deliberately instead of guessing.

Ranking metrics exist because position matters: models attend most to the earliest chunks, so where the right chunk lands changes the answer. Hit Rate@k is the blunt instrument — is the answer anywhere in the top k. MRR sharpens it by rewarding the rank of the first correct hit, so first place beats fifth. nDCG is the most refined, weighting each relevant chunk by a log-discounted position.

The practical use is comparing rerankers and retrievers. Two systems can have identical Hit Rate@10 but very different MRR — one buries the right chunk at rank 8, the other surfaces it at rank 1. The second will produce better answers, and only a rank-aware metric reveals it.

Faithfulness is the central anti-hallucination metric, and the claim-decomposition method is how it stays honest. An LLM judge splits the answer into atomic claims, then verifies each independently against the retrieved context. Faithfulness is the fraction of claims the context supports.

The subtle but crucial point: a claim that is true in the world but absent from the context still counts as unfaithful. Faithfulness measures grounding, not correctness. This is deliberate — a RAG system that gets the right answer from its own parametric memory rather than the retrieved evidence is unreliable, because next time the memory will be wrong and there's no source to catch it.

The flow diagram traces the faithfulness computation end to end: take the free-text answer, decompose it into atomic claims, verify each claim against the context, then score as supported over total. Each box is a discrete, inspectable step, which matters when you need to debug why a score looks wrong.

Decomposition is the step people skip and regret. Scoring a whole multi-sentence answer as one yes/no loses resolution — a mostly-grounded answer with one fabricated detail looks the same as a fully fabricated one. Atomic claims give you the granularity to catch the single bad sentence.

Answer relevance closes a gap faithfulness leaves open: an answer can be perfectly grounded and still fail to address the question. Imagine asking about refund timing and getting a faithful, well-sourced paragraph about shipping. To measure this, a judge generates the questions the answer would naturally answer, then compares them to the real question by embedding similarity.

High similarity means the answer is on target; low similarity flags evasion, padding, or topic drift. Pairing answer relevance with faithfulness catches the two distinct ways a generation fails: ungrounded (faithfulness) and off-topic (relevance).

This code makes the retrieval metrics tangible. context_recall checks how many required facts appear in the retrieved text — a deliberately simple substring version of the real claim-matching idea. hit_rate_at_k just asks whether the gold document id is in the top k. Both run in a few lines because, at heart, these metrics are set membership.

The simplification is intentional for teaching; production versions use embeddings or an LLM to match facts semantically rather than by substring. But the structure is identical, and writing the naive version first is the fastest way to understand what the library is doing for you later.

This snippet shows the faithfulness loop concretely: for each atomic claim, prompt an LLM judge with the context and the claim, demand a strict-JSON supported verdict, and tally the fraction supported. The strict-JSON contract matters — free-form judge output is a parsing nightmare, so you constrain it to a machine-readable shape.

Two production notes hide in this code. First, pin the judge model and set temperature to zero so the same input gives the same verdict. Second, decompose into truly atomic claims before this loop; the loop is only as good as the claims you feed it. Get those right and faithfulness becomes a stable, trustworthy number.

The recap shows how the pieces assemble into a scorecard. Recall and precision combine into a retrieval score; hit rate, MRR, and nDCG describe ranking quality; faithfulness measures grounding by claim; answer relevance measures on-topic-ness. You compute all of these per query, then average across the suite.

The assembly order is also the debugging order. Read the retrieval scores first to confirm the evidence was present, then read the generation scores to see whether the model used it. Keeping the columns separate, as the concept post insisted, is what makes the scorecard diagnostic rather than merely descriptive.

The CTA moves from theory to practice. You now know what each metric computes; the next post hands you a runnable harness — a from-scratch version plus the Ragas equivalent — that you can point at your own pipeline and wire into CI.

Save this post as the reference you return to when a metric on your dashboard moves and you need to remember exactly what it counts and why.