Reranking with Cross-Encoders

This post answers the 'so what' for reranking, and the cover frames it as a fix you are probably not using yet. The reader has learned what reranking is; now they need a concrete reason to add the latency and complexity to their stack.

The core argument runs through the whole post: vector search is optimized for the wrong metric for generation. It maximizes recall — getting the answer somewhere in the pool — when what your LLM actually needs is precision at a tiny k. Reranking closes that gap.

The opening slide states the problem bluntly because it is the crux of the whole post. Embedding retrievers are tuned and benchmarked on recall: did the relevant document appear in the top-N at all. That is the right target for the first stage.

But generation does not consume the top-50. Context windows, cost, and the 'lost in the middle' effect mean you feed the LLM only a handful of chunks. If the genuinely best passage sits at rank 12, it falls outside that handful and the model answers without it. High recall has quietly produced a bad answer.

This slide draws the precise distinction between the two metrics that the whole post hinges on. Recall@50 asks a yes/no question about the pool: is the answer in there somewhere? Precision@5 asks about the top of the list: are the few things I will actually use any good?

Retrievers are built to win the first question and are largely indifferent to the second. Rerankers are built to win the second. Naming the metrics gives the reader the vocabulary to diagnose their own pipeline — if recall is high but answers are poor, you have a precision problem, and that is exactly what reranking fixes.

The bar chart makes the abstract argument tangible. The same candidate pool, reordered, moves the best passage from rank 12 to rank 1, and precision@5 jumps accordingly. The exact numbers are illustrative, but the shape is what teams report repeatedly when they add a reranker.

The takeaway is that this gain comes with zero retraining and no new data. You did not change your embeddings, your documents, or your model. You added a scoring pass over results you already had. That is an unusually high return for the effort.

This slide connects reranking to the metric people actually care about: hallucination. An LLM is faithful to the context you give it; it cannot ground an answer in a passage it never received. So the practical cause of many 'hallucinations' in RAG is not the model being creative — it is the right evidence sitting just below the cutoff.

Reranking attacks this directly by changing what crosses the cutoff. Put the correct passage at rank 1 and the model paraphrases the truth and can cite it. The lesson: before blaming the LLM, check whether the evidence even reached it.

The before/after compare diagram crystallizes the payoff into two columns. Without a reranker, your top-5 is a noisy mix and the best answer is stranded out of reach. With a reranker, the top-5 is tightly relevant and the best answer leads.

This is the mental snapshot to keep: reranking does not make your data better or your model smarter, it makes the slice you actually use dramatically more relevant. Everything downstream — answer accuracy, citation quality, user trust — rides on that slice.

The code slide shows the entire value proposition in six lines. Retrieve 50 for recall, pair the query with each candidate, score, sort descending, and slice the top-5. That is the full reranking integration.

The comments matter: top_k=50 is the high-recall stage and ranked[:5] is the high-precision result you hand to the LLM. Seeing how little code stands between 'I have a retriever' and 'I have reranking' is itself the argument for doing it. Post 4 expands this into a complete runnable example.

Honesty about tradeoffs builds trust and pre-empts the objection. Reranking is not free: each candidate is a forward pass, so a cross-encoder adds tens to hundreds of milliseconds depending on model size and candidate count, all on the critical path before the LLM responds.

Two other limits matter. Cost scales with N, the number of candidates you rerank, so rerank dozens not thousands. And quality is capped by stage-1 recall — if the answer never made the pool, reranking cannot help. These constraints set up the 'when not to' slide and the entire mistakes post.

This slide keeps the post intellectually honest by naming the cases where reranking is wasted effort. If your retriever already returns one clearly correct document, reordering changes nothing. If latency is brutal and recall is already fine for your use case, the extra pass buys little.

Reranking earns its keep when results are numerous, semantically similar to each other, and ordering genuinely affects the answer — which describes most real RAG over large document sets. Knowing the boundary prevents cargo-culting a reranker into a system that does not need one.

The payoffs slide grounds the argument in outcomes a stakeholder cares about. Support bots stop citing a plausible-but-wrong policy and start citing the correct one. Search results feel intelligent rather than keyword-y. And because the top chunks are more relevant, you often need fewer of them, which directly lowers LLM token cost.

The last point is underappreciated: better ordering can make your whole pipeline cheaper, not just more accurate, because you can shrink the context you send to the model without losing the answer.

The bottom-line slide compresses the post into one claim: reranking is the highest-leverage, lowest-effort upgrade available to a RAG system. No retraining, no new labeled data, no model swap — just a second scoring pass.

That framing is what you want the reader to remember when they audit their own stack. If answers are weak and recall is fine, reranking is almost always the first thing to try, and it is cheap to try.

The CTA hands off to the mechanics post. Having argued why reranking matters and where it pays off, the natural next question is how the cross-encoder actually produces those better scores.

Day 74's teaser — tokens, attention, and the score head — promises to open the box. Post 3 walks through the joint input format, full cross-attention, the [CLS] token, the training data, and the cost math that explains why reranking is a query-time operation on a small set.