Reranking with Cross-Encoders

This is the mistakes post, and the cover sets the tone: reranking is easy to bolt on and easy to get subtly wrong. The failures are rarely dramatic crashes; they are quiet degradations where the code runs fine but the answers are worse than they should be.

The five mistakes covered are the ones almost everyone hits in the first week: reranking a pool that never contained the answer, re-scoring far too many candidates, treating logit scores as probabilities, letting truncation discard the relevant sentence, and never measuring the latency the reranker adds.

The first and most fundamental mistake is forgetting that a reranker is purely a reordering step. It can only sort the candidates the retriever handed it; it cannot retrieve anything new. If the correct document is not in stage one's top-N, the cross-encoder has nothing to promote.

The fix is to verify and improve recall before you even add a reranker. Widen top-N, improve your embeddings or chunking, add hybrid keyword search if needed — get the answer reliably into the pool first. Only then does reranking have something to work with. This single mistake invalidates more reranking deployments than any other.

The flow diagram drives the first mistake home visually: weak retrieval feeds a missing answer into the reranker, the reranker dutifully reorders the junk it received, and the output is still wrong. There is no arrow by which reranking can recover an absent document.

The phrase 'no fix possible' is deliberately blunt. The reader should leave with the reflex that when reranking does not help, the first thing to check is not the reranker but whether stage one is even surfacing the right candidates. Reranking amplifies a good retriever; it cannot rescue a bad one.

The second mistake is a direct consequence of the cost math from post 3. Because a cross-encoder runs one forward pass per candidate, the number of candidates is your dominant latency cost. Rerank a thousand-document pool and you have turned a millisecond search into a multi-second one.

The discipline is to separate the two stages' jobs cleanly: retrieve wide for recall — pulling, say, a few hundred — but rerank only the top 20 to 100 of those. You get most of the precision benefit at a fraction of the cost. The candidate count is the single most important latency dial in the whole pipeline.

The bar chart quantifies the second mistake so the reader feels the curve. Reranking 20 candidates is fast, 100 is acceptable for most applications, and 1000 pushes latency into territory that wrecks the user experience. The values are relative, but the steep growth is real and roughly linear in candidate count.

The practical reading is to pick the smallest candidate count that still captures your answer reliably — a decision you make by checking recall, which ties back to mistake one. Recall sets the floor on how few candidates you can safely rerank.

The third mistake is misinterpreting the scores. Cross-encoder outputs are raw logits on an unbounded, query-dependent scale. A score of 8 does not mean 80% confidence, and a score of 2 for one query may indicate stronger relevance than an 8 for another query with different phrasing.

The safe rule is that scores rank within a single query and nothing more. Sorting candidates for one query by score is exactly what they are for. Comparing scores across queries, or thresholding on a fixed absolute value as if it were calibrated, leads to brittle behavior that looks fine until the query distribution shifts.

This code slide shows the one legitimate exception to 'don't read scores as probabilities' — and frames it carefully. Applying a sigmoid maps a logit into 0-1, which can give you a rough, monotonic cutoff for dropping clearly weak chunks. It is a convenience, not a calibration.

The comment is explicit that this is NOT a true probability. The sigmoid output is still query-dependent and uncalibrated; a 0.5 threshold is a heuristic you must tune on your own data, not a universal confidence level. The slide gives the reader a usable tool while inoculating them against over-trusting it.

The fourth mistake is silent truncation, which is dangerous precisely because it is silent. Cross-encoders have a maximum sequence length, commonly 512 tokens covering the query and the document combined. Feed a long document and the tokenizer quietly cuts it off — and the discarded tail may contain the very sentence that answers the query.

The model then scores a passage it never fully saw, producing a confidently wrong relevance estimate. The fix is to chunk documents before reranking so each candidate comfortably fits the window, which also aligns with how you should be chunking for retrieval in the first place.

This code slide gives the concrete defense against truncation. Set max_length explicitly so you know exactly what window you are working with rather than relying on an unknown default. More importantly, rerank chunks rather than whole documents.

By pairing the query with bite-sized chunks that fit the window, you guarantee the relevant passage is actually seen and scored in full. This also dovetails with retrieval: if your index stores chunks rather than entire documents, both stages operate on the same units and truncation simply stops being a problem.

The fifth mistake is treating latency as an afterthought. Reranking adds a measurable cost on the critical path, before the LLM even begins generating. In a chat experience where users feel every hundred milliseconds, an unmeasured reranking step can quietly blow your latency budget.

The discipline is to measure the added latency on the real path, with your real candidate count and hardware. If it is too slow, you have clear levers: rerank fewer candidates, move to a GPU, or pick a smaller distilled reranker. The mistake is not the latency itself — it is discovering it in production instead of in testing.

The checklist slide gathers all five fixes into a scannable pre-flight list the reader can apply to their own pipeline. Verify recall before adding a reranker, rerank dozens not thousands, treat scores as ranks not probabilities, chunk so candidates fit max_length, and measure the added latency on the real path.

Run down this list and you avoid every failure mode in the post. It is deliberately ordered roughly by impact and by sequence — recall first because it gates everything, latency last because it is the final production check before you ship.

The CTA closes both the post and the day. The reader has now seen reranking from five angles: what it is, why it matters, how cross-encoders work, how to build a pipeline, and how it fails. They have a complete, deployable mental model.

The teaser keeps the series momentum without promising a specific topic, pointing simply to the next entry in the 100 Days of AI. The reader leaves equipped to add reranking to a real RAG system and to debug it when it misbehaves.