Reranking with Cross-Encoders

This is the framing post for reranking, and the goal is to install one core picture: search happens in two stages, and the reranker owns the second one. Most people learn vector search and stop there, assuming the order their database returns is the order they should trust. It usually isn't.

The cover headline deliberately says "fixing the order" because that is the entire job. Reranking does not find new documents and does not change your model. It takes a list you already have and puts it in a better order so the right thing ends up on top.

Reranking is a second pass, and the word 'pass' matters. The first pass is built for breadth: scan a huge index and pull back a rough candidate set quickly. The second pass is built for judgment: look hard at that small set and decide which items are genuinely most relevant.

The number 50 here is illustrative but realistic. You retrieve more candidates than you will ultimately use precisely because the retriever is approximate. The reranker's job is to recover the precision the fast retriever traded away. This division of labor — cheap-and-wide then expensive-and-narrow — is the whole design pattern.

The cross-encoder is the specific tool that does the reranking, and its defining property is that it consumes the query and a document together as a single input. Contrast this with the embedding model behind vector search, which sees the query and the document in completely separate forward passes and only compares the resulting vectors afterward.

Because the cross-encoder processes both texts jointly, it can model how specific words in the query relate to specific words in the document. That joint view is exactly what makes it more accurate — and exactly what makes it impossible to precompute, since the score only exists once a particular query meets a particular document.

This comparison is the heart of the post, so it gets its own slide. A bi-encoder encodes each side independently, which means you can embed and cache every document in your corpus once, ahead of time. At query time you only embed the query and do cheap vector math. That is why bi-encoders scale to millions of documents.

A cross-encoder gives up all of that caching. It must run a fresh forward pass for every query-document pair because the inputs are fused. The payoff is accuracy: it sees the interaction between the two texts. The practical reading of this table is simple — use the bi-encoder to scan everything, then the cross-encoder to judge the few survivors.

This slide explains the mechanism in intuitive terms before the 'how it works' post goes deep. Inside a transformer, attention lets every token look at every other token. When query and document share one sequence, attention can directly connect a question word to its answer phrase even when the surface words differ — 'refund' to 'money back', 'reset' to 'change'.

A bi-encoder cannot do this because it has already compressed each side into a single fixed vector before the comparison happens. All the fine-grained, token-level evidence is averaged away. That lost interaction is the precise reason cross-encoders rank better, and it is worth internalizing now.

The pipeline diagram anchors the reranker in a full RAG flow so the reader sees it is one stage, not a replacement for retrieval. Query comes in, the retriever returns a wide candidate set, the cross-encoder reorders those candidates, you keep a tight top-5, and only then does the LLM read them.

The key insight to carry forward is that the reranker sits strictly between retrieval and generation. It never touches the raw corpus and it never generates text. It is a pure scoring-and-sorting step, which is exactly why it is cheap to add to a system you already have.

Understanding what the score actually is prevents a class of mistakes covered later in the series. The cross-encoder emits a single number for each pair, and that number is a relevance signal: bigger means 'more relevant to this query.'

The crucial caveats are that the raw score is a logit, not a probability, and that it is only meaningful relative to other documents scored against the same query. You can sort by it confidently, but you should not read '8.9' as a percentage or compare it to a score from a different query. The number's job is ordering, nothing more.

This first code slide is intentionally tiny because the point is reassurance, not depth: using a cross-encoder is genuinely four lines. You load a pretrained model, hand it a list of (query, document) pairs, and get back a list of scores.

The ms-marco-MiniLM model is the standard starting reranker — small, fast, and trained on a large web-search relevance dataset. The printed score of roughly 8.9 is a logit; a clearly irrelevant document would score sharply lower or negative. The full runnable pipeline comes in post 4; here we just demystify the API surface.

This slide names the pattern explicitly so the reader has a label for it: two-stage retrieval. Stage one optimizes for recall and speed; stage two optimizes for precision and accuracy. Each stage is bad at the other's job, which is precisely why you use both.

The phrase 'broad reach AND precise ordering' captures the win. Neither model alone gives you both. A bi-encoder alone reaches far but orders loosely; a cross-encoder alone orders perfectly but cannot scan a large corpus in real time. Composed, they cover each other's weakness.

The mental-model slide gathers the takeaways into rules you can keep in your head. The most important one is that the reranker only ever sees what stage one hands it. This single fact explains the biggest failure mode in post 5: if the retriever misses the answer, the reranker is helpless.

The corollary — better upstream recall means better downstream reranking — frames how to improve the system. You tune the two stages together. Widen retrieval until the answer is reliably in the pool, then let the cross-encoder do the precise sorting.

The CTA closes the concept post and points at the motivation post. We have established what reranking is, what a cross-encoder is, and why processing texts jointly is more powerful than comparing them separately.

Day 74's teaser flags the obvious next question: this second pass costs latency, so why bother? Post 2 answers that by showing how much answer quality you recover, and where the tradeoff is and isn't worth paying.