Reranking with Cross-Encoders

This is the hands-on post, and the cover promises a complete two-stage pipeline in about 40 lines. After three posts of concept and mechanics, the reader wants to see retrieve-then-rerank actually run and watch the ordering change.

The whole post is structured to be copy-pasteable top to bottom: install, build a retriever, fetch candidates, rerank them, slice the precise top-k, optimize, and finally show the production-hosted variant. The payoff to watch for is a document that retrieval ranked third jumping to first once the cross-encoder reads it against the query.

The install slide lists exactly the dependencies the example needs and nothing more. sentence-transformers provides both the bi-encoder (SentenceTransformer) and the cross-encoder (CrossEncoder) under one roof, faiss-cpu gives a fast local vector index, and numpy handles the array plumbing.

Keeping the dependency set minimal is deliberate — the reader should be able to run the entire post in a fresh notebook without a database, an API key, or a GPU. The faiss-cpu build runs anywhere; the GPU optimizations come later as an option, not a requirement.

This slide builds stage one: the fast retriever. We load a small, well-known bi-encoder, define a handful of documents, encode them with normalize_embeddings=True, and add them to a FAISS inner-product index. Normalizing the embeddings means inner product equals cosine similarity, which is the standard choice for text.

The document set is tiny and intentionally includes near-duplicates about passwords plus unrelated distractors about refunds and shipping. That mix is what makes the reranking demonstration meaningful — there are several plausible candidates, and the ordering between them is exactly what the cross-encoder will sharpen.

Here we run the first-stage search and inspect the candidate pool. We embed the query the same way we embedded the documents — same model, same normalization — then ask FAISS for the top-N. Using the same encoder for query and documents is essential; mixing models would put them in incompatible spaces.

Printing the rank, score, and text matters pedagogically: the reader sees the bi-encoder's ordering, which is decent but imperfect. Often a near-duplicate phrasing outranks the genuinely best match. That imperfect ordering is precisely the problem stage two exists to fix, and seeing it first makes the improvement obvious.

This is the core of the post: stage two reranking. We load the cross-encoder, build (query, document) pairs by pairing the single query with each candidate, and call predict() to get one relevance score per pair. Then we zip candidates with their scores and sort descending.

The key structural detail is the pairing — every candidate is scored against the same query in its own forward pass, which is the O(N) cost from post 3 made literal. When the reader prints the reranked list, the best password-reset passage should now sit at the top, even if the bi-encoder had ranked a wordier near-match higher. That flip is the entire point of the post.

The pipeline diagram summarizes the four moves the code just performed so the reader can map syntax to concept. Embed the query with the bi-encoder, retrieve the top-4 candidates from FAISS, pair each with the query and score them with the cross-encoder, then sort best-first.

Placing this diagram after the two code stages lets it act as a recap rather than an introduction. The reader has now seen each stage in code; the diagram lets them step back and confirm they understand how the pieces connect end to end.

This slide closes the loop by showing what reranking is ultimately for: feeding the LLM. We take the reranked top-k — here just 2 — join them into a context block, and drop them into a grounded prompt that instructs the model to answer using only that context.

The comment that the prompt goes to your LLM keeps the example self-contained without hard-coding a specific provider. The lesson is that everything before this point existed to make these two chunks the right two chunks. Reranking's value is realized precisely here, at the moment the model reads its context.

The optimization slide addresses the practical question the cost math raised: how do I make this fast enough. predict() already batches internally, but you can set batch_size explicitly to control memory and throughput. Larger batches use the hardware better up to a point.

The bigger lever is the device. Cross-encoders are transformer forward passes, so a GPU dramatically cuts per-pair latency. Passing device='cuda' moves the model onto the GPU. This slide tells the reader exactly which two knobs to reach for when reranking latency becomes a problem in their own setup.

This tips slide pre-teaches the score-interpretation mistake that gets its own treatment in post 5. Cross-encoder outputs are logits on an open-ended scale, not probabilities between 0 and 1, so a score of 8 is not '80% relevant.'

The critical rule is that scores are only comparable within a single query. Sorting candidates for one query by score is exactly right; comparing a score from one query to a score from another is meaningless because the scale shifts per query. An optional cutoff to drop weak chunks is fine, but it should be tuned, not assumed.

The final code slide shows the production shape so the reader knows the local example scales. Instead of running a cross-encoder yourself, you call a hosted reranking API — here Cohere's rerank endpoint — passing the query and the candidate documents and asking for the top_n.

The structural parallel to the local code is the point: same inputs (query plus candidates), same output (relevance-scored, reordered results). Whether the reranker is a local CrossEncoder or a managed API, the two-stage pattern is identical. Only operations — scaling, latency SLAs, billing — differ, which is what production teams ultimately care about.

The takeaway slide compresses the whole pipeline into one memorable instruction: retrieve wide and cheap, rerank narrow and accurate. That sentence is the design pattern the reader should carry away from the entire day.

The second point — that the same five-line second stage works for a local cross-encoder or a hosted API — is reassurance that what they just learned in a notebook transfers directly to production. They are not learning a toy; they are learning the real shape.

The CTA hands off to the final post of the day. The reader can now build a working retrieve-then-rerank pipeline; the remaining risk is the subtle ways it breaks in practice.

Day 74's teaser flags the mistakes post: the failure modes — missing recall, reranking too many candidates, misreading scores, truncation, and ignored latency — that quietly degrade a reranking setup even when the code 'works.'