Reranking with Cross-Encoders

This is the mechanics post, and the cover signals depth: we are opening the cross-encoder and looking at what happens token by token. Posts 1 and 2 stayed at the level of 'it reads them together and scores them.' Now we make that concrete.

The goal is that by the end the reader understands the joint input format, why cross-attention is the entire advantage, how the score is read off the model, how these rerankers are trained, and the cost math that forces reranking to be a query-time, small-set operation.

The steps slide lays out the forward pass as a recipe so the reader has the skeleton before the details. Concatenate query and document into one sequence with special tokens, tokenize, run a single transformer pass, and read a relevance score from the [CLS] position via a small linear head.

The important realization here is 'one forward pass per pair.' There is no separate encoding of query and document that gets compared later — it is a single, fused computation. That structure is what every subsequent slide elaborates and what ultimately explains the cost profile.

The flow diagram visualizes the input construction because the joined-sequence format is the thing most newcomers picture wrong. Two separate texts go in; one sequence comes out, structured as [CLS] query [SEP] document [SEP].

That single sequence then flows through the transformer and out the other side as one float. Seeing the funnel — two texts to one input to one score — fixes the most common misconception, which is that the model somehow embeds each side and compares them. It does not; it fuses them first.

This slide states the central mechanism: cross-attention is the whole reason a cross-encoder outperforms a bi-encoder. Because both texts live in one sequence, the self-attention layers let query tokens attend to document tokens and vice versa, layer after layer.

That means the model can directly align a question word with its answer phrase even when the surface forms differ. A bi-encoder forecloses this by compressing each side into a single vector before any comparison, so the token-level evidence is gone. Internalizing this makes the accuracy gap feel inevitable rather than magical.

The compare diagram puts the two architectures side by side so the structural difference is unmistakable. On the left, the bi-encoder produces two independent vectors and compares them with a dot product at the very end — fast, cacheable, but blind to interaction. On the right, the cross-encoder fuses the inputs so tokens interact throughout, then reads a score from the joint context.

The phrase 'rich interaction, costly' names the exact tradeoff. Everything good and everything expensive about cross-encoders flows from this one design choice, which is why the diagram is worth pausing on.

This slide explains how a transformer turns a pair into a number. BERT-style models prepend a special [CLS] token, and after the forward pass its final hidden state acts as a summary of the entire input pair. A single linear layer — the score head — maps that summary vector to one scalar logit.

Training adjusts both the encoder and this head so the logit is high for relevant pairs and low for irrelevant ones. Knowing that the score literally comes from one token's representation through one linear layer demystifies the 'relevance score' and sets up why it is a logit rather than a probability.

This code slide peels back the convenient predict() API to show the raw transformers machinery underneath. You load a tokenizer and a sequence-classification model, tokenize the query and passage together — note the two-argument call that builds the joint sequence with separators automatically — and run a forward pass with gradients disabled.

The output is a logits tensor; that single value is the relevance score the high-level CrossEncoder wrapper returns. Showing this layer matters because it makes clear there is nothing mysterious happening: it is a standard classification head emitting one number for one fused input.

This slide demystifies where reranker quality comes from: the training data and objective. The canonical dataset is MS MARCO, built from real Bing queries paired with passages labeled relevant or not. Positives are genuinely relevant passages.

The subtle, important part is the negatives. The strongest rerankers train on 'hard negatives' — passages a retriever returned that look relevant but are not. Learning to separate those near-misses is exactly the skill reranking needs at inference, since the candidate pool is full of plausible-looking distractors. The loss simply pushes positive scores above negative ones.

The cycle diagram turns the training description into a loop the reader can hold onto. Take a query with a candidate, score the pair in a forward pass, compare the score against the other (positive vs negative) candidates, and backpropagate to update both the encoder and the score head.

Repeated over millions of examples, this teaches the model a general notion of query-document relevance that transfers to your domain at inference time. The point worth keeping: you usually do not train your own reranker — a pretrained one off MS MARCO already encodes a strong, reusable relevance prior.

The cost-math slide makes the performance profile concrete and explains a design constraint the reader will hit immediately. A bi-encoder costs one forward pass at query time because document vectors are precomputed and cached. A cross-encoder cannot cache anything, so it costs one forward pass per candidate — N passes for N candidates.

The little latency formula, N times per-pair-ms, is the dial you will tune for the rest of the series. It is the direct reason you rerank a top-N of dozens, never the whole corpus, and it foreshadows the latency mistakes in post 5.

This closing concept slide nails down why the cost structure is unavoidable rather than an implementation detail. The score is a function of the query and document seen jointly, so it simply does not exist until a specific query arrives. Change the query and every previously computed score is invalid.

That is the fundamental reason cross-encoders run online, at query time, over a small candidate set produced by a cheaper retriever. There is no offline shortcut — the joint dependency that makes them accurate is exactly what makes them impossible to precompute.

The CTA points from theory to practice. We have covered the joint input, cross-attention, the score head, training on MS MARCO with hard negatives, and the O(N) cost that mandates query-time reranking over a small pool.

Day 74's teaser promises the full runnable pipeline. Post 4 assembles everything into a working retrieve-then-rerank example you can paste into a notebook, including the production shape with a hosted reranker.