Hybrid Search (BM25 + Vector)

The mistakes post exists because hybrid search has more moving parts than single-retriever search, and every extra part is a new way to fail silently. The retrievers themselves are rarely the problem — they're mature, well-tested components. The problems cluster around the fusion step and the discipline (or lack of it) around measurement.

The through-line of this post is uncomfortable: hybrid is not automatically better than a single retriever. A sloppy fusion bolted onto two good retrievers can underperform the better one alone. These five mistakes are how that happens, and each comes with the fix.

Adding raw scores is the single most common hybrid bug, and it's the one Post 3 warned about. BM25 scores are unbounded positives often in the 0–30 range; cosine similarities sit in [0,1]. Summing them directly means the BM25 magnitude dominates purely arithmetically — the vector signal becomes a rounding error, and your 'hybrid' system is BM25 wearing a costume.

The insidious part is that it doesn't crash or look obviously wrong; it just quietly ignores the semantic retriever. The fix is one of two things: min-max normalize each retriever's scores to a common [0,1] range before applying any weight, or use RRF, which discards raw scores entirely and fuses on rank position — making the whole scale problem disappear.

Fusing too few candidates is a subtler trap that even careful implementations fall into. If each retriever only returns its top-3 before fusion, a document that the vector side ranks #1 but BM25 ranks #6 never enters the fusion at all — it was truncated out of the BM25 list before RRF could combine the evidence. Fusion can only reconcile documents it actually receives.

The fix is to fetch a generous pool from each retriever — top-20 to top-100 — fuse those wider lists, and only then trim to your final k. The retrieval cost of fetching more candidates is small, and it's what lets cross-retriever agreement actually surface. This is exactly why the implementation in Post 4 fetched 10 from each side before returning only 3.

This code slide contrasts the wrong and right candidate-pool sizes directly. The wrong version fetches only the top-3 from each retriever and fuses those — so any good document ranked 4th or lower in both lists is gone before fusion runs, and the final top-3 is built from an artificially impoverished pool. The right version fetches the top-50 from each side, fuses those wide lists, and only then trims to the final 3.

The asymmetry in cost is the key insight: fetching 50 candidates instead of 3 is nearly free (both retrievers already rank the whole corpus internally), but the recall difference can be dramatic. The rule of thumb is to fuse from a pool roughly an order of magnitude larger than your final k, then trim at the very end.

Guessing the fusion weight α is a quieter mistake than adding raw scores but just as real. With weighted score fusion, α controls how much you trust BM25 versus vectors — and picking 0.5, or any value, by intuition is essentially a coin flip. The optimal α depends on your specific corpus and query distribution; a code-heavy technical corpus might want more BM25 weight, a prose knowledge base more vector weight.

There are two honest fixes. Either sweep α across a range on a labeled evaluation set and pick the value that maximizes recall, treating it as the hyperparameter it is — or sidestep the entire question by using RRF, which has no per-source weight to guess. For most teams without the appetite for a tuning campaign, RRF's parameter-free robustness is the pragmatic choice.

Mismatched tokenization is the mistake that quietly destroys BM25's exact-match superpower — the very reason you added it. BM25 quality depends entirely on the analyzer: how text is lowercased, stemmed, split, and how stopwords and punctuation are handled. If 'SKU-449X' is tokenized as one unit at index time but split into 'SKU' and '449X' at query time (or vice versa), the exact match fails and the document you were counting on never surfaces.

The fix is a discipline, not a setting: use the exact same analyzer for indexing and querying. In a real search engine this means configuring one analyzer and applying it on both sides; in custom code it means calling the same tokenization function everywhere. Identifier-heavy corpora are the most sensitive to this, which is ironic because they're also the corpora that benefit most from BM25 in the first place.

The decision tree reframes the whole day as a practical 'should you even use hybrid' question. If your queries contain exact IDs, codes, or names, hybrid clearly helps and you should keep BM25 in the mix — that's its home turf. If they don't but involve lots of synonyms and paraphrase, pure vectors may suffice, though it's still worth testing hybrid since the cost of trying is low. If queries are neither identifier-heavy nor paraphrase-heavy, plain BM25 might be all you need.

The point of the tree is to push back on cargo-culting. Hybrid has become a default, but defaults deserve scrutiny. The honest answer for your system comes from understanding your query distribution — and, ultimately, from measuring, which is the next mistake.

Shipping without measuring is the meta-mistake that makes all the others invisible. 'Two retrievers must beat one' is an assumption, not a result, and it's sometimes false: on certain corpora a well-tuned single retriever wins, and a careless fusion drags it below that baseline. Without a labeled eval set you have no way to know which world you're in, so you ship on faith.

The fix is cheap relative to its value: assemble a small set of representative queries paired with their correct documents, and compute recall@k for BM25-only, vector-only, and hybrid. Even fifty hand-labeled query-document pairs are enough to catch a regression. You commit to hybrid only if the numbers say it actually beats both single retrievers on your data.

This code slide operationalizes the 'measure before shipping' rule. The recall_at_k helper takes any retrieval function and an eval set of (query, gold_document) pairs, and computes the fraction of queries where the correct document appears in the top-k. Running it three times — for BM25 alone, vectors alone, and the hybrid RRF combination — gives you the exact comparison you need.

The closing comment is the whole philosophy in one line: only ship hybrid if it actually beats both. This turns the question from a matter of faith into a matter of data, and it's the single most important habit in this post. Note that the hybrid line uses the fetch-wide pattern (50 from each) from the earlier mistake — the evaluation harness should test the system exactly as you'd deploy it.

Forgetting the reranker is a mistake of omission — leaving precision on the table. Fusion is fundamentally a recall mechanism: it assembles a good candidate set, but RRF has no real understanding of which fused document best answers the query, since it only ever saw rank positions. So the fused top-5 might contain the perfect answer at position 4 instead of position 1.

A cross-encoder reranker — a model that scores each (query, document) pair jointly — reorders the fused top-k with genuine query-document understanding. It's often the highest-leverage addition left once retrieval is solid, frequently buying more precision than any amount of fusion-weight tuning. It's also a natural segue to the next day's topic, which is reranking in depth.

The checklist slide is the post in six lines: normalize scores or use RRF rather than adding raw values; fetch a wide pool from each retriever before fusing; tune α on data or skip it with RRF; use the same analyzer at index and query time; measure recall before shipping hybrid; and add a reranker on the fused top-k.

Treated as a pre-launch checklist, these six items catch nearly every way hybrid quietly underperforms. The unifying theme is that hybrid's extra power comes with extra responsibility — most failures are operational discipline problems, not algorithmic ones.

This was the final post in the five-part day on hybrid search, mapping the failure modes: adding raw scores, fusing too few candidates, guessing the fusion weight, mismatched tokenization, and shipping without measuring — plus the missed opportunity of skipping a reranker.

That reranker is exactly where the series goes next. Day 73 covers rerankers in depth — how cross-encoders squeeze precision out of the candidate set that hybrid retrieval works so hard to fill.