Hybrid Search (BM25 + Vector)

This is the hands-on post. The previous three established the concept, the stakes, and the mechanics; now you build a working hybrid search end to end. The implementation deliberately uses the simplest possible components — rank_bm25 for the sparse side, sentence-transformers for the dense side, and plain numpy for the similarity math — so that nothing is hidden behind a vector database abstraction.

The goal is that you can run this on a laptop, watch each retriever return its own ranking, and see RRF merge them. Once the moving parts are clear, the final slides show exactly which pieces you swap out to make it production-grade: a real ANN index, a real BM25 engine, or a single database with native hybrid support.

The setup slide installs and imports the three dependencies. rank_bm25 gives a pure-Python BM25 implementation that's perfect for learning and prototyping. sentence-transformers loads a compact embedding model — all-MiniLM-L6-v2 produces 384-dimensional vectors, is fast on CPU, and is more than good enough to demonstrate semantic retrieval. numpy handles the vector arithmetic.

Loading the model once at module scope (rather than per query) matters even in this toy example, because model loading is the slow part. In production you'd load it once at service startup and reuse it for every request. The imports are intentionally minimal — no vector database, no search server — so every step downstream is visible in plain Python.

The corpus is five short documents chosen to exercise both retrievers. Some contain exact identifiers (SKU-449X, INV-7741) that BM25 will handle and vectors will fumble; others are pure prose about refunds, cancellation, and password recovery that vectors will handle via meaning. This mix is what makes the hybrid behavior visible: different queries will be won by different retrievers.

In a real system this list becomes thousands or millions of chunks loaded from your knowledge base, but the structure is identical — a list of text strings, each with a stable index or ID. Keeping it tiny here means you can read every document and verify by eye which one each query should return.

This slide builds both indexes from the same corpus. The sparse index tokenizes each document by lowercasing and splitting on whitespace, then constructs a BM25Okapi object that precomputes the corpus statistics (document frequencies, average length) BM25 needs. The dense index encodes every document into a normalized embedding matrix in one batched call.

Note the parallel structure: both are built once, offline, from the documents. That's the indexing phase. The normalize_embeddings=True flag is important — it makes the vectors unit length so that a plain dot product equals cosine similarity, which simplifies the retrieval code on the next slide. In production these two indexes might live in entirely different systems (a search engine and a vector database), but conceptually they're still just two precomputed views of the same corpus.

The retrieval functions show the symmetry between the two sides. bm25_topk scores all documents against the tokenized query and returns the indices of the top-k by descending score. vector_topk embeds the query once, computes cosine similarity against every document via a single matrix-vector product (the dot product equals cosine because everything is normalized), and returns the top-k indices the same way.

The deliberate design choice is that both functions return the same thing — a list of document indices in ranked order. That uniform interface is what lets the fusion step treat them identically. In production the brute-force numpy dot product becomes an ANN index query for the dense side and a search-engine call for the sparse side, but the contract — query in, ranked IDs out — stays exactly the same.

This is the RRF function from Post 3, reused verbatim. It takes a list of ranked lists (one per retriever) and accumulates 1/(k+rank) for each document across all lists, then sorts by the accumulated score. Because it operates purely on rank positions, it doesn't care that BM25 and cosine scores live on different scales — the central fusion problem simply never arises.

Reusing the exact same function across the teaching post (Post 3) and the implementation post (Post 4) is intentional: the fusion logic is genuinely this simple, and seeing it twice reinforces that the hard part of hybrid search is a seven-line function plus the discipline to fetch enough candidates before calling it.

The search() function ties everything together and is the only function a caller needs. It fetches a generous top-10 from each retriever (wide enough that good documents in either list get a chance to combine), fuses the two lists with RRF, and returns the final top-k documents by text. The trailing example runs the canonical mixed query — 'why is INV-7741 unpaid' — and the comments show the result: BM25 surfaces the exact invoice document, and the vector side contributes related billing context.

This is the payoff of the whole post. The query wins on the exact identifier via BM25 while still pulling in semantically relevant context via vectors, and the fusion produces a single sensible ordering. Notice the fetch-wide-then-trim pattern (10 from each, fused, then top-k) — fetching too few before fusing is one of the classic mistakes covered in Post 5.

The flow diagram is a map of the code you just wrote. search(query) is the entry point; it calls bm25_topk to produce the sparse list and vector_topk to produce the dense list; rrf merges them into the fused order that search returns. Four functions, one clear data flow.

Keeping the architecture this legible is the reason the post avoids a vector database: a database would collapse three of these boxes into one opaque call, which is great for production but bad for understanding. Once you've internalized this flow, adopting a database that does it for you is trivial — you'll know exactly what it's doing inside.

The production tips slide maps each toy component to its real-world replacement. The numpy brute-force dot product becomes an ANN index — FAISS or HNSW — so dense retrieval stays fast over millions of vectors. The pure-Python rank_bm25 becomes a real inverted-index engine like Elasticsearch or OpenSearch with proper analyzers. Or you sidestep running two systems entirely by using a single engine with native hybrid support, such as Weaviate or Qdrant.

Two additional upgrades matter: adding a cross-encoder reranker over the fused top-k for a precision boost, and tuning the RRF constant k (start at 60). None of these change the architecture you built — they swap implementations behind the same query-in, ranked-IDs-out contract.

This code slide shows the alternative to wiring two systems yourself: let one engine do the fusion. The Qdrant example uses prefetch to run a sparse query and a dense query server-side, then a FusionQuery with Fusion.RRF to merge them — the exact RRF logic from your hand-rolled version, but executed inside the database over a real ANN index.

The value of having built it by hand first is that this API is no longer mysterious. 'prefetch' is your two topk calls, 'FusionQuery RRF' is your rrf function, and 'limit' is your final trim. You can read engine documentation fluently because you know precisely what each piece corresponds to. This is the pattern most teams ultimately ship — native hybrid in a single engine — but understanding the internals is what lets you debug it when recall surprises you.

The recap slide confirms what the code achieved: two independent indexes queried by one request, each returning its own ranked top-k, merged by RRF without any score scaling, and a single search() function returning the fused best documents.

The deeper lesson is how little code real hybrid search requires once you stop treating it as magic. The retrievers are off-the-shelf, the fusion is seven lines, and the only judgment calls are how wide to fetch and which fusion to use — both of which the final post addresses as common mistakes.

This was the implementation post in the five-part day. We built a complete hybrid search from scratch — BM25 index, dense vector index, top-k retrieval from each, and RRF fusion — then mapped every toy component to its production-grade replacement, including native hybrid in a single engine like Qdrant.

The final post is the failure-mode map: the specific mistakes that turn a promising hybrid setup into a regression, from adding raw scores to shipping without measuring recall.