Embedding Models Compared

This is the hands-on post: a complete, runnable embedding bake-off. The philosophy is simple — stop debating which model is best and measure it on your own data. By the end you'll have three models scored head to head on recall, latency, and dimension.

The cover frames the deliverable: about 40 lines that load three models, embed a labeled eval set, search, and print a comparison table. Everything is designed so you can swap in your own corpus and let the numbers, not the leaderboard, pick the winner.

The setup imports keep dependencies minimal: sentence-transformers for local models, numpy for vector math, and the OpenAI client for an API model. The OpenAI client is instantiated once so it can be reused across calls.

This mirrors a realistic comparison stack — a couple of strong open-source models you can self-host plus one hosted API model — letting you weigh the classic tradeoff between local control and managed convenience on the same data.

A labeled eval set is the heart of an honest comparison, and this slide shows its minimal shape: documents plus queries, where each query is tagged with the index of its correct document. That gold label is what lets you compute recall objectively instead of eyeballing results.

Three pairs are shown only to keep the slide readable. The tips slide is explicit that you need 50-200 real pairs drawn from your own domain — with only three, every reasonable model scores perfectly and the test tells you nothing. The structure, though, stays exactly this: text plus the index of the right answer.

This slide wraps each model behind a uniform interface so the benchmark code doesn't care which model it's calling. embed_local handles sentence-transformer models and accepts an optional prefix, because prefix-based models like BGE need 'query: ' prepended. embed_openai calls the API and manually normalizes the returned vectors so every model's output is unit-length.

The uniform wrapper is what makes a fair comparison possible: each model gets its correct settings (prefix, normalization), but the downstream scoring code treats them identically. This is the practical answer to the prefix and normalization footguns raised in posts 3 and 5.

recall_at_k is the scoring function. For each query vector it computes cosine similarity against every document vector via a single matrix-vector product (valid because the vectors are normalized), takes the top k by score, and checks whether the gold document index is among them. Averaging over all queries gives recall@k.

This is the same metric from post 2, now wired into the bake-off. It's deliberately transparent: no vector database, no hidden index, just a dot product and a sort, so you can see exactly what's being measured and trust the comparison.

The bench function times and scores a single model end to end. It records wall-clock time around embedding the documents and queries, computes recall@k, and reports the vector dimension from the embedding shape. Returning name, recall, milliseconds, and dimension as a tuple makes the results trivial to tabulate.

Timing both document and query embedding together gives a rough but useful throughput signal. For a production decision you'd separate one-time index cost from per-query latency, but for a first-pass comparison this single number cleanly exposes which models are heavy.

This slide runs the bake-off: it benchmarks MiniLM, BGE (with its required 'query: ' prefix), and the OpenAI model, then prints a formatted table. Note the lambda for BGE passes the prefix while MiniLM and OpenAI don't — each model gets exactly the settings it expects.

The formatted print produces a clean, scannable table of model, recall, latency, and dimension. That table is the actual deliverable of the whole post: a side-by-side, same-data comparison you can read in two seconds and defend to a teammate.

The trace diagram shows representative output and, more importantly, how to read it. All three models tie on recall for this toy set, but MiniLM and BGE are far faster than the OpenAI call and use a quarter of the dimension. The comment spells out the conclusion: when recall ties, pick the cheap, fast, low-dimension option.

The numbers are illustrative — on a real 50-200 pair eval set you'd expect recall to actually separate the models, and that separation becomes your primary signal. The reading discipline stays the same: lead with recall, break ties with cost, latency, and dimension.

The honesty checklist is what separates a real benchmark from a misleading one. Use 50-200 real query→document pairs so models can actually be distinguished. Score recall@k, and add nDCG when the ordering of results matters, not just presence. Apply each model's required prefix. Measure latency at your real batch size, since batching changes throughput dramatically. And confirm on a held-out set before committing.

These aren't optional polish — each one corresponds directly to a way comparisons go wrong, which post 5 catalogs. Following them is the difference between a number you can stake a production decision on and one that just confirms your bias.

Reading the result is its own skill. When models tie on recall, the decision drops to operational factors: a 384-dim local model that matches a 1536-dim API model on accuracy is the clear winner on cost, latency, and control. When recall differs, that gap dominates — finding the right chunk is the whole point, and everything else is secondary.

The discipline is to rank by the metric that maps to user value first, then use cost and latency as tiebreakers. A model that's marginally more accurate but far more expensive and slower is rarely the right production choice unless that recall gap genuinely moves outcomes.

The CTA points to the final post of the day. You can now run a fair bake-off — but the results only mean something if the comparison was set up correctly. The next post is the field guide to the mistakes that quietly invalidate embedding comparisons, so your numbers stay trustworthy.