Embedding Models Compared

This closing post is a field guide to the traps that make embedding comparisons lie. The unifying theme is that embedding mistakes are silent — nothing crashes, the system keeps returning chunks, and the damage hides in recall you never measured.

The cover states the core insight bluntly: most 'model X is better' claims fail not because the model is bad but because the comparison was flawed. Avoiding these six mistakes is what makes your bake-off results trustworthy enough to base a production decision on.

Mistake one is mixing models. Query and document vectors only make sense when they live in the same learned space, which means they must come from the exact same model, version, and settings. Embedding documents with MiniLM and queries with OpenAI puts them in unrelated coordinate systems — and since their dimensions differ, the comparison may not even be computable.

Even subtler is mixing versions of the same model family or changing normalization between the two sides. The fix is a discipline: one model, one configuration, encodes both sides of every comparison. Lock it down and many mysterious 'bad retrieval' results disappear.

This code makes the mixing trap and its fix concrete. The wrong version embeds the query with a 1536-dim OpenAI model and the documents with a 384-dim MiniLM model — the dimensions don't match and cosine similarity is meaningless. The right version uses one model with consistent normalization for both query and documents.

The lesson generalizes beyond this example: any divergence in model, version, or settings between query and corpus poisons the similarity scores. Pinning a single model and configuration for both sides is the simplest, highest-impact rule in this entire post.

Mistake two is trusting the leaderboard as a verdict. A top MTEB score reflects broad average performance over generic, largely-web data. Your domain — contracts, clinical notes, support tickets, source code — has its own vocabulary and structure, and benchmark rank frequently fails to transfer.

The repeated real-world outcome is that a lower-ranked model fits a specialized domain better than the leaderboard champion. The leaderboard is a great way to build a shortlist of candidates; it is a poor way to make the final pick. That decision belongs to a bake-off on your own data.

Mistake three is skipping the instruction prefix. E5, BGE, and GTE-family models are trained to read role tags like 'query:' and 'passage:'. Omit them and the model operates outside its training contract, dropping similarity quality in a way that produces no error and no warning.

This is one of the most common reasons a strong model gets unfairly dismissed as weak. Worse, in a comparison it's unfair: if one model gets its prefix and another doesn't, you're measuring your setup, not the models. Read each model card and apply its exact encoding requirements to every model in the test.

The mindmap collects the four structural ways comparisons break: mixing models (different spaces, dimension mismatch), trusting only the leaderboard (generic data, wrong domain), skipping the prefix (silent quality drop from misreading the card), and using a bad metric (accuracy instead of recall, or a tiny eval set).

Seeing them together highlights the common thread — every one of these produces plausible-looking output while invalidating the conclusion. A comparison can be completely wrong and look completely fine, which is exactly why these mistakes are worth memorizing.

Mistake four is inconsistent normalization and metrics. If you compute cosine on normalized vectors for one model and raw dot products for another, you're not comparing models — you're comparing measurement procedures. The same applies to mixing metrics across candidates.

The fix is uniformity: normalize every model's output the same way and score all of them with the same metric (recall@k, and nDCG when order matters). Only when the procedure is identical for every candidate does the resulting ranking actually reflect the models rather than your inconsistent setup.

Mistake five is a toy evaluation set. Three or five hand-picked queries can't distinguish models, because almost any reasonable model gets them all right — every model scores 100% and the test conveys no information. The bake-off in post 4 used three pairs only for readability, and flagged this explicitly.

A trustworthy comparison needs dozens to hundreds of real query→document pairs drawn from production traffic, including the hard and ambiguous cases. That's where models actually separate. Investing in a solid eval set is the single best thing you can do to make every future comparison meaningful.

Mistake six is ignoring operational cost. Recall is necessary but not sufficient. A model that wins recall by a single point while costing four times the storage, many times the latency, and a per-query API fee may be the wrong production choice. And the migration cost compounds it — switching later forces a full re-embedding of the corpus.

A complete comparison logs cost, latency, and dimension alongside recall, then weighs them against the value of any accuracy gain. The right model is the one that wins the whole tradeoff, not just the accuracy column.

This checklist code operationalizes a fair comparison as assertions. It checks that the same model encodes query and documents, that the required prefix is applied, that vectors are normalized consistently, that the eval set is large enough to be meaningful, and that recall@k uses the same metric across models — while also logging latency, dimension, and cost.

Turning the rules into assertions is a practical trick: instead of remembering six caveats, you encode them so a violated assumption fails loudly. That converts the silent mistakes of this post into noisy ones you can catch before they corrupt a decision.

The 'do this instead' list is the post's positive summary — six rules that make a comparison trustworthy. One model encodes both sides. Test on your own data, not just MTEB. Apply each model's prefix. Normalize and use one metric for all. Score recall@k on at least 50 real pairs. And weigh cost, latency, and re-embedding alongside accuracy.

Follow these and your bake-off becomes a decision you can defend, not a number you have to caveat. Together with post 4's runnable code, this gives you both the procedure and the guardrails to compare embedding models honestly.

The CTA closes Day 70 and previews the next day. Embedding models decide how well text is placed in vector space — but chunking decides what text exists to be embedded in the first place. The next day covers chunking strategies and how splitting your documents determines what can ever be retrieved.