Embedding Models Compared

This post explains the machinery behind the comparison: why two models that both output 768-dimensional vectors can rank your documents completely differently. The answer is in how they were trained and how their outputs are scored.

The cover sets the agenda — contrastive learning, pooling, cosine similarity, instruction prefixes, and benchmarks. Understanding these turns model comparison from guesswork into something you can reason about, and explains many of the 'this model is weak' conclusions that are really usage mistakes.

Contrastive learning is the core training recipe for modern embedding models. The model sees pairs: a query paired with its true relevant passage (a positive), and the same query paired with unrelated or deliberately-tricky 'hard negative' passages. The loss function pulls the positive pair's vectors together and pushes the negatives apart.

Repeated over millions of pairs, this shapes a space where semantically related text converges and unrelated text spreads out. The choice of training data and especially the quality of hard negatives is a huge differentiator between models — it's largely why one model 'understands' your domain and another doesn't.

The flow diagram shows the contrastive loop in four beats: take a positive pair, push both through the shared-weight encoder, apply the contrastive loss that pulls matches close and shoves mismatches away, and over training arrive at an aligned space where 'similar means near.'

The shared weights matter — the same encoder embeds both query and passage, which is why query and corpus must always use the same model at inference. The space is a property of that one trained encoder, and mixing encoders means comparing points from two unrelated maps.

Pooling is the often-overlooked step that turns a transformer's per-token outputs into a single vector. A transformer emits one vector per input token; you need exactly one vector per text to search with. Mean-pooling averages all token vectors; CLS-pooling takes the dedicated leading token's vector.

The critical, practical point is that the correct pooling method is baked into how the model was trained. Apply the wrong one at inference and you silently degrade results — the library usually handles this, but rolling your own embedding code is a common way to break a model without any error appearing.

Cosine similarity is the scoring rule that converts two vectors into a single relevance number. It measures the cosine of the angle between them: 1.0 means they point the same direction (very similar), 0 means orthogonal (unrelated), and negative means opposed. Direction, not magnitude, carries the meaning.

Most embedding workflows normalize vectors to unit length, which makes cosine similarity and dot product identical and lets vector databases use the faster dot-product path. This is why 'normalize_embeddings=True' shows up everywhere — it standardizes the metric and speeds up search.

This code computes cosine similarity from scratch so the scoring rule isn't a black box. The dot product of the two vectors divided by the product of their norms gives the cosine. The example shows the payoff: a query about resetting a password scores high against an account-recovery passage and low against an unrelated refund passage.

The illustrative scores (0.71 vs 0.18) capture what good embeddings do — semantically related text scores clearly higher than unrelated text, even when they share almost no words. That gap is exactly what a model comparison is trying to measure across many query-document pairs.

Instruction prefixes are one of the most common silent footguns in embedding comparisons. Many strong modern models — the E5, BGE, and GTE families — were trained with explicit role tags like 'query:' and 'passage:' prepended to the text, so the model knows which side of an asymmetric search it's encoding.

If you omit the prefix, the model is being used differently than it was trained, and accuracy drops measurably but quietly. Countless 'this model underperformed' reports trace back to a missing prefix. The fix is free: read the model card and follow its encoding instructions exactly, for every model in your comparison.

This code shows correct asymmetric encoding for a prefix-based model. The query gets the 'query:' tag; each document gets the 'passage:' tag; both are normalized. This is the encoding contract these models expect, and honoring it is what lets them perform at the level their benchmark scores promise.

The broader lesson for comparisons is that each model may have its own contract. A fair test applies each model's required prefix and settings — comparing a correctly-prefixed model against an incorrectly-used one isn't measuring the models, it's measuring your setup.

The pipeline diagram lays out how a real comparison actually runs, mechanically. Start with an evaluation set of queries paired with their known-correct documents. Embed everything with the candidate model, run top-k search, then score the results with a metric like recall@k or nDCG.

Running this identical procedure across every candidate model — same eval set, same k, same metric — is what makes the comparison fair and the ranking meaningful. Post 4 turns this exact diagram into runnable code.

MTEB (the Massive Text Embedding Benchmark) is the standard public scoreboard, and understanding what it does and doesn't tell you is essential. It runs models across many task types — retrieval, clustering, classification, reranking, semantic similarity — over dozens of datasets, then reports per-task and average scores.

That makes it the best available cross-model signal for general capability. But it averages over generic, mostly-web data. A high MTEB score means 'broadly strong,' not 'best on your legal contracts or your support tickets.' Treat the leaderboard as a shortlist generator, then confirm the winner on your own data.

The summary ties the mechanics together. The training pairs and their hard negatives decide what the model considers similar. Pooling decides how token vectors collapse into one. Cosine on normalized vectors is the scoring rule. Prefixes are part of the model's contract, not optional decoration. And benchmarks rank models generally, not for your specific corpus.

Hold these five facts and most surprising comparison results stop being mysterious — they're usually explained by a training-data difference, a pooling or prefix mistake, or a benchmark that simply doesn't reflect your domain.

The CTA moves from theory to practice. Now that you know how these models learn and how comparisons are scored, the next post builds a real, runnable bake-off so you can rank three models on your own data instead of trusting a leaderboard.