Embeddings, Visually

The final post is a field guide to the ways embeddings fail, and the framing is the most important sentence in it: embeddings fail quietly. The code runs without errors, the numbers look reasonable, and the results are subtly wrong. That combination — plausible output, hidden defect — is exactly what makes these bugs dangerous and hard to catch in review.

The cover deliberately uses the word 'lie' to set the tone. The model is not malicious; it is faithfully reporting geometry that does not mean what you assume. Each of the five mistakes that follow is a place where the geometry and your assumptions quietly diverge.

Mixing models is the most common and most damaging mistake. Two models can both output 768-dimensional vectors, but those vectors live in completely different, incompatible spaces — the axes mean different things. Comparing a vector from model A with one from model B via cosine produces a number, and that number is noise.

The fix is a discipline, not a trick: embed your query and your corpus with the exact same model and version, and re-embed everything whenever you upgrade the model. This last part bites teams in production, where someone updates the encoder for new documents but forgets the old index was built with the previous version, silently corrupting retrieval.

Trusting a 2D projection is a subtler trap. Tools like PCA and t-SNE exist to make high-dimensional spaces viewable, but they necessarily discard most of the information. Points that appear adjacent in 2D can be far apart in the true 768-dimensional space, and the reverse happens too.

t-SNE deserves special caution: it is tuned to preserve local neighborhoods, so the distances between clusters and the sizes of clusters in a t-SNE plot are not meaningful. Use these projections to build intuition and to spot gross problems, but never make a ranking, threshold, or business decision based on apparent distances in a 2D plot.

This comparison contrasts the real space with the 2D plot so the loss is explicit. The real 768-dimensional space has the true distances, keeps all relationships intact, and is what the model and your queries actually use. The 2D plot has distorted distances and crowding artifacts; it is good for a glance and bad for decisions.

The practical rule that falls out: query against the full-dimensional vectors, visualize with the projection. Never let the picture you can see override the geometry the system actually computes. The plot is a communication and debugging aid, not the source of truth, and treating it otherwise is how people convince themselves of clusters that are partly artifacts of the projection.

Forgetting to normalize is a quiet numerical bug. Cosine similarity is defined to ignore vector length and measure only direction, but if your index computes a raw dot product on un-normalized vectors, longer vectors get inflated scores and dominate the results regardless of relevance. A verbose document can outrank a perfectly on-topic short one purely because its vector is longer.

The fix is to normalize vectors to unit length before indexing, or to use an index configured for cosine similarity. Once vectors are unit length, dot product and cosine are identical, so the ambiguity disappears. Many vector databases offer a metric setting precisely so you do not get this wrong; choosing cosine there is the safe default for text.

This code makes the normalization issue concrete. Vectors a and b point in the same direction, but a is ten times longer. Their raw dot product is 10.0, which a dot-product index would read as a very strong match driven entirely by magnitude. After normalizing both to unit length, the score is 1.0 — the honest answer, since they point the same way.

The lesson is that the metric and the preprocessing must agree. If you intend cosine semantics, either normalize up front or use a cosine index; do not feed un-normalized vectors into a dot-product search and expect cosine behavior. Running this snippet once makes the failure mode visceral and easy to remember.

The curse of dimensionality is the most theoretical trap but it has real consequences. In very high-dimensional spaces, the distances between randomly placed points concentrate — the nearest and farthest neighbors end up at almost the same distance, so 'closest' loses its discriminating power. Naive distance on raw high-dimensional data can become nearly useless.

Learned embeddings fight this because training deliberately structures the space so meaningful neighbors really are closer. But the phenomenon still explains why high dimensionality is not free, why approximate-nearest-neighbor indexes are needed for speed at scale, and why people sometimes reduce dimensionality before indexing. It is the reason 'just use more dimensions' is not automatically better.

The bars diagram visualizes distance concentration as dimensions grow. In 2D, the contrast between the nearest and farthest points is large and obvious. By 50 dimensions it has shrunk, and by 1000 dimensions almost everything is roughly equidistant, leaving little signal to rank on.

The chart is schematic, not measured, but it captures the right trend: more dimensions are not automatically more discriminating, and beyond a point they can hurt naive distance. This is the intuition behind choosing sensible embedding dimensions and using indexes designed for high-D search, rather than assuming bigger vectors always mean better retrieval.

The final and most insidious mistake is treating 'close' as 'correct.' Embeddings encode whatever patterns lived in their training data, including biases, stereotypes, and spurious correlations. Two items can be neighbors for reasons that have nothing to do with what you care about — shared boilerplate, similar length, or a learned association you would rather not reinforce.

The correct stance is that nearest-neighbor results are a hypothesis, not ground truth. They are usually good, which is precisely why the failures are easy to miss. Validate retrieval quality with real examples and held-out queries, and be especially skeptical when neighbors line up with a sensitive attribute, because the model may have learned a shortcut you did not intend.

This guardrail code turns the post's two most preventable mistakes into a safety check. It asserts that the two vectors came from the same model and have matching shapes before doing anything, then normalizes both with a small epsilon to avoid division by zero, and finally returns a clean cosine score.

The value is in the assertions, not the arithmetic. They convert silent, plausible-looking corruption — mixed models, dimension mismatches — into a loud, immediate failure at the exact line where the bug lives. Wrapping comparison logic in checks like this is the difference between catching a problem in development and shipping subtly wrong results to users.

The recap turns the five traps into five rules: use the same model and version for everything, treat plots as intuition rather than decisions, normalize before cosine or dot product, mind dimensionality at scale, and validate that your neighbors are actually right. Each rule maps directly to one failure mode from the post.

These are the habits that separate a demo that looks impressive from a system you can trust in production. None of them are advanced; they are disciplines. The reason they matter is the same reason the whole post exists — embeddings fail quietly, so the only defense is to build the checks in deliberately rather than wait for the geometry to mislead you.

This is the close of the day and the handoff to Day 53. The teaser keeps it open because the series moves to a new topic next, while encouraging the reader to maintain the streak. The implicit message is that the embedding mental model you built over these five posts is now a permanent tool.

It is worth restating what that tool is: meaning has a shape, distance measures relatedness, and nearly every modern AI system you touch is exploiting that geometry. Carry those ideas forward, stay alert to the quiet failure modes, and the rest of the deep-learning material in this series will keep connecting back to this foundation.