Embeddings, Visually

This post is the mechanical core of the day, and it covers two distinct gears that people often conflate: how embeddings get trained, and how we measure distance once we have them. Both are simple individually, but mixing them up causes real confusion, so the post keeps them clearly separate.

The cover frames it as building and measuring the space. Building is the training process that places the points; measuring is the distance metric you apply at query time. You can change one without the other, and understanding that independence is half the battle.

The distributional hypothesis is the philosophical bedrock, summed up by Firth's line: 'you shall know a word by the company it keeps.' The claim is that meaning is largely determined by context — words that appear in similar surroundings tend to mean similar things. 'doctor' and 'physician' show up around the same words, so they should get similar vectors.

This is the principle that makes unsupervised embedding training possible. We never label what words mean; we just train a model to predict words from their contexts, and meaningful geometry falls out as a byproduct. Every embedding method from Word2Vec to modern sentence encoders is, at bottom, an operationalization of this hypothesis.

Modern embedding models, especially for sentences, are trained contrastively, and this slide explains the pull-and-push intuition. You feed the model pairs that should be similar — a sentence and its paraphrase, a question and its answer — and the loss pulls their vectors together. Simultaneously, unrelated pairs are pushed apart.

Repeat this over millions of examples and the space self-organizes: regions form, neighbors become meaningful, and distance becomes trustworthy. The important mental shift is that nobody places the points by hand. The objective specifies which things should be near or far, and gradient descent discovers an arrangement that satisfies those constraints.

The flow diagram traces one contrastive training step. An anchor sentence is paired with a positive (its paraphrase) and several negatives (unrelated text). The loss computes how to pull the anchor and positive closer while pushing the negatives away, and the update shifts all the vectors accordingly.

Seeing it as a loop clarifies why training takes so much data: each step only nudges things slightly, and the space only becomes coherent after enormous numbers of these pulls and pushes. It also explains why the choice of negatives matters so much — hard negatives, things that look similar but are not, teach the model the fine distinctions that make retrieval sharp.

Now we switch gears entirely to measurement. Once vectors exist, you need a way to score 'how close,' and the two dominant choices are Euclidean distance and cosine similarity. Euclidean is straight-line distance between two points; cosine is the angle between the two arrows from the origin.

The key distinction is that cosine ignores length and cares only about direction, while Euclidean is sensitive to both. Most text-similarity systems use cosine. The final bullet is the reconciling fact: if you normalize all vectors to unit length first, Euclidean and cosine rank results identically, because the only remaining difference between vectors is their direction.

The direction-versus-distance comparison spells out when to reach for each metric. Euclidean measures the gap in position and is sensitive to vector length, which suits raw coordinate data where magnitude is meaningful. Cosine measures angle only, is length-invariant, and suits text where two documents on the same topic can have very different magnitudes.

The practical guidance: default to cosine for semantic embeddings unless you have a specific reason not to. The 'longer vector = farther' note under Euclidean is the trap — it means an unusually long vector can look distant from everything purely because of its length, not its meaning, which is rarely what you want.

This slide answers why direction usually beats magnitude for text. In learned text embeddings, a vector's length often correlates with incidental properties — how long the text is, how frequent its words are — rather than its topic. Two documents about the same subject can differ wildly in length and therefore in vector magnitude.

Cosine similarity discards length and keeps only direction, so those two documents score as highly similar, which matches our intuition. This is the empirical reason cosine dominates text retrieval. It is not a law of nature; it is a good match between the metric's invariance and the way these particular embeddings encode meaning.

The side-by-side code makes the metric difference impossible to ignore. Vectors a and b point in exactly the same direction, but b is twice as long. Euclidean distance reports 1.414 — they look 'far apart.' Cosine reports 1.0 — they are treated as identical.

This tiny example captures the entire intuition. If your application considers a = [1,1] and b = [2,2] to mean the same thing, you want cosine. If their different magnitudes are meaningful, you want Euclidean. Running it once cements which metric encodes which assumption, and that clarity prevents a whole class of retrieval bugs.

The analogy code revisits the most famous demonstration in embeddings: king minus man plus woman lands near queen. It works because the training process encoded consistent directions for concepts like royalty and gender. Subtracting 'man' and adding 'woman' shifts along the gender direction while preserving the royalty direction.

The vectors here are illustrative rather than from a real model, but the arithmetic is exactly what you would run on real embeddings. The deeper lesson is that relationships live as directions in the space, which is why vector arithmetic can express analogies at all. Modern sentence embeddings show this less cleanly than classic word vectors, but the underlying geometry is the same.

The vectors diagram shows the analogy as a parallelogram. king and queen sit at the top, man and woman at the bottom, and the vector from man to king is parallel to the vector from woman to queen. That parallelism is the visual signature of a consistent relational direction.

When people say embeddings 'capture relationships,' this picture is what they mean. The same offset that turns 'man' into 'king' turns 'woman' into 'queen,' so the relationship is a reusable translation in space. Seeing it as parallel arrows makes the abstract claim about analogies concrete and memorable.

The recap separates the two gears one last time. On training: similar context yields similar vectors, and contrastive learning pulls positive pairs together while pushing negatives apart. On measurement: cosine measures angle, Euclidean measures gap, direction usually beats magnitude for text, and analogies live as vector arithmetic.

Keeping these two columns distinct in your head is the durable takeaway. When something goes wrong, you can ask the right question: is this a training problem (the space is laid out badly) or a measurement problem (I picked the wrong metric or forgot to normalize)? That diagnostic split is exactly what the next post's failure modes build on.

This hands off to Post 4. You now understand how the space is built and how distance is measured, which is precisely the background needed to build something real. The teaser promises hands-on code: embed actual sentences, rank them, and plot the space.

The timing is intentional. Mechanics are most memorable right before you apply them, so the next post turns this engine-room knowledge into a small, runnable semantic search you can modify and feel respond.