Embeddings, Visually

This is the anchor idea for the entire day, so it is worth slowing down on. An embedding is nothing more exotic than a fixed-length list of numbers attached to a thing — a word, a sentence, an image, a product, a user. What makes it powerful is not the numbers themselves but the agreement we impose on them during training: that the position of each item should reflect its meaning.

The cover frames embeddings as 'a map' on purpose. A map turns the abstract idea of 'this town is related to that town' into something concrete you can measure with a ruler. Embeddings do the same for meaning: they give relatedness a geometry. Hold onto that picture, because every other slide in this day is just a consequence of it.

The precise definition matters because people often confuse an embedding with the model that produces it. The embedding is the output — a vector of d numbers. The model is the function that maps a raw item to that vector. The crucial, learned property is that the layout is meaningful: the encoder is trained so that items it should treat alike end up near each other.

That single property is what separates an embedding from an arbitrary feature vector. Any pile of numbers describes a thing; an embedding describes a thing in a way where distance equals dissimilarity. Once that holds, you can do search, clustering, and recommendation all with the same geometric machinery.

The leap people stumble on is going from 'list of numbers' to 'point in space.' But it is the same leap you already made in school: the pair (3, 4) is both two numbers and a single point on a grid. A vector of 768 numbers is just a point in a 768-dimensional grid. We cannot draw that, but nothing about the intuition changes — each number is a coordinate along one axis.

This reframing unlocks everything. Similarity becomes nearness. A cluster becomes a crowd of points. A relationship like 'capital city of' becomes a consistent direction you can travel. The geometry is real and computable even when it is unvisualizable, which is exactly why we lean on 2D pictures as crutches.

This diagram makes the abstract concrete with three arrows. 'king' and 'queen' point in nearly the same direction because the model has learned they appear in similar contexts and share most of their meaning. 'car' points off on its own, because it belongs to a different region of meaning entirely.

The takeaway is not the specific coordinates — those are illustrative — but the relationship between them. When you look at a real embedding space, you are looking at millions of arrows like these, arranged so that semantic neighbors are spatial neighbors. The vectors diagram is a deliberately tiny, honest window into that idea.

Here we name the core promise: distance is a proxy for relatedness. This is the property you will exploit in every downstream task. When you ask 'what is most similar to X,' you are really asking 'what is closest to X in the space.' The model never reasons about similarity symbolically; it simply arranges points so the answer falls out of the geometry.

It is worth internalizing that this is an emergent arrangement, not a rule someone wrote. The training objective rewards putting context-similar items together, and 'cat near kitten, cat far from spreadsheet' is the side effect. That is why embeddings can capture relationships nobody explicitly programmed.

The contrast between sparse and dense representations explains why embeddings exist at all. The old approach, one-hot encoding, gives every word its own dimension: a vocabulary of 50,000 words means 50,000-dimensional vectors that are all zeros except a single one. In that space, every pair of distinct words is exactly the same distance apart — there is literally no notion of similarity.

Dense embeddings collapse that into a few hundred shared dimensions where every number carries information. Because dimensions are shared, 'cat' and 'dog' can overlap heavily while 'cat' and 'galaxy' do not. The comparison diagram lays this out so the trade is obvious: you give up interpretability of individual dimensions and gain a space where geometry means something.

Expanding on why dense won: it is partly efficiency and partly expressiveness. One-hot vectors waste enormous memory storing zeros and scale linearly with vocabulary size. Dense vectors stay small regardless of vocabulary. But the deeper reason is expressiveness — one-hot encoding cannot represent that two things are similar, while dense embeddings can place them as close or as far as the data warrants.

This is the historical pivot that made modern NLP possible. Word2Vec and GloVe showed that dense, learned vectors captured analogies and clusters that one-hot vectors never could. Every embedding model since builds on that insight: pack meaning into a continuous space and let distance do the work.

The code is intentionally trivial because the point is conceptual: an embedding really is just an array. Printing its shape confirms it is a point in 4D space, and indexing into it shows that each entry is a coordinate along one axis. There is no hidden machinery in the object itself — the intelligence lives in how the numbers were chosen.

Running something this small is a useful grounding exercise. People build elaborate mental models of embeddings as mysterious objects; seeing that cat = np.array([...]) is the entire data structure removes the mystique and lets you focus on the part that actually matters, which is the training that placed the point.

This flow names the lifecycle you will see in every embedding system: a raw item goes into an encoder, the encoder emits a vector of d numbers, and then you exploit the geometry of that vector. The arrow from 'Vector' to 'Geometry' is where all the value is created — distance becomes meaning.

Keeping this pipeline in your head prevents a common confusion: the encoder and the geometry are separate concerns. You can swap encoders, but the downstream step — measure distance, find neighbors — stays the same. That separation is exactly why embeddings are such a flexible, reusable interface across tasks.

These five terms are the vocabulary you will reuse all week, so it helps to fix them now. Dimension is simply the length of the vector and the number of axes in the space. Dense means every number contributes, in contrast to mostly-zero sparse vectors. Embedding space is the high-dimensional room all the points live in.

Neighbor and similarity are the operational words: a neighbor is a nearby point, and similarity is how we score nearness, usually with cosine. Whenever a later slide says 'close' or 'far,' it is speaking this vocabulary. Lock these in and the rest of the day reads smoothly.

The recap compresses the post into five load-bearing claims. An embedding is a point in space; geometry encodes meaning; closeness means relatedness; dense representations beat one-hot for capturing semantics; and distance is the signal every downstream task exploits. If you remember nothing else, remember that distance equals relatedness.

These are not five separate facts — they are one idea viewed from five angles. The whole day, including the code and the failure modes, is just the working-out of 'meaning has a shape.' Internalize the picture and you will reason about embeddings correctly even in situations this post never covers.

This is the handoff to Post 2. Having established what an embedding is and why we picture it as a point, the natural next question is 'so what — why should I care?' The answer is that this one trick quietly powers semantic search, recommendations, and retrieval-augmented generation.

The teaser is meant to create a small itch: you now understand the object, but not yet its payoff. The next post fills that gap by showing how the simple move of 'embed, then find nearest neighbors' turns into systems you use every day.