Word2Vec & GloVe

This post introduces word embeddings as the foundational idea that made modern NLP possible. The cover frames it historically: before the transformer era of BERT and GPT, the breakthrough that made neural networks actually useful on language was learning to represent words as vectors instead of as arbitrary symbols.

The 'arrow' metaphor is deliberate. A word embedding is literally a direction and position in a high-dimensional space, and the entire power of the technique comes from the geometry of where words land relative to each other.

The definition is the anchor for the whole post. A word embedding is a dense vector — typically 100 to 300 floating-point numbers — that stands in for a word everywhere a model needs to process it. 'Dense' is the key word: every dimension carries a value, unlike the sparse one-hot encodings that came before.

The crucial property is that these vectors are learned, not designed. Word2Vec and GloVe read enormous amounts of raw text and adjust each word's vector until words used in similar ways end up near each other. Nobody hand-codes that 'happy' should be near 'glad'; it falls out of the statistics.

The distributional hypothesis is the philosophical core, and it's worth stating precisely: a word's meaning can be approximated by the distribution of contexts it appears in. Firth's 1957 line — 'you shall know a word by the company it keeps' — is the slogan, but the practical version is mechanical. Count what appears near each word, and words with similar neighbors get similar representations.

This is what lets the method work with zero supervision. There is no dictionary, no labeled meaning, no human annotation. The signal is entirely in co-occurrence patterns, which is why you can train these vectors on any raw text dump and get something useful.

This comparison contrasts the old and new ways of representing a word numerically. A one-hot vector has length equal to the vocabulary — 50,000 or more — and is all zeros except a single 1 marking which word it is. The fatal flaw: every pair of one-hot vectors is exactly equidistant, so the representation encodes identity but zero similarity. 'Cat' is as far from 'dog' as it is from 'democracy'.

Dense embeddings flip this. With a few hundred real-valued dimensions, the model can place 'cat' and 'dog' close while keeping 'democracy' far away. Meaning lives in the relative positions, and a downstream model can exploit that structure instead of treating every word as unrelated.

The vectors diagram makes the abstract idea concrete by plotting a few words in a tiny 2D version of the space. In reality the space has hundreds of dimensions, but the principle is identical: related words cluster. Here 'king' and 'queen' sit in one region, 'dog' and 'cat' in another.

Real embedding spaces are explored with dimensionality-reduction tools like t-SNE or PCA, which project the high-dimensional vectors down to 2D for visualization. When you do this on real Word2Vec vectors, you see exactly these clusters — animals together, royalty together, countries together — emerging purely from text statistics.

This slide introduces the property that made word embeddings famous: linear analogy structure. Because directions in the space are meaningful, you can do arithmetic on vectors. The offset from 'man' to 'woman' is roughly the same as the offset from 'king' to 'queen', so king − man + woman lands near queen.

It's important to keep this honest. The analogies work impressively often but not always, and the result depends on the corpus and dimensions. The deeper point for this post is that relationships — gender, tense, capital-of-country — get encoded as consistent directions, which is remarkable given the model only ever saw which words sit near which.

This comparison previews the two methods that the rest of the day explores in depth, so it stays at a high level. Word2Vec, from Mikolov's team at Google in 2013, is a predictive approach: a small neural network slides over text and learns by predicting context words. It processes the corpus locally, window by window.

GloVe, from Pennington, Socher and Manning at Stanford in 2014, takes a count-based approach. It first compiles global co-occurrence statistics for the entire corpus into a matrix, then factorizes that matrix so vector dot products reproduce the counts. Two philosophies — prediction versus counting — that converge on strikingly similar vectors.

This code slide gets the reader's hands dirty immediately so the concept feels real. Using gensim's downloader, you can pull a pretrained GloVe model in one line and start inspecting it. The shape (100,) confirms the abstract claim: each word really is just a list of 100 numbers.

The most_similar call is the payoff — it returns the nearest neighbors of 'king' by cosine similarity, and they're semantically sensible: queen, prince, throne. Seeing real neighbors come back from a real model is far more convincing than any diagram, and it sets up the deeper mechanics covered in later posts.

Setting boundaries is as important as the definition. The single biggest limitation is that these are static embeddings: a word gets exactly one vector regardless of context. 'Bank' by a river and 'bank' that holds money share the same point, an averaged compromise that fits neither sense cleanly. This is the explicit problem that contextual models like ELMo and BERT later solved.

They also have no model of word order or syntax — the window-based training only knows which words co-occur, not in what arrangement. And the vectors are just inputs; they are not a language model and cannot generate or reason. Naming these limits up front prevents the most common misuse.

The recap distills the whole post into five memorable lines for the carousel format. An embedding is a dense vector per word; it's built from context co-occurrence; similar usage produces nearby vectors; Word2Vec predicts while GloVe counts; and each word maps to one fixed vector.

These five points are the mental model a reader should walk away with. Everything in the following posts — the why, the mechanics, the code, the pitfalls — hangs off this scaffold.

The CTA closes the concept post and points forward to the 'why it matters' angle. Having established what embeddings are, the natural next question is why this representation was such a big deal that it reshaped a decade of NLP.

The teaser promises that payoff: the practical stakes — transfer learning, generalization, and cost — that made word vectors the default starting point for nearly every NLP system between 2013 and the transformer era.