Word2Vec & GloVe

This is the mechanics post, and the cover poses the central engineering puzzle: how do you teach meaning using nothing but a sliding window over text and gradient descent? The answer — turn it into a prediction game — is elegant once you see it, which is the satisfaction this post delivers.

Framing it as a 'guessing game' is accurate. The model never receives the meaning of a word; it only ever tries to guess which words appear nearby, and the byproduct of getting good at that game is a meaningful vector for every word.

This comparison lays out Word2Vec's two architectures before going deep, because they're easy to confuse. Skip-gram takes a center word as input and tries to predict each surrounding context word. It tends to work better on rare words and is the more widely used default.

CBOW (Continuous Bag of Words) inverts the task: it takes the context words and predicts the missing center word. It trains faster and tends to smooth over frequent words. The two share all the same machinery downstream; they differ only in which direction the prediction runs.

The first concrete step is generating training data from raw text, and it's purely mechanical. You slide a window of fixed radius — say two words on each side — across the corpus. For each position, the center word pairs with every word inside the window to form a training example.

The worked example on 'the cat sat on the mat' shows pairs like (cat, the), (cat, sat), (sat, cat), (sat, on). Notice there are no labels here in the supervised sense — the 'label' is just which words actually co-occurred, which is why this is self-supervised. The text generates its own training signal.

The network diagram shows skip-gram as a deliberately tiny neural net: one input (the center word, as a one-hot), a single hidden layer that IS the embedding, and an output layer producing a probability over context words. The magic is that the hidden-layer weight matrix, after training, is exactly the table of word vectors.

There's no deep stack here — it's essentially a single linear projection followed by a softmax. The embedding is a side effect of training this shallow predictor, which is part of why Word2Vec was so fast and influential compared to heavier language models of its time.

This slide names the bottleneck that the next step solves. The clean version of the skip-gram objective uses a softmax over the entire vocabulary: to score one (center, context) pair you compute and normalize across all 50,000+ words. That normalization is the killer.

With millions or billions of training pairs, doing a full-vocabulary softmax on every step is computationally hopeless. The whole practicality of Word2Vec hinged on finding a way to avoid this per-step cost — which is exactly what negative sampling provides.

Negative sampling is the key trick and deserves careful framing. Instead of a full softmax, reframe training as binary classification. For each genuine (center, context) pair, draw k random words that did not appear in the window — the 'negatives'. The model learns to output YES for the real neighbor and NO for the fakes.

The payoff is enormous: each training step updates only about k+1 vectors instead of all 50,000. Typical k is 5 to 20. The negatives are sampled with a frequency-smoothed distribution so common words don't dominate. This single approximation is what made training on billion-word corpora feasible.

This code slide turns the prose into the actual loss function. For a center vector v, the true context vector u_pos, and k negative vectors u_negs, the objective maximizes the sigmoid of v·u_pos (push the real pair together) while maximizing the sigmoid of −v·u for each negative (push the fakes apart).

The two terms encode the YES/NO intuition precisely. The first log term rewards a high dot product with the true neighbor; the summed term rewards low dot products with the random negatives. Gradient descent on this objective, repeated over the whole corpus, is all it takes to produce the famous vectors.

Here the post pivots to GloVe, which reaches similar vectors by a completely different philosophy. Rather than streaming windows and predicting, GloVe first scans the whole corpus once to build a co-occurrence matrix X, where X_ij counts how often word j appears near word i.

Then it learns vectors whose dot product approximates log(X_ij), using a weighting function that down-weights both extremely rare and extremely frequent pairs. This is matrix factorization on global statistics, not local prediction — it sees the entire corpus at once instead of one window at a time, which is its conceptual signature.

The flow diagram resolves the apparent paradox of two different methods yielding similar results. Word2Vec predicts neighbors; GloVe factorizes counts; yet research (notably Levy and Goldberg) showed that skip-gram with negative sampling is implicitly factorizing a shifted pointwise-mutual-information matrix.

So under the hood, both methods are doing a form of matrix factorization over co-occurrence statistics — they just arrive there by different routes, one online and predictive, one batch and explicit. That's why their resulting embedding geometries look so alike, and why the choice between them is often practical rather than fundamental.

The recap nails the mechanics into five lines for the carousel. The window makes (center, context) pairs; skip-gram predicts context from the center word; negative sampling replaces the full softmax with a cheap binary task; GloVe factorizes a co-occurrence matrix instead; and both end up producing meaningful vector geometry.

These are the load-bearing ideas. A reader who internalizes them understands not just that embeddings work but the specific computational shortcut and the count-based alternative that made them practical.

The CTA closes the mechanics post and hands off to the hands-on code post. Having seen the algorithms in principle, the natural next move is to run them.

The teaser promises the concrete payoff: training your own vectors with a few lines of gensim and reproducing the king − man + woman analogy that everyone has heard about but few have actually executed.