Word2Vec & GloVe

This is the code-heavy post, and the cover sets the expectation clearly: enough theory, time to run it. The promise is a complete, copy-paste loop in under thirty lines that goes from raw sentences to trained vectors to the famous analogy result.

The goal of this post is muscle memory. Reading about negative sampling is one thing; watching real vectors come out of model.wv and querying them yourself is what makes the concept stick permanently.

The setup slide keeps things minimal. gensim is the standard, well-maintained Python library for Word2Vec and related embedding methods, and it bundles both training and a downloader for pretrained vectors. One pip install and two imports is the entire dependency footprint.

gensim.downloader (aliased api) is worth highlighting — it lets you fetch large pretrained models like Google News Word2Vec or Stanford GloVe with a single call, which is essential for the later slides where a tiny toy corpus can't produce meaningful results.

This slide prepares the input data and clarifies the expected format: gensim's Word2Vec takes an iterable of token lists, where each inner list is one sentence already split into lowercased words. The toy corpus here is deliberately tiny and structured so 'king'/'queen' and 'man'/'woman' share contexts.

The comment is an honest disclaimer — in real use you stream millions of sentences, often via a generator so the corpus never fully loads into memory. The four hand-written sentences exist only to make the API mechanics visible; they're far too small to yield good vectors, which the next slides address by loading pretrained models.

This is the core training call, and each argument matters. vector_size=100 sets the embedding dimension. window=5 defines the context radius. min_count=1 keeps every word (only acceptable for a demo; in production you raise it to drop noise). sg=1 selects skip-gram over CBOW. negative=5 sets the number of negative samples per real pair. epochs=50 compensates for the tiny corpus.

The single line model.wv extracts the trained KeyedVectors object — the actual lookup table of word vectors. Everything downstream operates on wv, which decouples the trained vectors from the training scaffolding so you can save and reuse just the vectors.

This slide demonstrates the two most common queries. similarity('king', 'queen') returns a cosine score, and most_similar returns the nearest neighbors by cosine. These are the bread-and-butter operations for inspecting any embedding model.

The inline comment is an important honesty check: on a four-sentence corpus the numbers will be noisy and not very meaningful. The point of the slide is the API and the workflow, not the specific values — which is exactly why the analogy slide switches to a properly pretrained model.

This slide pauses the code to explain the knobs, because choosing them well is most of the practical skill. vector_size trades richness against size and overfitting risk; 100 to 300 is the standard range. window controls how local or broad the notion of context is. sg=1 picks skip-gram, generally better for rare words. negative sets how many fakes anchor each real pair.

min_count is the one most people get wrong: it drops words appearing fewer than N times. Setting it to 1 (as in the demo) keeps noise and rare typos; in real corpora you raise it to 5 or higher to get cleaner, better-trained vectors and a smaller vocabulary.

This is the payoff slide — the analogy everyone has heard about, run for real. Crucially, it first loads word2vec-google-news-300, a properly trained 300-dimensional model, because the toy corpus from earlier cannot produce this result. The most_similar call with positive=['king','woman'] and negative=['man'] computes king − man + woman and returns the nearest word.

The result, queen with similarity around 0.71, is the concrete demonstration that relationships are encoded as consistent vector directions. Running this yourself is the single most convincing moment in the whole topic, which is why it gets its own slide with a real model.

The vectors diagram visualizes why the arithmetic works. Plot 'king', 'man', 'woman', and 'queen' as a rough parallelogram: the vector from 'man' to 'king' is approximately equal to the vector from 'woman' to 'queen'. That shared offset is the 'royalty' direction, and the gender difference is another consistent direction.

When you compute king − man + woman, you remove the 'male' component and the lone-word component, add back the 'female' component, and land near where 'queen' sits. The parallelogram is the geometric intuition behind the headline result, made visual.

This slide solves the practical problem most readers hit: you rarely have a big enough corpus of your own. gensim's downloader pulls Stanford's GloVe vectors trained on Wikipedia and Gigaword in one line, giving you high-quality embeddings instantly.

The demo shows two handy methods: most_similar('paris') returns related cities and country terms, and doesnt_match picks the outlier from a list — here 'car' among meal words. doesnt_match is a quick, intuitive way to sanity-check that a model's geometry is sensible, and it's a nice party trick that also demonstrates real semantic structure.

The production tips translate the demo into real-world practice. Train on a large, domain-relevant corpus — medical text for medical embeddings, not generic news. Raise min_count to filter noise. Use 300 dimensions as a robust default. Persist your model with model.save() and reload with Word2Vec.load() so you don't retrain.

The final tip is the pragmatic escape hatch: if you don't have a big corpus or the time, just use pretrained GloVe or Word2Vec vectors. For most applications, well-trained public vectors beat poorly-trained custom ones, and they're free.

The CTA closes the hands-on post and sets up the final angle: the failure modes. Now that the reader can train and query embeddings, the responsible next step is learning where they quietly go wrong.

The teaser lists the traps the last post covers — polysemy, out-of-vocabulary words, bias, and cosine-similarity gotchas — the exact issues that turn a working demo into a broken production feature if ignored.