Word2Vec & GloVe

This final post is about failure modes, and the cover states the stakes bluntly: the fastest way to ship a broken NLP feature is to trust word vectors without knowing their limits. The previous posts built up the power of embeddings; this one is the necessary counterweight.

The framing is deliberately practical. These aren't academic caveats — each limitation here has caused real production incidents, from search engines surfacing the wrong sense of a word to hiring tools amplifying bias. Knowing them is the difference between using the tool well and getting burned.

Polysemy is the most fundamental limitation of static embeddings. Because each word gets exactly one vector, a word with multiple meanings — 'bank', 'apple', 'bat', 'spring' — collapses all its senses into a single averaged point that represents none of them well. The vector for 'bank' is a blurry blend of riverbank and financial-institution usage.

The practical fix is to recognize when your task actually depends on sense disambiguation. If it does, you need contextual embeddings like ELMo or BERT, where the same word gets different vectors depending on the surrounding sentence. Word2Vec and GloVe simply cannot represent context-dependent meaning, and no amount of tuning changes that.

The flow diagram visualizes the blurring. 'River bank' (sense A) and 'money bank' (sense B) both feed into the one vector the model has for 'bank', and the result is an average that fits neither sense cleanly. A search for 'bank' on a fishing site and 'bank' on a finance site would retrieve the same muddy neighbors.

This picture is the single clearest way to internalize the limitation. The information loss happens at training time and is irreversible at query time — once the senses are averaged into one point, you can't recover them from the static vector alone.

Out-of-vocabulary words are the second cliff. Classic Word2Vec and GloVe build a fixed vocabulary at training time, and any word not in it simply has no vector. In code, indexing a missing word raises a KeyError. New slang, brand names, domain jargon, and typos all fall into this gap.

The slide names the modern fix: FastText, which represents words as bags of character n-grams. Because it composes a vector from subword pieces, it can construct a reasonable vector for a word it never saw, as long as it shares character chunks with known words. If OOV is a real risk in your domain, FastText or subword-based models are the answer.

This code slide shows the defensive pattern every production user needs. Naively writing wv['someword'] crashes on any unknown word, which is unacceptable in a live system fed unpredictable user input. The safe_vec helper checks membership in key_to_index first and returns None for OOV words, letting the caller handle the miss gracefully — skip the word, fall back to a default, or log it.

This tiny guard is the kind of detail that separates a notebook demo from production code. The example confirms that a gibberish string returns None rather than throwing, so downstream code stays robust.

Bias is the most consequential limitation, because it's invisible until it causes harm. Embeddings learn from human-written text, so they faithfully absorb the stereotypes embedded in that text. The widely-cited finding from Bolukbasi et al. is that 'man : computer_programmer :: woman : homemaker' — the gender bias is literally encoded as a vector direction.

The danger is amplification at scale. Drop raw vectors into a resume screener, a search ranker, or a recommendation engine and you can systematically disadvantage groups, multiplied across every decision the system makes. The responsible practice is to audit embeddings for bias and apply debiasing techniques before deploying anything that affects people.

The pipeline diagram traces how bias enters and spreads, mirroring the structure of a typical fairness discussion. It starts in the biased human corpus, gets copied faithfully into the vectors during training, and then gets amplified downstream as the embeddings drive automated decisions at scale.

The key insight is that the bias isn't introduced by a buggy algorithm — the algorithm is doing exactly its job of capturing patterns in the data. That's why you can't fix it by 'fixing the model'; you have to either curate the training data, debias the vectors, or add fairness constraints downstream. Knowing where bias enters tells you where you can intervene.

This slide attacks a subtle and very common mistake: assuming that nearby vectors mean similar things. They mean similar usage, which is not the same. 'Good' and 'bad' are extremely close in embedding space because they appear in identical contexts — 'the movie was ___', 'a ___ idea'. Antonyms are often near-neighbors.

The practical consequence bites in sentiment and filtering tasks. A naive rule like 'words near a positive word are positive' will happily classify 'bad' as positive because it sits next to 'good'. Embeddings capture relatedness and analogy, not polarity, and conflating the two produces baffling bugs.

This slide covers two operational gotchas that silently degrade results. First, always use cosine similarity rather than raw Euclidean distance: a vector's magnitude often correlates with word frequency rather than meaning, so cosine — which ignores magnitude and compares direction — is the right metric for semantic similarity.

Second, your preprocessing must match how the vectors were trained. If the embeddings were trained on lowercased text, querying 'King' will miss. If the training tokenizer split contractions a certain way, 'don't' versus 'do' + 'n't' matters. Mismatched casing, tokenization, or punctuation handling causes silent lookup failures that quietly hollow out your model's coverage.

This code slide makes the cosine point concrete and shows the correct implementation: dot product divided by the product of the norms. It also notes that gensim's similarity method already does this for you, so in practice you rarely hand-roll it — but understanding the formula explains why magnitude is ignored.

The reason this matters: if you accidentally compare raw vectors with Euclidean distance, frequent words with large-magnitude vectors can appear artificially far from or close to others for reasons that have nothing to do with meaning. Standardizing on cosine removes that confound and is the universal default for embedding similarity.

The takeaways compress the five failure modes into actionable rules. If polysemy matters, switch to contextual models. Handle OOV explicitly or use FastText. Audit for bias before shipping anything sensitive. Remember that near does not mean synonym — antonyms cluster. And always use cosine similarity while matching the training preprocessing.

These five rules are a practical checklist. A reader who applies them avoids the great majority of real-world embedding bugs, which is the entire purpose of a 'common mistakes' post.

This CTA closes both the post and the day, and points to the next topic. Word2Vec and GloVe's defining weakness — one vector per word — is precisely the problem the next day's subject solves.

The teaser sets up contextual embeddings: ELMo and BERT generate a different vector for each occurrence of a word based on its sentence, fixing the polysemy problem head-on. It's the natural narrative bridge from static to contextual representations, and the historical arc of NLP itself.