Distance Metrics: L1, L2, Cosine

This cover frames the final post as a field guide to failure, and the framing is deliberate: distance bugs are dangerous precisely because they don't crash. They quietly skew results, and skewed results at production scale are expensive — worse recommendations, degraded search, mis-clustered data — all while every individual computation remains technically correct.

The post catalogs the handful of failures that account for the vast majority of real-world distance bugs. Each one produces plausible-looking output, which is exactly why they slip through code review and persist unnoticed. Knowing them in advance is the cheapest insurance against shipping a silently broken similarity system.

The single most common distance mistake is failing to standardize features before an L1 or L2 computation. Because these metrics sum contributions across all columns, a feature with a large numeric range overwhelms everything else. Salary in dollars (tens of thousands) completely drowns out age in years (tens), so your distance effectively becomes a distance in salary alone.

The insidious part is that nothing signals the problem — the code runs and returns numbers. The fix is mechanical and non-negotiable: standardize (subtract mean, divide by standard deviation) or normalize every feature before any Euclidean-style metric, so each feature contributes on equal footing. Make this a reflex for every k-NN, k-means, or distance-based pipeline.

This snippet demonstrates the scaling failure and its fix concretely. With raw features, the L2 distance between two people differing by 35 years of age and 2,000 dollars of salary comes out as roughly 2,000 — almost entirely the salary gap, with age contributing essentially nothing. The age dimension has been silently erased from the distance.

Applying StandardScaler transforms both columns to comparable scales (zero mean, unit variance), so the 35-year age gap and the 2,000-dollar salary gap now contribute proportionally to their importance rather than their raw units. The contrast between the raw distance and the scaled features is the clearest possible illustration of why standardization is mandatory, not optional.

The second common mistake is the mirror image of the first: using cosine when magnitude actually carried information. Cosine deliberately discards length, so if two entities point in the same direction they are judged identical regardless of scale. Usually that's the goal, but sometimes the scale is the signal.

Consider two users who like the same genres, but one has rated 500 movies and the other only 3. Their preference vectors point the same way, so cosine calls them identical — yet the activity level is hugely informative and the cosine view throws it away entirely. When magnitude carries meaning (engagement, spend, counts), use a scaled L2 instead so that the size of the vector continues to matter.

The third mistake is trusting Euclidean distance in very high-dimensional spaces. The curse of dimensionality causes L2 distances between all pairs of points to concentrate toward nearly the same value as dimensionality climbs into the hundreds or thousands. When everything is roughly equidistant, the concept of a meaningful "nearest neighbor" dissolves.

The practical responses are to reduce dimensionality first (with PCA, UMAP, or a learned embedding) so Euclidean geometry regains its contrast, or to switch to cosine, which compares direction and holds up better in the high-dimensional embedding spaces that modern ML inhabits. Blindly running k-NN or k-means with L2 on raw high-dimensional data is a recipe for results that look fine but mean little.

This bar chart visualizes dimensional concentration. In two dimensions there is sharp contrast between near and far points. By fifty dimensions the contrast is blurring, and by a thousand dimensions the distances between near and far pairs have collapsed to nearly the same value — the bars shrink toward a flat line.

The visual drives home why "nearest neighbor" loses meaning at high dimensions: if the nearest and farthest points are almost equidistant, ranking by distance conveys almost no information. It is the strongest single argument for dimensionality reduction or for switching to cosine before doing distance-based work in high-dimensional spaces.

The fourth mistake is the zero-vector divide-by-zero, a failure mode unique to cosine. Because cosine divides by each vector's length, a vector of all zeros — an empty embedding, an empty TF-IDF row for a document with no recognized tokens, a padding vector — has length zero and triggers a division by zero, producing NaN or crashing the computation.

These zero vectors sneak in from real data: empty documents, out-of-vocabulary inputs, or padding in batched pipelines. Unlike the scaling mistakes, this one can be loud (a crash) or subtly poisonous (a NaN that propagates and corrupts downstream aggregates). The defense is to guard the norm or filter out zero rows before computing cosine, as the next slide shows.

This snippet implements a safe cosine that guards against the zero-vector trap. Before dividing, it checks whether either vector's norm falls below a tiny epsilon; if so, it returns 0.0 — treating an undefined comparison as "no similarity" rather than letting a NaN escape.

The choice to return 0.0 is a reasonable default: a zero vector has no direction, so claiming zero similarity is sensible and keeps downstream code numerically stable. The broader lesson is to make your distance functions defensive about degenerate inputs, because real data will eventually feed them an empty or all-zero vector, and a single NaN can silently corrupt an entire batch of results.

The fifth mistake is pairing a metric with an algorithm that mathematically assumes a different one. k-means is the prime example: its update step moves each centroid to the mean of its assigned points, and that mean-based update provably minimizes squared Euclidean distance and nothing else. Substituting cosine into vanilla k-means is internally inconsistent — the assignment and update steps optimize different objectives.

The correct approach for angular clustering is to L2-normalize the vectors first, which makes Euclidean k-means equivalent to clustering by angle (this is spherical k-means), or to use an algorithm explicitly designed for cosine. The general principle is to verify that your metric is compatible with the assumptions baked into your algorithm, rather than swapping it in because the API permits it.

This decision tree compresses the entire day into a single actionable flow. The first question is whether vector length carries meaning. If yes, the follow-up is whether features are on the same scale: if they are, use L2 (or L1 when outliers are a concern); if not, scale the features first and then use L2. If length does not carry meaning, use cosine and normalize first.

This is the diagram to memorize. It captures the magnitude-versus-direction decision, the scaling requirement, and the normalization step in one walk-through. A practitioner who internalizes this tree can make the right metric choice for almost any similarity task in a few seconds of reasoning.

The final mistake circles back to a mathematical subtlety with practical teeth: cosine distance violates the triangle inequality, so it is not a true metric. Algorithms and data structures that rely on the triangle inequality — ball trees, certain pruning bounds, some clustering correctness arguments — can misbehave or give wrong results when handed raw cosine distance.

The remedies are to use angular distance, defined as arccos(similarity)/π, which is a proper metric, or to run L2 on normalized vectors, which is both a true metric and ranks identically to cosine. Knowing this prevents the subtle class of bugs where an index or algorithm silently relies on a property cosine doesn't provide.

This checklist is the takeaway artifact for the whole day, condensing the failure modes into five pre-flight questions. Are features scaled before L1/L2? Does magnitude matter — and if so, are you avoiding cosine? Are you in high dimensions, and if so have you reduced or switched to cosine? Are there zero vectors to guard against? And does the metric match the algorithm's assumptions?

Running through these five questions before deploying any distance-based system catches the overwhelming majority of metric bugs before they ship. It transforms the day's conceptual, mathematical, and code-level lessons into a habit that prevents the silent failures that make distance metrics so quietly dangerous.

The CTA closes both the post and the topic. Day 15 begins a new subject, but the framing emphasizes that these three metrics are not a one-off lesson — they are tools the reader will carry into everything that ranks by similarity, from search to clustering to recommendation to retrieval.

The message is that distance metrics are foundational infrastructure for a huge swath of ML, and the understanding built over these five posts — what they measure, why the choice matters, the math behind them, how to compute them, and how they fail — will keep paying off long after the series moves on.