Distance Metrics: L1, L2, Cosine

This cover sets the tone for a hands-on, code-heavy post. The argument is that intuition about distance metrics only really lands when you compute all three on the same data and watch them produce different answers. Reading formulas is necessary but not sufficient; running them is what makes the difference stick.

The post is structured as a progression: implement the metrics, vectorize them, verify against a trusted library, then demonstrate the payoff with a k-NN classifier that flips its prediction purely from a metric swap. The reader is explicitly encouraged to run, modify, and break the code, because experimentation is where the understanding consolidates.

This first snippet implements all three metrics from scratch in a few lines each, then computes them on the running example pair [3,1] and [1,3]. The output — 4.0, 2.828, 0.4 — gives concrete numbers to anchor against. L1 is the largest because the grid walk is longer than the straight line; cosine distance of 0.4 reflects a moderate angle between the vectors.

Implementing them by hand, rather than importing, is pedagogically deliberate. Writing np.abs(x-y).sum() for L1 and the normalized dot product for cosine forces you to see that each metric is just a short arithmetic recipe. Once you've typed them, the formulas stop being abstract symbols and become operations you control.

Real workloads never compute one distance at a time, so this snippet shows the vectorized pattern. Subtracting X[0] from the whole array and reducing along axis 1 computes the L2 distance from the first row to every row at once, with no Python loop. The cosine block normalizes every row to unit length and then a single matrix multiply U @ U.T produces the full pairwise cosine-similarity matrix.

This is the form you actually ship. The keepdims=True argument is the small but crucial detail that keeps the norms shaped for broadcasting. Internalizing this pattern — broadcast the differences, reduce along the feature axis, or normalize-then-matmul for cosine — is what lets you compute thousands of distances efficiently instead of crawling through nested loops.

Verification against a trusted implementation is a habit worth building, and this snippet cross-checks the from-scratch numbers against scikit-learn's pairwise distance functions. manhattan_distances returns 4.0, euclidean_distances returns 2.828, and cosine_distances returns 0.4 — matching the hand-written results exactly.

The value here is twofold. First, it confirms the from-scratch implementations are correct, which builds confidence that you understand the formulas. Second, it introduces the production-ready API: in real code you'll usually call these library functions rather than your own, because they are optimized and battle-tested. Knowing both the manual version and the library version means you understand what the library is doing under the hood.

This is the punchline of the entire day, expressed in code. A single 1-nearest-neighbor classifier is fit on three points and queried with the same point [8,8] under two different metrics. Under Euclidean, the query's nearest neighbor is [10,10] (class 0) because it is closest in straight-line distance. Under cosine, the nearest neighbor is [1,1] (class 1) because [8,8] points in exactly the same direction as [1,1].

Same data, same query, opposite prediction — produced by nothing but the metric argument. This demonstration is far more convincing than any prose, because it shows the metric directly determining a model's output. It is the concrete proof that choosing a metric is a real modeling decision with real consequences.

This vector diagram explains the k-NN flip geometrically. The query [8,8] sits close to [10,10] in straight-line terms, which is why Euclidean picks class 0. But [8,8] points in precisely the same direction as [1,1] (both lie along the 45-degree line), which is why cosine, caring only about direction, picks class 1.

The diagram makes the disagreement visually obvious: position-based proximity and direction-based alignment point to different neighbors. This is the same tension introduced on day one of this topic, now made fully concrete. Seeing it drawn reinforces that the two metrics are not competing approximations but genuinely different questions about the same three points.

This snippet proves the normalization equivalence from the math post in code. After scaling every row of a random matrix to unit length, the squared L2 distance between any two rows equals exactly 2 − 2·cos(θ). The print shows the two quantities matching to four decimal places.

The demonstration matters because the equivalence underpins how production vector search works. Systems normalize embeddings once, then use fast Euclidean or dot-product indexes to serve what are semantically cosine queries. Running this snippet turns an abstract identity into a verified fact you've seen with your own eyes, which is far more durable than taking it on faith.

This slide steps back to interpret what the code demonstrated. The matching numbers between from-scratch and scikit-learn are the sanity check that validates understanding. The k-NN flip is the headline result: identical inputs, opposite output, driven only by the metric. And the normalization snippet proves L2 and cosine collapse to the same ranking once vectors are unit length.

Reading the results this way — sanity check, headline, equivalence — gives the reader a mental summary to carry forward. Each snippet wasn't an isolated exercise but part of a single argument: metrics genuinely differ, the difference changes model behavior, and normalization is the bridge between the geometric and angular views.

These suggestions turn the post from a demo into a sandbox. Changing the query point to flip k-NN back teaches that the boundary between metrics is itself a movable thing. Adding a huge-scale feature and watching L2 break reproduces the scaling failure from the "Why It Matters" post in code. Using scipy's cdist introduces the right tool for large arrays.

The last two tips preview the mistakes post: normalizing before cosine avoids divide-by-zero on zero vectors, and timing the loop against the vectorized version makes the performance gap visceral. Active experimentation along these lines is what converts passive reading into durable, hands-on understanding.

This mistake addresses a performance trap that beginners fall into constantly: computing pairwise distances with explicit Python for-loops. The result is correct but catastrophically slow once you have more than a few hundred points, because Python-level iteration is orders of magnitude slower than vectorized array math.

The fix is to always push the computation into optimized C code: vectorized NumPy operations, scikit-learn's pairwise functions, or scipy.spatial.distance.cdist for large arrays. On a real dataset the difference between a hand-loop and cdist can be the difference between minutes and milliseconds. Never hand-loop distances at scale — reach for the vectorized tool every time.

This final snippet distills the whole post into a deployable recipe. The two commented steps are the practical rules: first, scale features so no single column dominates an L2 or L1 computation; second, for embeddings or text, L2-normalize the vectors and then either Euclidean or cosine will rank neighbors identically.

The code uses scikit-learn's normalize to make every row unit length, and the assertion confirms it. This is the takeaway a reader can paste into their own pipeline: normalize first, and the choice between cosine and Euclidean stops being a source of inconsistency. It operationalizes the day's central lessons into a few defensive lines of preprocessing.

The CTA closes the code post and points toward the failure modes. Having seen the metrics work correctly and disagree predictably, the reader is now equipped to recognize when they go wrong.

The final post is the field guide to those failures — unscaled features, cosine where magnitude mattered, L2 in high dimensions, and the zero-vector divide-by-zero — each with the one-line fix. It converts the conceptual and practical understanding built so far into a checklist that prevents real bugs.