Unsupervised Learning

This cover poses the central puzzle of the mechanics post: how do you find groups when nobody told you what the groups are? The answer — measure distance — is the thread that runs through everything here. Because there's no label to compute a loss against, unsupervised learning substitutes geometry for supervision.

The promise of the post is decompression: the field looks like a sprawling zoo of algorithms, but once you see that they all reduce to a few ideas (distance, centroids, variance, density), the zoo collapses into a small set of strategies you can actually reason about.

Distance is the most consequential and most overlooked choice in unsupervised learning. Euclidean distance treats each feature as a coordinate and measures the straight-line gap between points — intuitive, and the default in most libraries. Cosine distance measures the angle between vectors instead of their separation, which is what you want when magnitude doesn't matter, as with text or normalized embeddings.

The reason this matters so much is that everything downstream depends on it. Clustering, anomaly detection, and nearest-neighbor search all inherit whatever notion of 'close' you pick. Choose a metric that doesn't match your data — Euclidean on sparse high-dimensional text, say — and you'll find structure that is real only in the wrong geometry.

k-means is the canonical clustering algorithm, and its elegance is that it's just two alternating steps. The assignment step puts every point in the cluster of its nearest centroid. The update step recomputes each centroid as the mean of the points now assigned to it. You alternate until assignments stop changing.

The key theoretical fact is that each step can only decrease (or hold) the total within-cluster distance, so the process is guaranteed to converge. The catch — which post 4 and 5 return to — is that it converges to a local optimum that depends on where the centroids started. That's why you run it multiple times with different initializations and keep the best result, the role of the n_init parameter.

The cycle diagram captures k-means as the loop it really is: place k centroids, assign points to the nearest, move centroids to the mean of their members, and check whether anything shifted. If centroids barely move, you've converged and you stop.

Seeing it as a cycle rather than a formula is the fastest path to intuition. It also makes the failure modes obvious: a bad starting placement can trap the loop in a poor configuration, and choosing the wrong k forces the loop to carve the data into a number of pieces that may not reflect any real grouping.

This code shows the single most important habit in distance-based unsupervised learning: scale before you cluster. StandardScaler transforms every feature to mean 0 and standard deviation 1, so no single column dominates the distance calculation. Then KMeans fits on the scaled data, and you can read out both the per-point labels and the learned centroids.

The n_init=10 and random_state=0 arguments are deliberate. n_init runs the algorithm ten times from different random starts and returns the best, mitigating the local-optimum problem. random_state makes the run reproducible. These two lines encode best practice that beginners routinely skip, with predictably poor results.

Scaling deserves its own slide because forgetting it is the most common way unsupervised analyses go silently wrong. Distance sums contributions from every feature, so a feature ranging in the thousands (annual salary) completely overwhelms one ranging in the tens (age). The clusters you get back are then just a coarse slicing of the dominant feature, dressed up as multivariate structure.

Standardizing puts every feature on a comparable scale so each gets a fair vote in the distance. The blunt rule to remember: if your method uses distance — and almost all of them do — scale first, every time, or your structure is an artifact of your units rather than a property of your data.

PCA is the headline dimensionality-reduction technique, and this slide gives the geometric intuition. PCA rotates the coordinate system to a new set of axes ordered by variance. The first principal component points in the direction where the data spreads out the most; the second points in the direction of most remaining spread that's perpendicular to the first; and so on.

The payoff is compression with minimal information loss. Keep only the top few components and you've captured most of the variance — the 'interesting' part of the data — while discarding low-variance directions that are often just noise. That's why PCA both de-noises and shrinks the data, and why it's the standard first move before plotting high-dimensional datasets.

The vector diagram visualizes what PCA actually produces: new axes, PC1 and PC2, oriented along the directions of greatest spread rather than along the original feature axes. PC1 is longer and points where the data varies most; PC2 is orthogonal and captures the next-most variation.

The crucial caveat, which post 5 hammers on, is that these axes are blends of the original features, not the features themselves. PC1 is not 'age' or 'salary' — it's a weighted combination. Reading them as if they were original variables is a classic misinterpretation. Here the goal is just to see that PCA finds and keeps the directions that carry the most signal.

Not all clustering works like k-means, and this comparison introduces the major alternative. k-means is centroid-based: you specify k, it assumes roughly spherical blobs, and it assigns every point to a cluster. It's fast and simple but struggles with non-round shapes and is thrown off by outliers, which it's forced to assign somewhere.

DBSCAN is density-based: it groups points that are packed closely together and labels sparse points as noise, so it can find arbitrarily shaped clusters and explicitly handle outliers — and it doesn't require you to pick k. The trade is sensitivity to its neighborhood-radius parameter. Knowing both means you can match the algorithm to the geometry of your data instead of forcing everything into k-means.

Evaluation without labels is the final piece of the mechanics, and the silhouette score is the workhorse metric. For each point it compares how close it is to its own cluster versus the nearest other cluster, yielding a value from -1 (badly placed) to +1 (cleanly separated). Averaged over all points, it summarizes how tight and well-separated the clustering is — without ever needing a true label.

The comment about the elbow method points to the complementary approach: plot the within-cluster inertia as k increases and look for the 'elbow' where adding clusters stops helping much. Used together, silhouette and the elbow give you principled, quantitative ways to choose and grade clusterings, replacing the gut-feel guessing that wrecks so many analyses.

This slide consolidates the question that hangs over all clustering: how many clusters should there be? The honest answer is that there's rarely a single true k, so you triangulate. The elbow method finds where added clusters stop paying off. The silhouette score finds the k with the cleanest separation. Domain knowledge asks whether the resulting groups actually mean something. Stability checks whether the same groups reappear when you resample or reseed.

No single one of these is authoritative, which is why you use them together and treat agreement among them as the real signal. Internalizing that k is a modeling decision — not a fact waiting to be discovered — is what separates careful practitioners from people who just trust the default.

The CTA hands off to the hands-on post. Having covered the mechanics conceptually — distance, k-means, scaling, PCA, density, evaluation — the next step is to run the whole thing on real data. The teaser previews the exact arc: load, scale, cluster, reduce, and visualize.

This sets the expectation that post 4 is where theory becomes muscle memory, turning the ideas just introduced into a reproducible workflow the reader can actually execute.