Support Vector Machines

This is the engine-room post. An SVM can sound mystical — 'maximum margin,' 'kernel trick,' 'support vectors' — but underneath it's a tidy optimization problem with one clever twist. Three ideas carry the whole thing: framing the margin as something to maximize under constraints, the soft-margin parameter C that trades violations against width, and the kernel trick that lets a fundamentally linear method draw curves.

The payoff for understanding all this is that the library calls in the next post stop looking like magic. Every argument you pass — C, gamma, the choice of kernel — maps directly to a piece of the machinery developed here. You'll be setting knobs you actually understand rather than guessing.

Step one is turning the geometric idea of a 'wide margin' into something a computer can optimize. The decision boundary is the set of points satisfying w·x + b = 0, where w is a weight vector perpendicular to the boundary and b is an offset. A short calculation shows the margin's full width equals 2 divided by the length of w (written ‖w‖).

That relationship is the whole trick of setup: a wider margin corresponds to a smaller ‖w‖. So 'maximize the margin' becomes the equivalent 'minimize ‖w‖' — and minimizing a length is a clean, well-behaved objective. This is the move that converts a fuzzy geometric goal into a precise optimization problem we can actually solve.

Step two adds the constraints, because minimizing ‖w‖ on its own has a degenerate answer: shrink w toward zero and the 'margin' grows without bound, but the boundary stops separating anything. The constraints anchor the problem to the data. For each training point we require y(w·x + b) ≥ 1, where y is +1 or -1 for the two classes.

This condition says every point must lie on the correct side of its class's margin edge, not merely on the correct side of the boundary. Minimizing ‖w‖ subject to all these constraints yields exactly the maximum-margin separator, with the support vectors being the points where the constraint is tight (equals 1). It's a constrained optimization — a specific, solvable quadratic program.

This pipeline diagram lays the optimization out end to end. The goal is to minimize ‖w‖² (squaring is a mathematical convenience that doesn't change the answer). The constraints are y(w·x + b) ≥ 1 for every point. A solver tackles this as a convex quadratic program — a class of problems with reliable, efficient algorithms and a guaranteed single optimum. The output is the weight vector w, the offset b, and the identity of the support vectors.

Seeing it as a four-box pipeline strips away the mystique. There's no iterative guessing or random initialization here; it's a deterministic optimization with a unique solution. That determinism is the source of the convexity advantage praised in the previous post — same inputs, same model, every time.

Step three is what makes SVMs usable on real, overlapping data. Hard constraints demanding perfect separation usually have no solution, so we introduce slack variables that let individual points violate their margin, and we add a penalty proportional to C times the total violation. The objective becomes: minimize ‖w‖² plus C times the sum of violations.

C is the dial that balances the two terms. A large C makes violations very expensive, so the optimizer accepts a narrow margin to avoid them — fitting the training data tightly and risking overfit. A small C makes violations cheap, so the optimizer happily widens the margin even if some points end up inside it — a smoother, more regularized boundary that risks underfitting. C is the single most consequential knob on an SVM, and the mistakes post is largely about not getting its direction backwards.

This comparison fixes the meaning and, crucially, the direction of C in your memory. On the left, a large C allows few violations, producing a narrow margin that fits the training data tightly and risks overfitting. On the right, a small C tolerates violations, producing a wide, smooth margin that risks underfitting.

The counterintuitive part — and a frequent source of bugs — is that large C means less regularization, not more. People coming from other models often assume the big number is the 'stronger' setting and tune in exactly the wrong direction. Internalize the picture: large C chases the training points (overfit-leaning); small C relaxes toward a wide, general margin (underfit-leaning). The right value sits in between and is found by cross-validation.

Step four is the kernel trick, the SVM's most celebrated idea. The key observation is that, when you write the optimization in its dual form, the data appears only inside dot products between pairs of points — never as individual coordinates. That means if you can compute the dot product of two points in some richer space, you can run the whole SVM in that space without ever constructing the coordinates.

A kernel function does exactly that: it takes two original points and returns the dot product they would have in a (possibly enormous, even infinite-dimensional) feature space. Swap every dot product in the algorithm for a kernel evaluation, and a linear classifier transparently becomes a nonlinear one. No high-dimensional vectors are ever built — that's why it's a 'trick' and not just 'add more features.'

This snippet makes the kernel concrete by implementing the most popular one, the RBF (Gaussian) kernel, in three lines. It takes two points, computes the squared distance between them, and exponentiates the negative of that distance scaled by gamma. The result is a similarity score: 1 when the points coincide, falling smoothly toward 0 as they move apart.

The crucial conceptual point is what this single number represents — it is the dot product of the two points in an infinite-dimensional feature space, computed without ever entering that space. The SVM only ever asks the kernel 'how similar are these two points?' and that one scalar is enough to build a curved boundary. Run the example and you'll see the kernel is just a function of distance: nearby points are similar, distant points are not.

Step five is choosing a kernel, which is really choosing your assumption about the boundary's shape. A linear kernel gives a straight hyperplane — fast, hard to overfit, and the right default when features already outnumber samples (high-dimensional data is often linearly separable). An RBF kernel produces smooth, flexible curves and is the go-to when you don't know the shape; its gamma parameter controls how wiggly the boundary can get. A polynomial kernel gives curved boundaries of a fixed degree, useful when you expect interaction effects.

The practical guidance: start linear on high-dimensional data, reach for RBF when you suspect nonlinearity and have enough samples to tune it, and treat polynomial as a more specialized choice. The kernel is not a free lunch — a more flexible kernel needs more careful tuning to avoid overfitting, which is the subject of the gamma slide and the mistakes post.

This snippet isolates gamma, the RBF kernel's flexibility dial, because misunderstanding it is a leading cause of overfit SVMs. gamma sets how far a single training point's influence reaches. A low gamma means each point influences a broad region, producing a smooth, almost-linear boundary. A high gamma confines each point's influence to a tight bubble around itself, letting the boundary wrap intricately around individual points.

The failure mode is high gamma: the boundary becomes so local that it can enclose each training point in its own little island, achieving perfect training accuracy while generalizing terribly. gamma and C interact — both affect the bias-variance balance — which is exactly why the next post insists on tuning them jointly rather than one at a time.

This stack diagram answers a practical question: what does a trained SVM actually store? Not the whole dataset and not a giant weight matrix — just four things. The support vectors (the boundary-defining points), one dual coefficient (a learned weight) per support vector, the bias term b that offsets the boundary, and the kernel together with its parameters so it knows how to compare new points to the support vectors.

This is why the model is sparse and memory-efficient: prediction reduces to comparing the new point against the stored support vectors using the kernel, weighting by the dual coefficients, adding b, and reading off the sign. Understanding the stored ingredients also explains the cost structure — prediction time grows with the number of support vectors, which is one reason very noisy data (which produces many support vectors) makes SVMs slow.

These bullets are the entire engine in five lines: maximizing the margin is the same as minimizing ‖w‖; the constraints keep every point on the correct side of its margin; C trades violations against margin width and is the master regularization knob; kernels replace dot products to turn straight boundaries into curves; and the trained model stores only the support vectors, not the whole dataset.

If you can recite this list, you understand SVMs more deeply than most people who use them daily. Every item reappears in the next post as a line of real scikit-learn code or a parameter you set — C, gamma, the kernel choice, support_vectors_ — now with meaning attached rather than memorized.

This post traded intuition for mechanism: the margin as a constrained minimization of ‖w‖, the soft-margin penalty C, the kernel trick that exploits dot-product-only structure, the common kernels and their parameters, and the compact set of things a trained model stores. You now know not just what an SVM does but why each piece is there.

The next post cashes all of this in. We'll build the full workflow in scikit-learn — split, scale (mandatory for SVMs), fit linear and RBF kernels, jointly tune C and gamma, and evaluate with cost-aware metrics — and because you understand the engine, the code will read like narration of ideas you already hold.