Support Vector Machines

This is the hands-on post. Theory becomes muscle memory only when you run the code, so everything here is copy-paste runnable on the breast-cancer dataset bundled with scikit-learn. We walk the complete professional workflow for an SVM: import, split, scale (which is mandatory, unlike for trees), fit an RBF kernel, jointly tune C and gamma, and evaluate with metrics that match the problem's costs.

The dataset is a good choice — real medical features, two classes, mild imbalance — so the steps mirror what you'd do on an actual problem rather than a toy. Type it out and change things: raise C and watch the boundary tighten, crank gamma and watch training accuracy soar while test accuracy sags. The feedback is the fastest teacher.

Step zero is the imports, and for an SVM they're more involved than for a tree because scaling and pipelining are not optional. load_breast_cancer supplies the data, train_test_split and GridSearchCV handle the holdout and tuning, StandardScaler does the mandatory feature scaling, SVC is the model, and make_pipeline chains the scaler and model so the scaler is only ever fit on training data.

Notice what's present here that was absent in tree-based workflows: StandardScaler and a pipeline. That difference is the single most important practical fact about using SVMs, and putting the import right at the top is a deliberate reminder. An SVM script without scaling is almost always a buggy SVM script.

Loading and splitting is the foundation, and the details matter. We pull features X and target y, then split off 20% as a test set. Two arguments do important work: stratify=y preserves the class proportions in both splits, which matters because the dataset is mildly imbalanced toward the benign class, and random_state=42 makes the split reproducible so your numbers match across runs.

The printed shape, (455, 30), confirms 455 training rows and 30 features. Always eyeball shapes after a split — a surprising number here is the fastest way to catch a data-loading bug before it silently corrupts everything downstream. This step is identical to any other classifier's; the SVM-specific care begins at scaling.

This slide states the rule that separates working SVMs from broken ones: you must scale your features, and it is not optional. Because an SVM measures distances between points to find the margin, a feature that ranges from 0 to 1000 will completely dominate one that ranges from 0 to 1 — the larger feature alone will shape the boundary while the smaller ones become invisible. The model doesn't error; it just quietly performs badly.

The fix is to standardize features to comparable scales, and to do it inside a Pipeline. The pipeline ensures the scaler's mean and standard deviation are computed from the training fold only and then applied to the test fold, rather than being fit on the whole dataset — which would leak test information into training. This is the discipline that the mistakes post elevates to the number-one SVM rule.

Fitting an RBF SVM is anticlimactic once the pipeline is in place: make_pipeline chains a StandardScaler with an SVC using the RBF kernel, and a single fit call scales the training data and trains the model in one shot. We start with C=1.0 and gamma='scale', sensible defaults, and test accuracy lands around 0.97.

The elegance is that the pipeline behaves like a single estimator — predict and score automatically apply the fitted scaler before the SVM sees the data, so you can never accidentally feed unscaled data at test time. gamma='scale' sets gamma based on the number of features and their variance, which is a far better starting point than the old default and usually close to optimal before tuning.

Tuning is where you invest effort after the basics are solid, and for an SVM the two knobs that matter are C and gamma. This grid search sweeps four values of each and evaluates every combination with 5-fold cross-validation. Crucially, the parameter names are prefixed with svc__ because we're tuning the SVC step inside the pipeline — that's how you address a component's parameters through a pipeline.

Note scoring='recall': we're optimizing for recall, not accuracy, because of the medical context explained on a later slide. GridSearchCV refits the best configuration on the full training set automatically, so search.best_estimator_ is ready to use. Because C and gamma interact, sweeping them jointly on a grid — rather than one at a time — is what finds the genuinely best pair, a point the mistakes post hammers home.

This slide explains why the two knobs must be tuned together. C controls the tradeoff between a wide margin and few training errors; gamma controls how far each point's influence reaches and thus how wiggly the RBF boundary can be. They both shift the bias-variance balance, and they do so interactively: the best C for a smooth, low-gamma boundary is different from the best C for a tight, high-gamma one.

That interaction is why tuning them one at a time finds a false optimum — you'd be optimizing C against whatever arbitrary gamma you happened to fix, and vice versa. A joint grid (or random search over the 2D space), evaluated by cross-validation, explores the genuine landscape. And the iron rule of all tuning applies: the test set stays untouched until the very end, or your reported numbers are fiction.

This snippet runs the real evaluation on the held-out test set. We pull the best estimator from the search, predict on the untouched test data, and print both a confusion matrix and a full classification report — per-class precision, recall, and F1. The confusion matrix shows exactly how the four outcome types break down; the report turns those counts into the rates you actually reason about.

The reason we look past the single accuracy number is that accuracy can hide a model that does well on the easy majority class while failing the minority class that matters. The per-class view exposes that immediately. On this dataset a good SVM achieves both high accuracy and high recall, but you only know that — and can only defend it — by looking at the breakdown rather than one summary statistic.

This slide states the principle behind scoring the tuning on recall. In cancer screening, the two errors are not equal: a false negative means telling a patient with a malignant tumor that they are fine, which can be fatal, whereas a false positive merely triggers an additional test. The cost asymmetry is enormous, and your evaluation metric must reflect it.

That's why the grid search optimized recall on the malignant class rather than overall accuracy — recall directly measures the fraction of true malignant cases the model catches. The general lesson transcends this dataset: identify which error is most expensive in your real problem, choose the metric that penalizes that error, and optimize for it. Defaulting to accuracy on any cost-asymmetric or imbalanced problem is a quiet but serious mistake.

This snippet reaches into the fitted pipeline to inspect the support vectors, closing the loop back to the concept post. We pull the SVC step out by name, then read support_vectors_ — the actual points that define the boundary — and n_support_, which counts support vectors per class. You'll typically find that only a fraction of the 455 training rows are support vectors.

This is the sparsity property made tangible: the entire trained model is defined by these few boundary-hugging points, and everything else could be discarded. It also offers a diagnostic. If almost every training point becomes a support vector, your model is likely overfitting (often gamma is too high or C too large) or the classes overlap heavily — a signal to revisit the hyperparameters rather than trust the fit.

This pipeline diagram is the whole post in four stages: split (stratified), scale (StandardScaler), fit the SVC with an RBF kernel, then tune and evaluate using a grid search scored on recall. The scaling stage is the one that distinguishes this from a tree-based workflow, and the diagram puts it front and center precisely because forgetting it is the most common SVM error.

Internalizing this shape pays off broadly. The split, the cost-aware evaluation, and the deliberate tuning are universal good practice; only the model box and the mandatory scaling are SVM-specific. Wrapping scaling and the model in a single pipeline object is what makes the whole thing both leak-free and easy to deploy as one unit.

These pitfalls are the workflow's sharp edges, gathered in one place. Always scale features, and do it inside a Pipeline so scaling is fit on training data only. Tune C and gamma together rather than separately, because they interact. Score on recall or AUC rather than accuracy when errors are asymmetric or classes imbalanced. Only set probability=True if you genuinely need probabilities, since it adds a slow internal calibration step. And watch runtime: kernel SVMs scale poorly with the number of rows, so a workflow that's instant on hundreds of rows can stall on hundreds of thousands.

Every one of these turns a 'working' notebook into a model that disappoints or misleads. Internalize them now, because the next post is entirely about mistakes — and several of these reappear there with deeper treatment and concrete fixes.

You now have a support vector machine working end to end and, thanks to the earlier posts, you understand every step rather than just running it. Split with stratification, scale inside a pipeline (the non-negotiable SVM step), fit an RBF kernel, jointly tune the two knobs that matter, evaluate with metrics that match your costs, and inspect the support vectors that define the model.

The final post is the field guide to what goes wrong. We'll catalog the six mistakes that most often quietly wreck an SVM — forgetting to scale, using it on huge data, getting C backwards, overfitting with gamma, expecting free probabilities, and tuning the two knobs in isolation — and the specific fix for each, so your powerful classifier stays powerful when it actually matters.