Overfitting & Regularization

This is the hands-on post, and its goal is a reusable workflow rather than a specific result. We'll deliberately build an overfit model, watch the train-test gap blow open, then apply regularization techniques one at a time and measure the gap closing. The dataset is synthetic and unimportant; what matters is the loop — fit, measure both scores, regularize, re-measure — that you'll run on every real problem you encounter.

Everything here is runnable. The numbers in the comments are representative of what you'd actually see, but the real lesson is the shape of the workflow and the habits it builds: always measure two scores, always tune on validation, always report test once.

Step zero sets up the experiment. We use make_classification to generate 400 samples with 40 features, but only 5 of them are actually informative — the other 35 are noise. This is a deliberately overfitting-prone setup: a high-dimensional feature space, most of it irrelevant, with a modest number of samples. It's exactly the small-data, high-complexity danger zone from Post 2.

The train_test_split holds out 30% of the data as a test set we won't touch until the end. Fixing random_state makes the whole experiment reproducible, so you get the same split and the same numbers every run — essential when you're comparing models and need the comparison to be apples-to-apples.

Step one overfits on purpose to establish the baseline failure. We use logistic regression with C set to one million. The crucial, frequently-misunderstood detail is that in scikit-learn, C is the inverse of regularization strength — so a huge C means almost no regularization, giving the model nearly unconstrained freedom.

With 40 features (35 of them noise) and only 280 training rows, that unconstrained model happily fits the noise, achieving near-perfect training accuracy. But on the held-out test set, performance is much lower — the wide gap that signals overfitting. This deliberate failure is the control case; every regularized model that follows is measured against it. Inducing the problem on purpose is the fastest way to learn to recognize and fix it.

Step two applies L2 regularization and reveals the payoff. We drop C to 0.1, which (since C is inverse strength) means a strong L2 penalty. The penalty discourages the large weights the model was using to fit noise, forcing it toward a simpler solution that relies on the genuine signal.

The results tell the whole story of regularization in two numbers. Training accuracy drops — from near-perfect to around 91% — because the model is no longer allowed to memorize. But test accuracy goes up, from roughly 78% to 87%. That is the trade at the heart of this entire day: you sacrifice a little in-sample performance and gain substantially more out-of-sample performance. A lower training score with a higher test score is a win, not a regression.

This bars diagram makes the before-and-after visceral by putting all four numbers side by side. The overfit model: 100% on train, 78% on test — a 22-point gap screaming overfitting. The L2 model: 91% on train, 87% on test — a gap of just 4 points, and a higher test score than before.

The visual drives home the counterintuitive lesson that the bar that went down (training) is good news and the bar that went up (test) is the one that matters. A beginner optimizing for the training bar would have rejected the regularized model; an engineer who understands generalization recognizes it as clearly superior. Reading these four bars correctly is the core skill this whole post is teaching.

Step three replaces guesswork with cross-validation. Instead of hand-picking C, LogisticRegressionCV tries 20 candidate values spread logarithmically from 0.001 to 1000, evaluating each via 5-fold cross-validation on the training data and keeping whichever generalizes best across the folds.

This is the defensible, standard way to set a regularization strength. Cross-validation splits the training data into folds, trains on some and validates on the held-out fold, rotating through all combinations — so each candidate C is judged on data it didn't train on. The chosen C isn't a guess you'd have to justify; it's the value the data itself selected. Note that the test set still plays no role here — it's reserved for the single final measurement after C is locked in.

This slide steps back to name what cross-validation accomplished, because the concept is more important than the one function call. We had a hyperparameter — the regularization strength — that we couldn't set from the training fit alone, because the best value for generalization isn't visible in training error. Cross-validation solved this by repeatedly holding out folds and scoring each candidate strength on data it hadn't seen.

The result is that the data, not your intuition, chose lambda. This matters for two reasons: it usually finds a better value than manual tuning, and it's reproducible and defensible to reviewers. Whenever you have a knob whose ideal setting depends on generalization rather than training fit, cross-validation is the principled way to turn it.

Step four implements early stopping for a neural network, the regularizer that needs no model change at all. The loop tracks validation loss each epoch. Whenever validation loss improves, we save the current weights and reset a patience counter. When validation loss fails to improve for 'patience' consecutive epochs — here five — we break out of the loop, having captured the best weights before overfitting set in.

Two details make this production-grade. First, we save weights to disk at each improvement (torch.save), so we keep the genuinely best model, not whatever happened to be in memory when we stopped. Second, the patience parameter prevents stopping on a single noisy uptick — we only quit after sustained failure to improve. This pattern is nearly identical across every deep learning framework and is worth memorizing.

This flow diagram captures the reusable workflow that is the real deliverable of the post. First, split the data into train, validation, and test. Second, measure the train-test gap to detect overfitting. Third, apply regularization — L2, dropout, or early stopping as appropriate. Fourth, tune the regularization strength on validation data or via cross-validation. Finally, report the test score exactly once.

This loop is dataset-agnostic and model-agnostic. Whether you're working with logistic regression, gradient-boosted trees, or a deep network, the same five steps apply. Internalizing this flow is more valuable than memorizing any single technique, because it's the disciplined process that reliably produces models that generalize — and that survive contact with real users.

These gotchas are the practical traps that silently corrupt this exact workflow, so they're worth flagging explicitly. The most common surprise is that in scikit-learn, C is inverse regularization strength — a small C means strong regularization, which trips up nearly everyone the first time. Second, L1 and L2 penalize weights by magnitude, so you must scale features first or the penalty is applied unfairly across features with different units.

Third, tune the regularization strength on validation and judge final performance on test — never tune on test. Fourth, during early stopping, save and restore the best weights rather than using the final ones. Fifth, always compare train and test together at every step; a single number in isolation hides the very gap you're trying to manage. Each of these is the difference between a clean experiment and a misleading one.

This closes the code post. You now have a complete, runnable workflow: induce overfitting, diagnose it via the train-test gap, apply L2 and early stopping, tune the strength with cross-validation, and report an honest test score. More importantly, you have the habits — two scores always, tune on validation, test once — that make the workflow trustworthy.

The final post is the field guide to the mistakes that quietly break this process even when you know the techniques. Tuning on the test set, forgetting to scale, data leakage, and over-regularizing all make an overfit model look healthy. Knowing the traps is what keeps the workflow honest when it counts.