Overfitting & Regularization

This post opens the engine. Having defined overfitting and motivated why it matters, we now get concrete about the machinery: what regularization actually does to a model under the hood. The unifying insight that ties the whole post together is that most regularization works by changing the objective function — the thing the optimizer is trying to minimize.

Once you see that L1, L2, and friends are just different penalty terms bolted onto the loss, the zoo of techniques collapses into a single, understandable idea. The differences between methods become differences in what shape of penalty you choose and what that shape does to the resulting weights.

The core mechanism is adding a penalty term to the loss. Without regularization, training minimizes only the data loss — some measure of how wrong the predictions are. The optimizer is free to set the weights to absolutely anything that reduces that error, including extreme values that fit noise.

Regularization adds a second term that grows as the weights grow. Now the optimizer faces a tradeoff: it can reduce data loss by fitting the training data more closely, but doing so via large weights increases the penalty. The optimum becomes a balance between fitting the data and keeping the weights small. That balance is what stops the model from chasing noise, because fitting noise typically requires large, specific weights that the penalty makes expensive.

This code slide shows the mechanism literally, in two lines. The plain objective is just the data loss. The regularized objective adds lambda times the sum of squared weights. That single added term is the entire difference between an unregularized and an L2-regularized model.

The lambda multiplier is the control knob: at lambda zero, the penalty vanishes and you're back to the plain objective; as lambda grows, the optimizer cares more about small weights and less about fitting every training point. Seeing the math this explicitly demystifies regularization — it's not a separate algorithm, it's a small addition to the quantity you were already minimizing. Everything else in this post is a variation on which penalty you add.

L2 regularization, also called Ridge, adds the sum of squared weights to the loss. The squaring matters: it penalizes large weights disproportionately hard, so the optimizer is strongly discouraged from letting any single weight grow extreme. The effect is to shrink all weights smoothly toward zero without forcing any of them to be exactly zero.

The practical consequence is a smoother model that distributes importance across many features rather than betting heavily on a few. L2 is the workhorse default in most settings — it handles correlated features gracefully and almost always improves generalization. When someone says 'add some regularization' without specifying, they usually mean L2.

L1 regularization, also called Lasso, adds the sum of absolute weights instead of squared weights. This seemingly small change has a dramatic geometric consequence: the optimization tends to push many weights all the way to exactly zero rather than just making them small. A weight of zero means the corresponding feature is effectively removed from the model.

This gives you automatic feature selection. L1 is the right choice when you suspect that most of your input features are irrelevant and you want the model to identify the few that matter — producing a sparse, interpretable model. The tradeoff is that L1 can behave erratically with groups of correlated features, arbitrarily picking one and zeroing the rest, which is one reason L2 or a blend (Elastic Net) is often preferred.

This comparison crystallizes the L1-versus-L2 decision into a side-by-side you can use when choosing. On the left, L1 uses an absolute-value penalty, drives weights to exactly zero, produces a sparse model with fewer active features, and gives you built-in feature selection. On the right, L2 uses a squared penalty, shrinks weights smoothly without zeroing them, keeps all features in play, and handles correlated inputs well.

The choice follows from your goal. If you want interpretability and believe most features are noise, lean L1. If you want robust general-purpose shrinkage and have features that are correlated or all somewhat useful, lean L2. When you're unsure, Elastic Net combines both penalties and is a reasonable default that captures the strengths of each.

Dropout is the dominant regularization technique for neural networks, and its mechanism is delightfully different from weight penalties. During each training step, dropout randomly switches off a fraction of neurons — say half of them. The forward and backward pass run as if those neurons don't exist, and a different random subset is dropped on the next step.

Because the network can never count on any specific neuron being present, it's forced to learn redundant, distributed representations rather than fragile dependencies on individual units. The standard intuition is that dropout approximates training an enormous ensemble of smaller sub-networks and averaging their predictions — and ensembles are famously good at reducing variance, which is exactly the overfitting problem.

This PyTorch snippet shows how trivially dropout slots into a real network — a single nn.Dropout layer with a probability p of dropping each activation. Here p=0.5 means half the activations are zeroed on each training step, the most common default for fully connected layers.

The critical detail, called out in the comment, is that dropout behaves differently in training versus inference. During model.train() it actively drops neurons; during model.eval() it's turned off and all neurons are used, with activations scaled appropriately so the expected output matches. Forgetting to call model.eval() at inference time is a classic bug that makes predictions noisy and non-deterministic — so this train/eval distinction is worth burning into memory.

Early stopping is the simplest regularizer of all and requires no change to the model — just to when you stop training. As training proceeds, you monitor the loss on a held-out validation set. Typically it falls steadily, reaches a minimum, and then begins to rise as the model starts fitting noise in the training data. That upturn is overfitting happening in real time.

Early stopping halts training at the validation-loss minimum and restores the best weights seen. It regularizes by limiting how much the model gets to adapt to the training set — fewer effective updates means less opportunity to memorize. It's cheap, broadly applicable, and combines well with other techniques, which is why it's nearly universal in deep learning training loops.

This bars diagram visualizes the lambda knob and why it's a dial rather than a switch. At lambda equals zero, there's no penalty, the model overfits, and test error is high. At a tuned lambda, the penalty is calibrated just right, the model balances fitting and simplicity, and test error hits its minimum. At a huge lambda, the penalty overwhelms the data loss, the model underfits, and test error climbs again.

The U-shape is the same one from the bias-variance tradeoff, now controlled by a single hyperparameter. The takeaway is operational: there's an optimal regularization strength for every problem, and your job is to find it empirically — not to assume more regularization is always better. That search is what cross-validation in the next post automates.

This cheat-sheet compresses the post's mechanisms into a reference you can act on. L2 gives smooth shrinkage and keeps all features — the sensible default. L1 produces sparsity and deletes features — choose it for selection and interpretability. Dropout buys robustness through randomness — the go-to for neural networks. Early stopping simply prevents training past the validation-loss dip — cheap and universally applicable.

The final, non-negotiable item ties back to evaluation discipline: every one of these has a strength parameter — lambda, the dropout probability p — and you tune it on validation data or via cross-validation, never on the test set. The mechanism only helps if you set its strength honestly. Get that right and you'll pick the appropriate tool deliberately rather than defaulting blindly.

This closes the mechanics post. You now understand regularization not as a black box but as a precise modification of what the model optimizes — a penalty for complexity in the loss, neurons randomly silenced, training stopped at the right moment. You can articulate why each technique curbs overfitting and when to prefer one over another.

The next post turns all of this into running code. We'll take a model, overfit it on purpose, then apply L2 and early stopping step by step, measuring the train-test gap closing in real numbers. Seeing the mechanism produce concrete results is what cements the understanding and gives you a workflow you can reuse on every project.