Bias-Variance Tradeoff

This is the code-heavy post, where the tradeoff becomes numbers you can reproduce. The cover sets the expectation that we'll build a complete, runnable experiment in strict order — generate data, estimate the two error terms, sweep complexity, and pick the sweet spot — using nothing beyond NumPy and scikit-learn.

The value of doing this in code rather than describing it is that seeing bias fall while variance climbs, as an actual column of numbers in your terminal, makes the tradeoff impossible to forget. Each block is written to stand alone so you can paste and run it as you read.

We start with imports, kept minimal and explicit. The experiment needs only NumPy for the resampling and arithmetic, plus a few scikit-learn pieces: a polynomial-feature transformer, linear regression, a pipeline to chain them, and validation_curve for the confirmation step later.

Keeping the imports in their own block matters for a copy-and-run tutorial — readers can verify their environment has everything before moving on, and the comment about installing scikit-learn and NumPy heads off the most common first error. Everything downstream assumes exactly these names are in scope.

This block builds a controlled experiment where we know the ground truth, which is what makes measuring bias possible at all. The true relationship is a sine wave; we sample inputs uniformly, evaluate the true function, and add Gaussian noise to create realistic, messy training targets.

The crucial detail is that the test set uses the clean true function with no noise, so when we later compare average predictions to y_test we're comparing against truth, not against noisy labels. Because we control the data-generating process, we can compute genuine bias — the gap from the real pattern — which is impossible with real-world data where the truth is unknown.

Here is the reusable estimator, and it's the engine of the whole experiment. It takes a model-building function and runs many bootstrap rounds: each round resamples the training data with replacement, fits a fresh model, and records its predictions on the fixed test set.

After the loop, the per-point average prediction compared to the truth gives bias squared, and the average per-point variance across rounds gives variance. This directly mirrors the theoretical decomposition: bootstrapping stands in for 'many possible training sets,' the deviation of the average from truth is bias, and the spread is variance. Writing it as a function lets us reuse it across every complexity setting in the next block.

This block runs the actual sweep, which is where the tradeoff reveals itself. The poly helper returns a model-builder for a given polynomial degree, and we feed degrees 1 through 15 into the bias-variance estimator, printing bias squared, variance, and their total for each.

The deliberate design choice is to print the total alongside the two components. That total is what you actually want to minimize, and watching it dip and then rise across degrees is the tradeoff happening in front of you. Low-degree models will show high bias and low variance; high-degree models the reverse; somewhere in the middle the total bottoms out.

The trace shows representative output, and the numbers tell the whole story at a glance. At degree 1 bias squared is large and variance is negligible — a near-straight line underfitting the sine wave. As degree climbs, bias squared collapses toward zero while variance grows, slowly at first and then explosively: by degree 15 variance dominates and the total error is the worst of any setting.

The total column makes the sweet spot obvious — degree 3 has the smallest sum, beating both the underfit and overfit extremes. This single block of output is the empirical proof of everything the concept and mechanics posts claimed: bias and variance really do trade off, and the best model minimizes their sum, not either one alone.

This slide pauses the code to interpret the numbers, because the interpretation is the lesson. Degree 1 is the classic underfit: high bias, trivial variance. Degree 15 is the classic overfit: bias driven to near-zero but variance exploding as the wiggly polynomial chases every noisy point. Degree 3 minimizes the total.

The point worth dwelling on is that you didn't reason your way to this — you measured it. You ran an experiment and read the answer off a column of numbers. That's the shift this post is trying to instill: bias-variance is something you can empirically verify on any model, not just argue about in the abstract.

Here we cross-check the resampling result against scikit-learn's built-in validation_curve, which is the tool practitioners actually reach for. It sweeps the polynomial degree inside the pipeline across five cross-validation folds and returns train and validation scores for each degree.

We negate the scores (scikit-learn reports negative MSE so that 'higher is better' holds) and pick the degree with the lowest validation MSE — the bottom of the U. The fact that this should land near the degree our bootstrap estimator flagged is the reassuring confirmation: two independent methods, the hand-rolled bias-variance estimate and the standard CV curve, point at the same sweet spot.

The bar chart visualizes the total error across the sweep so the U-shape is unmistakable. Degree 1 is a tall bar (underfit), degree 3 a short one (the sweet spot), degree 9 taller again (overfitting setting in), and degree 15 towering (variance run wild).

This is the same information as the printed totals, rendered as a shape. The reason to include both is that numbers convince the analytical reader while the picture convinces the visual one — and the U-shape is the single most recognizable signature in model selection. Once you can see it, you know which direction to move on the complexity axis.

The final code block locks in the winning model: we rebuild the pipeline at the best degree found by cross-validation, fit it on all the data, and report its test MSE against the clean true function. Training at the bottom of the U is the entire goal of the exercise — it's the complexity that minimizes expected error on unseen data.

The comment reinforces the takeaway: this model wasn't chosen because it scored best on training data (degree 15 would win that), but because it sits at the balance point of the tradeoff. That's the discipline the whole series is building toward — selecting models by generalization, not by in-sample fit.

The tips slide inventories what the reader just built: a working bias-variance estimator, empirical proof that the two terms trade off, a complexity sweep that reports totals, a cross-validated validation curve that confirms the result, and the final sweet-spot model locked in.

This is a genuinely portable toolkit. The bias_variance function works on any scikit-learn regressor, the sweep pattern generalizes to any complexity parameter, and the validation_curve confirmation is one call away. A reader who runs all five blocks has a reusable method for diagnosing the tradeoff on their own models, not just a story about someone else's.

We close by pointing to the final post on common mistakes — the traps that quietly push you to the wrong end of the curve even when you understand the theory. The teaser frames them as subtle shoves rather than obvious blunders, which is exactly why they catch experienced people.

Having measured the tradeoff with your own code, you're now equipped to recognize when your own habits are sabotaging it. That's the natural final lesson: knowing the theory and the tooling is necessary but not sufficient, because the everyday workflow is full of ways to undermine both.