Supervised Learning

This closing post is the failure-mode checklist that turns a beginner into someone you can trust with a model in production. Supervised learning is unusually easy to run and unusually easy to fool yourself with: scikit-learn will happily hand you a confident, high-scoring model that quietly fails on real data.

The mistakes collected here share a sinister property — none of them throw an error. The code runs, the metric looks great, and only later, in production or in a careful review, does the problem surface. Learning to anticipate them is what separates a working deployment from an embarrassing one.

Data leakage is the most dangerous mistake because it produces the most convincing fake success. Leakage happens when a feature secretly contains information about the target that wouldn't be available at prediction time — often something computed from or after the outcome itself.

The tell is suspiciously high accuracy, the kind that makes you proud rather than suspicious. When a model scores far better than the problem should allow, the right reaction is to hunt for a leaking feature, not to celebrate. In production that feature won't exist or won't carry the same information, and the model collapses.

This code slide makes leakage concrete with a lending example. A feature like days_until_default is computed from the very event you're trying to predict; it essentially hands the model the answer, producing near-perfect training scores that mean nothing. The fix is to restrict features to information genuinely available at the moment of decision.

The discipline here is temporal: for every candidate feature, ask whether its value would actually be known at prediction time in production. If it depends on the outcome or on the future, it leaks. This single question catches a large fraction of real-world leakage before it ever reaches a model.

Evaluating on training data is the most basic error and still a common one, especially in quick experiments. A flexible model can memorize its training rows, so scoring it on those same rows measures memorization, not the generalization you actually care about. The number will be near-perfect and tell you nothing useful.

The remedy is non-negotiable: always report metrics on a held-out test set the model never trained on. This is why the code post split the data before fitting. Any reported performance figure should implicitly answer the question 'on data the model has never seen?' with yes.

Accuracy on imbalanced data is a trap that fools even experienced people because the misleading number looks so good. When one class dominates — 99% legitimate transactions, 1% fraud — a model that always predicts the majority class scores 99% accuracy while being completely useless, catching none of the fraud you built it to catch.

The fix is to choose metrics that actually see the rare, important class: precision, recall, F1, and ROC-AUC. These reveal whether the model finds the minority cases and at what cost in false alarms. On any skewed problem, leading with accuracy is a sign you haven't thought hard enough about what success means.

The bar diagram dramatizes the imbalanced-accuracy trap with the always-predict-legit baseline on a 99/1 split. Its accuracy bar stands near the top at 99%, looking like a triumph, while the recall and precision bars for the fraud class sit flat at zero — the model catches no fraud and makes no useful positive predictions.

The juxtaposition is the whole lesson: a single headline metric can hide total failure on the thing that matters. Whenever classes are imbalanced, look past accuracy to the per-class metrics, because those are where a useless model gets exposed.

Overfitting is the gap between memorizing and learning. A model that's too complex for the data fits the training set's noise as if it were signal, scoring excellently on train and noticeably worse on test. The size of that train-minus-test gap is your primary diagnostic.

The standard remedies all reduce effective complexity or add information: collect more data, choose a simpler model, apply regularization, or stop training early when validation loss turns up. The key habit is to never judge a model by its training score alone — always hold the train and test numbers side by side and watch the gap.

This snippet operationalizes overfitting detection in two lines: score the model on the training set and on the test set, then compare. A small gap, like 0.97 versus 0.95, is healthy. A large gap, like 0.99 versus 0.74, is a clear overfitting signal — the model learned the training data far better than the underlying pattern.

Making this comparison a reflex is one of the cheapest, highest-value habits in applied ML. It takes seconds, requires no extra data, and immediately tells you whether your impressive training score reflects real learning or just memorization that won't survive contact with new data.

Fitting preprocessing before the split is a subtle leakage variant that slips past many practitioners. If you fit a scaler or imputer on the entire dataset before splitting, statistics from the test rows — their mean, their variance — leak into the transformation applied to training, contaminating the honesty of your test estimate.

The correct procedure is to fit any data-dependent preprocessing on the training set only, then apply the fitted transform to the test set. This keeps the test set genuinely unseen. It's easy to get wrong by hand, which is exactly why pipelines exist.

This code slide shows the clean fix: wrap preprocessing and the model in a scikit-learn Pipeline. When you fit the pipeline, the scaler is fit on the training data only, and during cross-validation it's correctly re-fit on each training fold, never on the validation fold. Leakage from preprocessing becomes structurally impossible.

Beyond preventing leakage, pipelines make your workflow cleaner and easier to deploy: the same object that learned the scaling parameters applies them at prediction time, so train and serving preprocessing can't drift apart. Reaching for a Pipeline by default is a mark of disciplined practice.

The final mistake is the most consequential and the least technical: trusting your labels uncritically. A supervised model is a mirror of its training labels, and if those labels encode historical human bias — in hiring, lending, policing, or anything socially loaded — the model will faithfully reproduce and often amplify that bias while wearing a veneer of mathematical objectivity.

Stale, mislabeled, or skewed labels poison everything downstream silently, because the code runs fine and the metrics may even look strong. Auditing where labels came from, who produced them, and what assumptions they bake in is essential before you trust a model's output — a theme that ties directly back to the ethics and bias material in the broader series.

The checklist consolidates all six mistakes into a pre-flight routine you can run before shipping any supervised model: report metrics on a true holdout, hunt for leaking features, match your metric to the class balance, watch the train/test gap, fit preprocessing inside a pipeline, and audit the provenance of your labels.

None of these steps is hard, and together they catch the large majority of ways a supervised model fools its own builder. Running through them deliberately is the difference between a model that demos well and one that survives contact with the real world.

The cover frames this as the cautionary capstone of the day — the post that arms you against the quiet ways supervised learning misleads its practitioners. After learning what it is, why it matters, how it works, and how to build it, you finally learn how it breaks.

The teaser points beyond this day entirely, toward unsupervised learning on Day 29. Having spent five posts inside the world of labeled answer keys, the natural next step is to ask what you can do when there are no labels at all — finding structure in raw data with no supervision to guide you.