Train / Validation / Test Splits

This final post is the field guide to how splits get corrupted, and it may be the most practically valuable of the five, because broken splits are behind a huge share of ML results that look brilliant and fail in reality. The recurring theme is that these failures are quiet by nature: the code runs without error, the metrics look great, and nothing warns you that the number is a lie.

None of these traps are exotic. They appear constantly in real projects and in published results, they are diagnosable once you know the symptoms, and most are prevented by a few simple habits. Treat this post as a checklist you run whenever a score looks too good — or whenever a model that tested well disappoints in production.

Preprocessing leakage is the most common single mistake in applied ML. It happens when you fit a transform that learns parameters — a scaler, an imputer, a feature selector, an encoder — on the entire dataset before splitting. The transform absorbs statistics from what will become your test set, and through it those statistics flow into the model, making the test score optimistic.

The symptom is a suspiciously high held-out score that collapses in production, where the test-set statistics were of course never available. The fix is the discipline from posts 3 and 4: split first, then fit every preprocessing step on the training portion alone and merely apply it to validation and test. Wrapping the whole sequence in a scikit-learn Pipeline enforces this automatically and is the most reliable defense.

This snippet contrasts the leak and the fix side by side so the mistake is unmistakable. In the wrong version, StandardScaler().fit_transform runs on the full X before the split, so by the time you separate train and test, the test rows have already influenced the scaler's mean and standard deviation — leakage is baked in. The right version splits first, fits the scaler on the training set only, and then transforms both sets with those train-derived parameters.

The ordering is the entire lesson: split, then fit. It is a tiny change in line order with an outsized effect on the honesty of your results. Because the leaky version produces better-looking numbers, it is genuinely tempting and easy to write by accident — which is exactly why making the correct order a habit, or enforcing it with a Pipeline, matters so much.

Tuning on the test set is leakage committed through your own decisions rather than your code. You check the test score, adjust a hyperparameter or swap a feature, check again, and repeat. Each cycle bleeds a little information from the test set into your choices, and after enough iterations you have effectively fit your decisions to that specific test set. The final number is no longer an honest estimate of unseen performance.

The defense is the strict separation from post 1: make every tuning decision against the validation set, which is allowed to get a bit optimistic, and reserve the test set for a single look at the very end. The mantra is simple — test is for one look, after everything is frozen, full stop. If you have looked at the test score and then changed the model, you need a fresh, untouched test set.

This comparison lays the honest and corrupted workflows next to each other so the difference is stark. The honest workflow tunes and compares exclusively on validation, freezes the chosen model, and touches the test set exactly once. The corrupted workflow tunes against test, re-checks it repeatedly, and selects whatever wins on test — at which point the final score is pure fantasy, reflecting how well you overfit the test set rather than how the model will perform.

The insidious part is that the corrupted workflow feels productive. Each peek-and-tweak seems like rigorous optimization, and the numbers improve. But you are optimizing the wrong thing. Keeping this two-column picture in mind helps you catch yourself the moment a tuning loop starts reaching for the test score instead of validation.

Duplicate and near-duplicate rows are a quiet form of leakage that survives even a careful random split. If the same record, or a near-identical one, lands in both train and test, the model can succeed by recognizing it rather than by generalizing — the equivalent of having seen the exam questions in advance. Your test score reflects memorization of those shared rows, not real skill.

Two habits prevent this. First, deduplicate the dataset before splitting so identical records cannot straddle the divide. Second, when the natural unit is an entity with many rows — a user, a patient, a document with many sentences — use a grouped split so all rows from one entity stay on the same side. The next slide shows exactly how to do that in scikit-learn.

This snippet implements a group-aware split with GroupShuffleSplit. You pass a groups array — here user_ids — and the splitter guarantees that every row sharing a group ends up entirely in train or entirely in test, never split across both. Calling next on the splitter's generator yields the train and test index arrays for one such grouped partition.

The effect is that no user appears on both sides of the divide, so the model cannot cheat by recognizing an identity it has already seen. The resulting test score is usually lower than a naive random split would give — but it is the honest score, the one that reflects performance on genuinely new entities. Whenever your rows are not independent, this kind of grouped splitting is essential to a trustworthy estimate.

Random splits on time-series data are a guaranteed way to fool yourself. Shuffling time-ordered records and splitting randomly means the model can train on examples that occurred after the ones it is tested on — it literally learns from the future to predict the past. That is impossible at deployment, so the impressive offline score has no bearing on live performance, and the model fails the moment it faces real, forward-flowing time.

The correct approach is always to split by time: train on the earliest periods, validate on a later slice, and test on the latest, strictly respecting chronological order. This mirrors how the model will actually be used — modeling the past to predict the future — and yields a score that genuinely reflects deployment conditions. For any temporal problem, chronological splitting is non-negotiable.

This timeline visualizes a correct chronological split for time-series data. The earliest months — January through August — form the training set, where the model learns from the past. The next slice, September and October, is the validation set used for tuning. The latest months, November and December, are the test set, on which the model is judged as if predicting the genuine future.

The key property is that time only ever flows forward across the sets: training precedes validation, which precedes test. No fold ever trains on data that comes after its evaluation period. This forward-chaining structure is what makes the resulting estimate trustworthy for a deployed forecasting or sequential system, in sharp contrast to the fantasy scores a shuffled split would produce.

Ignoring class imbalance during splitting produces unstable, untrustworthy metrics. On a dataset where the positive class is rare — say 2% — a plain random split can, purely by chance, leave the test set with almost no positive examples. Any metric computed on a handful of positives swings wildly from run to run and tells you almost nothing reliable about real performance.

The fix is to stratify every classification split so each set preserves the true class ratio, as covered in posts 3 and 4. Without stratification your evaluation is hostage to the luck of the draw; with it, train, validation, and test all mirror the real distribution and your metrics become stable and comparable. On imbalanced problems this is not optional polish — it is what makes the numbers mean anything.

This decision tree is a quick triage for a suspicious result. If the test score looks too good to be true, first check whether any preprocessing was fit before the split — that is leakage, fixed by splitting first. If preprocessing is clean, look for duplicate or grouped rows bridging train and test. If instead the score is not suspiciously high but the model tested well and then disappointed in production, the likely culprit is a split that did not match deployment conditions.

And if validation is strong and production holds up too, the pipeline is probably sound and you can proceed. Running through these branches converts a vague unease about a number into a structured diagnosis, pointing you straight at the specific mistake — leakage, contamination, or distribution mismatch — that is most likely responsible.

These sanity checks distill the entire post into habits you can run on autopilot. Split before any preprocessing so no statistics leak from held-out data. Touch the test set exactly once, after everything is frozen. Deduplicate and group by entity so no record or identity bridges the sets. Split time series by time, never by shuffle. And stratify your splits while ensuring the held-out data actually matches the production distribution.

None of these is sophisticated; together they prevent the overwhelming majority of split-related disasters. The unifying principle across all five posts is to protect the integrity of unseen-data estimates at every step, because that estimate is the only honest signal you have about how your model will behave in the real world. Guard it, and your results will hold up when it counts.

This wraps Day 31. Across five posts you have moved from the three roles of train, validation, and test, through why honest evaluation is the difference between a demo and a deployable system, into the mechanics of stratification, cross-validation, and leakage-free preprocessing, then a complete runnable pipeline, and finally the field guide of mistakes that quietly corrupt splits.

The 100 Days of AI series builds cumulatively, and proper evaluation underpins everything that follows — every model you train, tune, or compare in later days relies on trustworthy held-out estimates. Keep the cardinal rule and the anti-leakage habits handy; they transfer directly to model selection, hyperparameter tuning, and the more advanced workflows ahead.