Train / Validation / Test Splits

The train / validation / test split is one of the most fundamental disciplines in all of machine learning, and it is the thing that separates honest results from self-deception. The exam analogy is not a beginner's simplification; it is exactly right. Judging a model on the same data it learned from tells you how well it memorized, which is almost never what you care about.

This first post stays deliberately at the level of roles and intuition. The mechanics — stratification, cross-validation, leakage prevention — come later. Before any of that, you need three concepts locked in: which slice the model learns from, which slice you tune against, and which slice stays sealed until the very end.

The core move is to partition your dataset into three disjoint parts, each with a single clear job. The training set is the only data the model's parameters are fit on. The validation set is where you make decisions: which architecture, which hyperparameters, when to stop. The test set is locked in a drawer and opened exactly once, to produce the number you actually report.

The word disjoint matters. No row may appear in more than one set. The instant a test example also lives in training, the model has effectively seen the answer, and your final estimate is corrupted. Keeping the three sets cleanly separated is the foundation everything else rests on.

Generalization is the entire point of the exercise. A model that does brilliantly on its training data has proven only that it can fit data it has already seen — which a lookup table can do perfectly. What you actually want to know is how it behaves on inputs it has never encountered, because that is what production is: a stream of new, unseen data.

Splits exist to manufacture a stand-in for that unseen data while you still have the labels to check against. The held-out sets let you measure generalization honestly before you deploy, turning the vague hope that the model 'works' into a concrete, defensible number.

The bar chart shows a common default: roughly 70% of the data for training, 15% for validation, and 15% for test. These proportions are a starting point, not a law. The training set gets the largest share because more data generally means a better-fit model, while validation and test only need to be large enough to give stable, trustworthy estimates.

The right ratio depends heavily on dataset size, a point we return to in the tips slide. With a few hundred rows you lean toward more held-out data and cross-validation. With tens of millions of rows, even 1% is a huge, stable test set, so you can afford to give the vast majority to training.

This slide pins down precisely what each set is allowed to influence. The model's weights are fit only on the training set — that is the literal definition of training. The validation set never touches the weights; instead it informs your decisions, the choices you make as the human in the loop about model structure, hyperparameters, and when to stop training.

The test set is the strictest of all: it influences nothing. It does not update weights and it does not guide your choices. It exists solely to report a verdict after everything is frozen. Confusing these roles — letting validation update weights, or letting test guide choices — is the source of most evaluation disasters.

Why three sets and not two? The comparison makes the failure mode concrete. With only train and test, you inevitably tune by looking at the test score — try something, check test, adjust, repeat. Each adjustment fits your decisions a little more to that particular test set, so its score creeps upward and stops reflecting true performance. You have quietly overfit the test set through your own choices.

Adding a validation set absorbs all that tuning pressure. You make your dozens of decisions against validation, which is allowed to get a bit optimistic, and reserve the test set for a single, untainted final measurement. The clean separation of jobs is what keeps the final number honest.

The cardinal rule deserves its own slide because violating it is the most common way to ruin an evaluation. Never let the model learn from the test set — and 'learn from' is broader than it sounds. It includes the obvious case of training on test rows, but also subtler leaks: fitting a scaler on test statistics, selecting features using the full dataset, or repeatedly tuning until test looks good.

The moment any information flows from test into your model or your decisions, the test score stops being an honest estimate of unseen performance. Since that honest estimate is the only reason the test set exists, leaking into it throws away the entire benefit. Guard it like a sealed envelope.

This snippet shows the mechanically simplest way to get three sets with scikit-learn's train_test_split. Because the function splits in two, you call it twice: first peel off 20% as a temporary holdout, then split that holdout in half to get validation and test. The result is an 80/10/10 partition. Setting random_state makes the split reproducible, so you and your collaborators get identical sets.

This is the bare-bones version — it shuffles and splits randomly. In the next post we add the refinements that make it production-grade: stratification to preserve class balance, and the discipline of fitting all preprocessing on the training portion only. For now, the takeaway is that getting three honest sets is just two function calls.

The flow diagram traces the lifecycle of a real project through the three sets. You fit the model on the training set, then use the validation set to tune and compare candidates, then pick the best configuration and freeze it. Only after the model is completely frozen do you bring out the test set, run it once, and report the result.

The directionality matters: information flows left to right and never loops back. Once you have opened the test set, you do not return to tuning — doing so would mean the test set informed your choices, which is exactly the leak the whole structure is designed to prevent. Treat 'report once' as a terminal step.

Split ratios are guidelines tuned to dataset size, not sacred numbers. For small datasets, 60/20/20 keeps the held-out sets large enough to give stable estimates, though you will often prefer cross-validation here to use the data more efficiently. For medium datasets, 70/15/15 or 80/10/10 strikes a good balance.

For very large datasets — millions of rows — the logic flips. The held-out sets only need to be big enough to estimate performance stably, and 1% of ten million rows is a hundred thousand examples, plenty. So a 98/1/1 split is entirely reasonable and gives the model the maximum possible training data. The principle: validation and test need enough rows for a stable estimate, and everything beyond that should go to training.

This slide answers a question that trips up almost everyone at first: if both validation and test are held out from training, why do we need both? The distinction is how often each is used. The validation set is consulted repeatedly — every model you compare, every hyperparameter you try, touches it — so over many decisions your choices slowly fit to its particular quirks. Its score gently inflates.

The test set is used exactly once, which is precisely what keeps its estimate unbiased. A single measurement cannot be tuned toward. So the two sets are not redundant; they play different statistical roles. Validation is the sparring partner you train against many times; test is the one official match that counts.

This closing slide points ahead to post 2, which makes the case for why this discipline is worth the effort. Saving and following keeps the five-post arc together: concept, then why it matters, then how to do it, then code, then the mistakes that bite.

The teaser frames post 2 around the costly, invisible failures that proper splits catch — overfitting, leakage, and dishonest comparisons that look fine in a notebook and detonate in production. Having the three roles locked in from this post is exactly what makes that motivation land.