Train / Validation / Test Splits

This is the hands-on post, and the goal is that you can copy each block in order and end up with a complete, leakage-free evaluation pipeline. We use scikit-learn and the built-in breast cancer dataset because it is small, real, and slightly imbalanced — enough to exercise stratification without any download or setup. Every snippet runs on its own and builds on the previous one.

By the final block you will have done a stratified three-way split, scaled features without leaking, trained a model, checked the train/validation gap, cross-validated for stability, and produced a single honest test number. That sequence is a template you can paste into almost any tabular ML project and trust.

The setup imports exactly the tools the pipeline needs and nothing more. From scikit-learn we pull the dataset loader, the splitting and cross-validation functions, the StandardScaler for feature scaling, a LogisticRegression model, and an accuracy metric. A single pip install covers all of it.

Keeping the imports explicit at the top makes the pipeline's shape visible before you read a single line of logic: load data, split it, scale it, train, cross-validate, score. Each import maps to one stage. This is a deliberately conventional, boring stack — and that is the point. The splitting discipline, not exotic tooling, is what makes the results trustworthy.

Here we load the data and take a quick look. The breast cancer dataset has 569 samples and 30 numeric features, with a binary target. Printing the shape confirms the dimensions, and printing y.mean() shows roughly 0.63, meaning about 63% of samples are one class — a mild imbalance that is exactly why we will stratify our splits.

Getting in the habit of inspecting shape and class balance immediately after loading is good practice. The class ratio in particular drives a downstream decision: any meaningful imbalance is a signal to pass stratify when splitting, so that train, validation, and test each preserve this 63/37 mix rather than leaving it to chance.

This block produces the three-way split with stratification, using two calls to train_test_split. The first call peels off 20% as the test set, passing stratify=y so the class balance is preserved. The second call splits the remaining 80% into train and validation, using test_size=0.25 — which is 25% of the 80%, or 20% of the original — and stratifying again on that subset.

The arithmetic lands at 60% train, 20% validation, 20% test. Note that random_state is fixed in both calls for reproducibility, and stratify is applied at each step so every one of the three sets faithfully mirrors the original class distribution. This two-call pattern is the standard idiom for a stratified three-way split in scikit-learn.

This pipeline diagram narrates the row-level flow of the previous code block. You start with the full dataset of 569 rows. The first split seals off 20% as the test set, which is then locked away and not touched again until the final block. The second split divides the remaining rows into training and validation. The end state is three clean sets in a 60/20/20 ratio.

Visualizing it as a sequence of cuts clarifies why the test set is special: it is separated first and immediately set aside, never participating in scaling, training, or tuning. That early sequestration is the structural guarantee that nothing downstream can leak into it, which is exactly what makes its final score honest.

This block applies the fit/transform discipline that prevents preprocessing leakage. We create a StandardScaler and call fit_transform on the training features only — this learns the per-feature mean and standard deviation from the training data and standardizes it in one step. We then call transform on validation and test, applying those same learned statistics without recomputing them.

The comment states the guarantee: the scaler never sees validation or test statistics. This is the concrete realization of the rule from post 3, and it is the step most often gotten wrong. If you had called fit_transform on the test set, its distribution would leak into the preprocessing and inflate your final score. Fitting on train alone keeps the test set a true stand-in for unseen data.

Now we train the model and immediately run the most valuable diagnostic: the train-versus-validation gap. We fit a LogisticRegression on the scaled training data, then compute accuracy on both the training and validation sets. Printing them side by side — here about 0.99 train and 0.98 validation — shows a small gap, the signature of a model that is generalizing well rather than memorizing.

Making this comparison right after training should be reflexive. A small gap is healthy. A large gap, say 0.99 train against 0.72 validation, would scream overfitting and tell you to simplify the model, add regularization, or get more data before going any further. This two-line check is the cheapest, highest-value habit in the whole pipeline.

This block uses cross-validation to get a more stable estimate of model quality than a single validation split can give. Because cross-validation rotates the validation fold internally, we combine our separate train and validation sets back into one development set, then hand it to cross_val_score with cv=5. The function trains and evaluates the model five times on rotating folds and returns the scores.

Reporting the mean and standard deviation — here about 0.98 plus or minus 0.01 — tells a richer story than one number. The mean is the central estimate, and the small standard deviation confirms the model performs consistently regardless of which rows it trains on. A large standard deviation here would warn that the single-split result was lucky or unlucky and should not be trusted on its own.

This trace summarizes the pipeline's four load-bearing actions in plain language. First, the split carves off the test set, then divides the rest into train and validation. Second, the scaler is fit only on the training data via fit_transform, with validation and test merely transformed. Third, the model is fit and the train-versus-validation gap is checked as an overfitting diagnostic. Fourth, cross-validation produces a stable, low-variance estimate.

Reading these four lines in order is a compact mental model of correct evaluation. Each one corresponds to a principle from earlier posts — clean separation, leakage prevention, overfitting detection, and stable estimation — now expressed as concrete code. If you internalize this trace, you can reconstruct the entire pipeline from memory.

This is the moment everything has been protecting: the single, one-shot test evaluation. Only now, after the split, the leakage-free scaling, the training, the gap check, and the cross-validation — after every decision is frozen — do we touch the test set, computing accuracy once. The result, around 0.97, is the number you report and the most honest estimate of real-world performance you have.

The comment carries the discipline that makes it honest: do not go back and tune after seeing this. The instant you adjust the model in response to the test score and re-run it, you have used the test set to tune, and it stops being unbiased. Treat this print statement as the terminal step of the project. One look, one number, done.

These are the levers you adjust to adapt the pipeline to your own problem. test_size controls how large each held-out set is, traded against how much data the model gets to learn from. Passing stratify=y preserves class balance and should be on by default for any imbalanced classification. The cv argument sets the number of cross-validation folds, balancing estimate stability against compute. random_state fixes reproducibility.

The most important property is the last bullet: the entire structure is model-agnostic. Swap LogisticRegression for a random forest, a gradient booster, or a neural network and every line around it stays the same. The split-scale-train-validate-test discipline is a fixed scaffold; the model that sits inside it is interchangeable.

These four warnings are the FrozenLake-style traps specific to this pipeline, each mapping to a mistake covered in depth in post 5. Calling fit_transform on validation or test instead of transform leaks their statistics and inflates the score. Forgetting stratify on imbalanced data leaves your class proportions to luck and destabilizes every metric. Tuning after seeing the test score converts your unbiased estimate into a tuned, dishonest one.

The last warning — reusing the test set as a working validation set — is the most common shortcut and the most damaging, because it quietly collapses the three-set structure back into a leaky two-set one. Watching for these four specific symptoms while writing the code trains the instincts that keep your evaluations honest on far messier real-world data.

The takeaway from this build is that a fully correct evaluation pipeline is short, conventional, and reusable. A stratified three-way split, a scaler fit only on train, a train/validation gap check, a cross-validation for stability, and a single test score — that is the entire template, and it transfers to almost any tabular problem with a one-line model swap.

This code worked cleanly because the dataset is well-behaved and we followed every rule. Real projects are messier, and the rules are easy to violate without noticing. The final post catalogs exactly how splits get corrupted in the wild — leakage through preprocessing, tuning on test, duplicate rows, mis-split time series, and imbalance — so you can recognize the symptoms before they cost you.