Cross Validation

This final post is the field guide to how cross validation lies to you, and it is arguably the most useful post in the set, because a corrupted CV score is more dangerous than no score at all — it gives you false confidence. The recurring theme is leakage: information from the held-out data sneaking into the training process so that the 'test' is no longer a real test. The score looks great right up until production data, which cannot leak, corrects you brutally.

The 0.97-becomes-0.81 story in the hook is the canonical experience. The cross validation procedure did exactly what you told it to; the problem was that you told it something subtly wrong. None of these mistakes are exotic — they appear in a large fraction of real projects, they are predictable, and they are diagnosable once you know the symptoms. Treat this post as a checklist you run whenever a score looks too good.

Preprocessing leakage is the most common and most insidious CV mistake. It happens when you fit a data transformation — scaling, imputation, feature selection, anything that learns statistics from the data — on the entire dataset before cross validation splits it. Once you do that, the transform has already seen every fold's test data, so when CV holds out a fold, that fold is not truly unseen: its statistics already shaped the preprocessing.

The damage is an inflated, optimistic score that does not survive contact with genuinely new data, where no such peeking is possible. Feature selection on the full dataset is especially dangerous — choosing features using labels from rows that later become test data can manufacture impressive scores out of pure noise. The rule is absolute: any step that learns from data must be fit inside the cross validation loop, on the training fold only.

This comparison contrasts the leaky and clean orderings of operations, which is the entire crux of the mistake. On the leaky left, you scale on all the data and then split into folds — by then the scaler's mean and variance already encode the test rows, so test information has bled into the transform and the score comes out too optimistic. On the clean right, you split first, fit the scaler on each training fold, and merely apply it to the corresponding test fold.

The difference looks tiny on a slide and is enormous in practice. The clean order guarantees that nothing the model or its preprocessing learned ever came from the data it is being scored on. Internalizing this ordering — split first, then learn everything from the training portion only — is the mental model that prevents the entire category of leakage bugs.

This snippet shows the clean fix, and it is gratifyingly simple: wrap the preprocessing and the model together in a scikit-learn Pipeline, then cross-validate the pipeline. When cross_val_score runs the pipeline on each fold, it refits the entire pipeline — scaler included — on that fold's training data only, then applies it to the held-out fold. The leakage becomes structurally impossible.

This is why pipelines are not just tidiness but correctness. The moment you have any learned preprocessing step, doing CV on a bare model with pre-transformed data is a bug waiting to happen, whereas doing CV on a pipeline is automatically correct. The comment captures the one rule to never break: never fit a transform on the full X before cross validation. Make pipelines your default and you eliminate the most common leakage mistake by construction.

Tuning then testing on the same data is a subtler leak, and it catches even experienced practitioners. When you use cross validation to search over hyperparameters and then report the best CV score you found as your result, that number is optimistically biased. The reason: by selecting the configuration that scored highest on those particular folds, you have implicitly fit to the idiosyncrasies of those folds. The 'best' score partly reflects luck that will not repeat.

The principled fixes are two. Nested cross validation uses an inner loop to tune and an outer loop to estimate performance, so the data used to choose settings is never the data used to judge them. The simpler, cheaper fix is to lock away a final test set before any tuning, as the code in post 4 did, and report performance only on that. Either way, the data that drove your choices cannot also be the data that grades them.

This stack diagram shows the structure of nested cross validation, which is the rigorous answer to the tuning-bias problem. The outer loop's job is to estimate performance: it splits off a test fold and asks 'how well does the whole tuning-plus-training procedure do on data it never saw?' The inner loop, running entirely within the outer loop's training data, does the hyperparameter search. The innermost level simply fits the model on a training fold.

The separation of duties is the point. Because the inner loop only ever sees the outer loop's training portion, the outer test folds remain genuinely untouched by the tuning process, so the outer scores are an unbiased estimate of real performance. Nested CV is more expensive — folds times folds times configurations of model fits — but when you need a defensible performance number from a small dataset and also need to tune, it is the correct tool.

Ignoring groups and time order is leakage wearing a different costume. The random-fold assumption is that rows are independent, but they often are not. If the same user, patient, or product contributes multiple rows and those rows are scattered across train and test, the model can learn to recognize that specific entity rather than the general pattern — and its inflated score evaporates on truly new entities. GroupKFold prevents this by keeping every group entirely on one side.

Time ordering is the other classic violation. Shuffling time-stamped data lets a fold train on future events to predict past ones, which is impossible at deployment and produces absurdly optimistic scores. TimeSeriesSplit enforces the arrow of time, always training on earlier data and testing on later. Both mistakes share a root cause — using a splitter that ignores how the data is actually structured — and both are fixed by matching the splitter to the data, exactly the decision tree from post 3.

This snippet shows the structural fixes in code, and they cost only a different splitter. GroupKFold takes a groups array — here a per-row user id — and guarantees that all rows sharing a group land together in either train or test, never split across the boundary. That single change closes the entity-recognition leak entirely. Passing groups to cross_val_score wires it through.

TimeSeriesSplit needs no group information; it simply respects row order, producing folds where the training set always precedes the test set in time and the training window grows fold over fold. The lesson the comments drive home is that respecting structure is not extra effort, just a more appropriate tool. Whenever you catch yourself reaching for plain KFold, pause and ask whether your rows are really independent — if not, one of these is the correct choice.

Choosing the wrong metric quietly defeats the entire point of cross validation. On a severely imbalanced dataset — say 99% negatives — a degenerate model that predicts 'negative' for everything achieves 99% accuracy while learning absolutely nothing useful. Plain k-fold with accuracy as the metric will report that 99% with a straight face, and you will believe you have a great model when you have a useless one.

The fix has two parts that work together. Use stratified folds so each fold actually contains the rare class in representative proportion, and use a metric that respects imbalance — F1, precision and recall, or ROC AUC — so that ignoring the minority class is penalized rather than rewarded. A high CV score under accuracy on imbalanced data is not evidence of a good model; it is an illusion produced by the wrong yardstick. Pick the metric that matches what you actually care about.

This decision tree is a triage procedure for the most dangerous situation in applied ML: a cross validation score that looks suspiciously high. The first suspect is preprocessing before the split — if any transform was fit on the full dataset, move it into a Pipeline so it refits per fold. If preprocessing is clean, the next suspect is structure: were groups or time order ignored? If so, switch to GroupKFold or TimeSeriesSplit.

If neither applies, examine the metric against the class balance — a great accuracy on imbalanced data is a red flag for the wrong-metric trap. And if you find none of these, the score may genuinely be fine, but you should still confirm it on a locked-away test set before trusting it. Running this checklist whenever a number looks too good to be true is the single most valuable habit this post can give you, because a too-good CV score is almost always a bug, not a triumph.

These sanity checks are the habits that separate reliable cross validation from accidental self-deception, and they map directly onto the mistakes this post covered. Putting all preprocessing in a Pipeline makes leakage structurally impossible. Locking away a final test set early defeats the tuning-bias trap. Matching the splitter to the data shape — stratified, grouped, or time-series — closes the structural leaks. Choosing a metric that fits the imbalance prevents the wrong-yardstick illusion. And reporting the mean alongside the standard deviation keeps you honest about stability.

None of these are sophisticated; they are simply discipline. The unifying instinct is suspicion: when a score looks great, assume something leaked until you have ruled it out. The practitioners who ship models that actually work in production are the ones who have internalized that a clean-looking CV number earns trust only after it has survived this checklist.

This wraps Day 32. Across five posts you have moved from the concept of a rotating held-out set, through why a single split poisons every downstream decision, into the k-fold loop and its structural variants, then a complete runnable workflow, and finally the leakage and metric traps that decide whether your CV score means anything at all.

The meta-lesson is that cross validation is only as honest as the discipline around it. The mechanics are simple; the failures are subtle and they all share a root — letting the held-out data influence what you do before you score on it. Carry the checklist from this post into every project. The 100 Days of AI series builds cumulatively, and trustworthy evaluation is the foundation that makes every model and technique that follows worth measuring at all.