Cross Validation

This is the engine-room post. Having established what cross validation is and why it matters, we now open the hood on the machinery. The reassuring news is that the core is a single loop — partition, train, score, rotate, average — and every variant you will ever meet is just that loop adapted to a particular shape of data. Once you see the loop clearly, stratified, grouped, time-series, and leave-one-out stop looking like a confusing zoo and start looking like sensible patches.

The through-line for the whole post is that the variants exist because real data violates the assumptions of the naive loop. Plain k-fold assumes rows are interchangeable and independent. When they are not — imbalanced classes, repeated entities, time ordering — you need a variant that respects the structure, or your estimate quietly becomes wrong.

The k-fold loop is the entire algorithm, and it fits in a sentence: shuffle, split into k folds, then for each fold train a fresh model on the other k-1 folds and score it on the held-out one, and finally average the scores. Shuffling first matters because it breaks any accidental ordering in the data, so the folds are representative random samples.

The word 'fresh' is important. Each round starts from an untrained model; you are not continuing to train one model across folds, you are training k independent models and collecting k independent scores. That independence is what makes the average a meaningful estimate. And reporting the spread alongside the mean, as post 2 stressed, turns the loop's output from a point guess into an estimate with an honest error bar.

This cycle diagram traces one complete pass of 5-fold CV as a loop you go around. You shuffle and split into folds once at the start. Then you enter the cycle: hold out fold i, train on the rest, score on fold i, record it, advance to the next fold, and repeat until all five folds have been held out. When the cycle completes, you average the recorded scores.

Drawing it as a cycle rather than a straight line emphasizes that the same operation repeats with only the held-out fold changing each time. There is no special first or last fold; the structure is perfectly symmetric. This symmetry is exactly why every row gets equal treatment — each one is tested exactly once and contributes to training in every other round.

Choosing k is the one genuine tuning decision in basic CV, and it is a bias-variance-cost tradeoff. A small k like 3 means each model trains on only two-thirds of the data, so it learns from less and tends to underperform the model you will eventually train on everything — that makes the estimate pessimistically biased. It is also cheaper, since you fit only three models.

A large k trains each model on nearly all the data, reducing that bias, but the training sets across folds now overlap heavily, so the fold scores are highly correlated and the averaged estimate can actually have higher variance, all while costing more fits. The empirical sweet spot that the field has converged on is k equals 5 or 10 — enough training data per fold to keep bias low, enough independence between folds to keep variance and cost reasonable.

This comparison lays the k tradeoff out side by side so the competing pressures are visible at once. Small k is cheaper and faster but starves each model of training data, biasing the estimate low and leaving more variance. Large k feeds each model almost all the data, lowering bias, but the heavily overlapping training sets make the fold scores correlated and the whole procedure slower.

The practical reading is that there is no free lunch at either extreme, which is why the conventional 5 or 10 exists. Reach for 10 when your data is small enough that you want to maximize training data per fold and can afford the extra fits; reach for 5 when models are expensive or data is plentiful enough that the bias from a smaller training set is negligible. Either way, you are navigating this exact tradeoff.

Stratification is the first variant you should internalize because class imbalance is everywhere. If 5% of your rows are positive, plain random folds can, by chance, produce a fold with almost no positives — and a test fold that barely contains the class you care about yields a score that means nothing. Worse, the imbalance can vary fold to fold, inflating the apparent variance.

Stratified k-fold solves this by constructing each fold to preserve the overall class proportions. If the dataset is 5% positive, every fold is roughly 5% positive. For essentially any classification task, especially imbalanced ones, this is the default you should reach for rather than plain k-fold. Scikit-learn even applies it automatically when you pass an integer cv to a classifier, but knowing to demand it explicitly keeps you safe when you build your own splits.

This snippet shows stratified k-fold done explicitly, which is the form you will write whenever you want control. You construct a StratifiedKFold object with the number of splits, turn on shuffling, and fix a random_state for reproducibility, then hand that splitter to cross_val_score via the cv argument. From then on, every fold preserves the class balance of y.

Two details earn their place. shuffle=True breaks any ordering in the data before stratifying, and random_state makes the exact split reproducible so your results do not change run to run — essential when you are comparing models and need an apples-to-apples split. Passing the splitter object rather than an integer is also what lets you reuse the identical folds across several models, which is the fair way to compare them.

Grouped and time-series splits exist because the independence assumption behind random folds often fails in real data. If the same patient contributes several rows and those rows are scattered across train and test, the model can learn to recognize that specific patient rather than the underlying condition — a subtle but serious form of leakage that inflates your score. GroupKFold prevents it by keeping every row from a given group entirely on one side of the split.

Time-ordered data has its own trap: shuffling lets the model train on future data to predict the past, which is impossible at deployment and produces absurdly optimistic scores. TimeSeriesSplit enforces causality by always training on earlier data and testing on later data, with the training window growing over successive folds. Choosing the right splitter for your data's structure is not optional polish; it is the difference between a valid estimate and a fantasy.

This decision tree is a practical flowchart for picking a splitter, and it is worth committing to memory because choosing wrong silently corrupts your estimate. First ask whether the data is ordered in time — if so, TimeSeriesSplit, full stop, because shuffling would let you cheat by training on the future. If not, ask whether rows belong to groups like users or patients that must not straddle the split — if so, GroupKFold.

If neither applies, ask whether it is a classification problem with class imbalance — if so, StratifiedKFold to keep proportions intact. Only when none of these structural concerns apply is plain KFold the right tool. Running through these questions before every project takes ten seconds and prevents the most damaging category of CV mistake, the kind covered in detail in post 5.

Leave-one-out CV is k-fold pushed to its logical extreme: set k equal to n, the number of rows, so each fold is a single example. Every model trains on all but one row and is tested on that lone held-out point. Because the training set is almost the entire dataset, the estimate has very low bias — each model is nearly the model you would deploy.

The costs are steep, though. You fit n models, which is expensive for anything but tiny datasets, and because the training sets differ by only one row, the models are nearly identical and their scores are highly correlated, which can give the averaged estimate surprisingly high variance. The honest verdict: leave-one-out is a specialist tool for very small datasets where every row is precious and the n fits are affordable, not a general-purpose default.

This flow diagram shows the relationship that confuses people most: cross validation does not replace hyperparameter tuning, it lives inside it. To tune, you pick a candidate setting — say, tree depth 5 — then run a full k-fold CV to get a stable score for that setting. You repeat for every candidate setting, comparing their CV scores. Finally you keep the best setting and refit the model on all the data.

The key insight is that each evaluation in the tuning search is itself an entire k-fold run. A grid search over twelve settings with 5-fold CV trains sixty models. This is exactly what GridSearchCV automates, and seeing the nesting clearly here is what makes the code in post 4 — and the nested-CV warning in post 5 — read as obvious rather than mysterious.

This closing slide points to post 4, where all of this machinery becomes a program you can run. Saving and following keeps the build connected to the concepts you just learned, so the code reads as a direct realization of the k-fold loop and its variants rather than unfamiliar syntax.

The teaser promises a complete build: a basic k-fold score, the stratified version, a grid search that uses CV to tune, and a from-scratch loop that strips away scikit-learn's convenience so you can see the rotation happening with your own eyes. Watching the manual loop produce the same number as cross_val_score is the moment the engine room stops being abstract.