Cross Validation

This is the hands-on post, and the goal is that you can copy each block in order and end up understanding cross validation from both the convenient and the from-scratch angle. We use the breast cancer dataset bundled with scikit-learn — 569 rows, 30 features, a binary label — because it is small enough that CV runs instantly yet realistic enough to show real behavior, including mild class imbalance that makes stratification matter.

Going block by block in strict order means each snippet is runnable on its own and builds on the last. By the final block you will have a basic CV estimate, a stratified one, a hyperparameter tuned by CV, a manual loop that demystifies the whole thing, and a final locked-away test — the complete, correct workflow rather than just an isolated function call.

The setup is intentionally minimal: scikit-learn supplies both the dataset and the model. load_breast_cancer returns features X and binary labels y, and printing their shapes confirms 569 samples and 30 features. Using a built-in dataset means anyone can run this immediately with no downloads or file paths to fuss over.

The RandomForestClassifier is a deliberate choice for a CV demo: it is a strong default that works well out of the box, it has obvious hyperparameters to tune later, and fixing random_state makes the whole notebook reproducible. Keeping the imports this lean is itself a teaching point — the entire cross validation workflow needs nothing beyond scikit-learn and, later, NumPy for the manual loop.

This block is cross validation at its most convenient: one function call. cross_val_score takes the model, the data, and cv=5, then handles everything — splitting into five folds, cloning a fresh model for each, fitting on four folds, scoring on the fifth, and returning the five scores as an array. Printing the array shows the per-fold results; printing the mean and standard deviation gives the estimate you would actually report.

The '+/- std' formatting is the habit from post 2 made concrete. Reporting 0.96 plus or minus 0.02 communicates both the typical performance and its stability in a single line. If you ever find yourself writing down just the mean, this block is the reminder to add the spread — it is one extra method call and it changes how trustworthy your number looks.

This block upgrades to stratified k-fold, the correct default for classification. By constructing a StratifiedKFold object with shuffling and a fixed seed and passing it as cv, you guarantee that every fold preserves the dataset's class balance — important even here, where the classes are somewhat uneven. The scoring='f1' argument also switches the metric from plain accuracy to F1, which respects both precision and recall.

The two changes work together. Stratification ensures each fold is a fair representation of the class distribution, and F1 ensures the score is not fooled by imbalance the way accuracy can be. This is the pairing post 5 will insist on: when classes are uneven, use stratified folds and an imbalance-aware metric. Getting it right here, on a clean dataset, builds the reflex for when it genuinely matters.

This tree diagram visualizes the search space that the upcoming grid search explores. There are two values for n_estimators and three for max_depth, and the grid is every combination of them — the tree shows how each top-level choice branches into the depth choices beneath it. Six leaf paths means six hyperparameter configurations to evaluate.

The point worth absorbing is the multiplication. Six configurations, each scored by 5-fold CV, means thirty model fits in total. Grid search is exhaustive — it tries every combination — so the cost grows as the product of the option counts times the number of folds. Seeing the branching structure here makes the runtime of the next block predictable and explains why large grids get expensive fast.

This block is hyperparameter tuning done correctly, with CV as the scoring engine. GridSearchCV takes the model, the parameter grid, the stratified splitter, and the F1 metric, then exhaustively trains and cross-validates every combination. After fit, best_params_ reports the winning configuration and best_score_ reports its cross-validated F1.

Notice that the same stratified splitter from the previous block is reused as cv, so every configuration is judged on identical folds — the only fair way to compare them. This is the nesting from post 3 made real: each of the six configurations triggers a full 5-fold CV internally. One important caveat, which post 5 elaborates: the best_score_ here is slightly optimistic precisely because you chose the configuration that scored best on these folds, which is why a separate test set still matters.

This block strips away scikit-learn's convenience to show there is no magic underneath. skf.split(X, y) yields the train and test indices for each fold; you loop over them, build a fresh model each time, fit it on the training indices, score it on the held-out indices, and collect the scores. Taking the mean reproduces what cross_val_score did in one line.

Writing it out by hand is clarifying in two ways. First, you see explicitly that a brand-new model is created and trained every fold, never reused — the independence that makes the average meaningful. Second, you see that the held-out indices are never touched during fitting, which is the whole no-cheating guarantee of CV. Once you have written this loop, the convenience functions feel like exactly what they are: a tidy wrapper around these few lines.

This trace annotates the four load-bearing lines of the manual loop in plain language. skf.split(X, y) produces the train and test index arrays for each fold, encoding the rotation. m.fit on the training indices trains a fresh model on the k-1 folds. m.score on the test indices evaluates that model on the one fold it never saw. And np.mean of the collected scores yields the cross validation estimate.

Reading these four operations together is reading the entire algorithm. Everything else in the block — the loop, the list, the imports — is just plumbing to feed these steps. This is the same skeleton that underlies cross_val_score, GridSearchCV, and every variant from post 3; they differ only in how split generates the indices and how the scores are aggregated.

This final block demonstrates the workflow discipline that protects you from fooling yourself, and it is the most important block in the post. Before any cross validation or tuning, you split off a stratified test set with train_test_split and lock it away. All CV and grid search then happen only on the training portion. At the very end, you score the tuned model once on the untouched test set.

The ordering is everything. Because the test set was separated before tuning, the optimistic bias in best_score_ — caused by choosing the configuration that won on the CV folds — does not contaminate it. The test score is therefore an honest estimate of real-world performance. This is the train/validation/test separation from post 1, realized in code, and the single most effective guard against the tuning-then-testing-on-the-same-data trap that post 5 dissects.

These are the levers you turn to adapt the workflow, and experimenting with them builds intuition fast. Switching cv between 5 and 10 changes the bias-variance-cost tradeoff from post 3. The scoring argument lets you optimize for what actually matters — accuracy, F1, ROC AUC, or many others — rather than blindly defaulting to accuracy. shuffle with a fixed random_state makes runs reproducible so comparisons are fair.

n_jobs=-1 is the practical one: it runs the independent folds in parallel across all your CPU cores, which can turn a slow grid search into a fast one at no cost to correctness, since the folds are independent by construction. And because the loop is model-agnostic, you can swap RandomForestClassifier for any scikit-learn estimator and everything else stays the same — CV is a wrapper around models, not tied to any one.

These are the specific traps lurking in this exact workflow, previewing post 5 in the concrete. Tuning on data you also report on inflates your number, which is why the final block locked away a test set. Plain KFold on imbalanced labels can produce meaningless folds, which is why we used StratifiedKFold. Scaling before the split leaks test information into training and is the single most common leakage bug — the fix is a Pipeline, shown in post 5. And reporting the mean without the standard deviation hides how stable or shaky your estimate really is.

Noticing these here, while the code is fresh and the dataset is forgiving, trains the instincts you will need on messier real data where the same mistakes are far harder to spot and far more costly when they slip through.

The takeaway from this build is that cross validation's intimidating reputation reduces to a short, runnable workflow: a one-line score, a stratified upgrade, a CV-driven grid search, a from-scratch loop that demystifies it, and a locked-away test set that keeps you honest. Everything more advanced is a variation on these same pieces.

Next we turn to the failure modes. This code worked cleanly because the dataset is well-behaved and we were careful about ordering. Real projects rarely are. Post 5 catalogs the traps — preprocessing leakage, tuning on your test data, ignoring groups and time, and the wrong metric on imbalanced data — that quietly inflate CV scores and then ambush you in production.