Random Forests

This is the hands-on post. Theory becomes muscle memory only when you run the code, so everything here is copy-paste runnable on the breast-cancer dataset bundled with scikit-learn. We walk the complete professional workflow: import, split, fit, check the out-of-bag score, evaluate beyond accuracy, read importances, and tune.

The dataset is a good choice — real medical features, two classes, mild imbalance — so the steps mirror what you'd do on an actual problem rather than a toy. Type it out and change things: drop the n_jobs, shrink the trees, flip a parameter, and watch how the metrics and timing respond.

Step zero is the imports, and they preview the whole workflow. load_breast_cancer supplies the data, train_test_split handles the holdout, and RandomForestClassifier is the model. Notice what's absent: there's no StandardScaler import, because forests don't need feature scaling — a first hint at how little preprocessing this model demands.

The pip comment is there because scikit-learn isn't in the standard library. Grouping imports by their role makes the script self-documenting: anyone reading these lines can already guess the shape of what follows — load, split, fit — before reading a single line of logic.

Loading and splitting is the foundation, and the details matter. We pull features X and target y, then split off 20% as a test set. Two arguments do important work: stratify=y preserves the class proportions in both splits, which matters because the dataset is imbalanced, and random_state=42 makes the split reproducible so your numbers match across runs.

The printed shape, (455, 30), confirms 455 training rows and 30 features. Always eyeball shapes after a split — a surprising number here is the fastest way to catch a data-loading bug before it silently corrupts everything downstream.

Fitting is anticlimactic by design: instantiate RandomForestClassifier and call fit. We set n_estimators=300 for a stable ensemble, oob_score=True to get the free validation estimate (covered next), random_state=42 for reproducibility, and n_jobs=-1 to train the trees in parallel across all CPU cores. Test accuracy lands around 0.965.

The comment flags the key convenience from earlier posts: no feature scaling is needed. You can hand the forest raw features of wildly different magnitudes and it will split on thresholds without complaint. That's a real reduction in the surface area for preprocessing bugs compared with scale-sensitive models.

This slide states the discipline that still applies even though scaling doesn't. Trees split on thresholds, so the absolute scale of a feature is irrelevant — standardizing changes nothing about which splits are chosen. That genuinely removes a step many other models require.

But the rest of good practice is unchanged: split before you fit anything that learns from data, use stratify on imbalanced labels so the test set is representative, and keep the test set untouched until final evaluation. Forests are forgiving about preprocessing, not about leakage — the mistakes post drills into exactly how leakage sneaks in.

This step shows the out-of-bag shortcut in action. Because we passed oob_score=True, scikit-learn computed, for every training row, a prediction using only the trees whose bootstrap sample excluded that row, then scored those predictions. The result, around 0.958, is an honest accuracy estimate obtained with no separate validation split and no cross-validation loop.

This is the free-lunch property from post 3 made real. When data is scarce, the OOB score lets you assess the model without sacrificing rows to a holdout. It tracks cross-validation accuracy closely in practice, so it's a quick, cheap gauge of generalization you get just by flipping one flag.

Accuracy is a trap on imbalanced data, so this step insists on better metrics. The confusion matrix shows the four outcome types; precision and recall separate the two error types; ROC-AUC on predicted probabilities measures ranking quality independent of any threshold.

The medical framing makes the stakes vivid. A false negative here means telling someone with a malignant tumor that they're fine — far worse than a false positive that triggers an extra test. Your evaluation metric must encode that asymmetry, which usually means prioritizing recall on the malignant class rather than chasing a single accuracy number.

This snippet runs the real evaluation. We pull the positive-class probabilities from predict_proba (column 1), get hard labels from predict, then print a full classification_report — per-class precision, recall, and F1 — alongside roc_auc_score on the probabilities, which comes out around 0.995.

Note that AUC takes probabilities, not labels: it's threshold-independent by construction and measures how well the forest ranks positives above negatives. Pairing the per-class report (threshold-dependent) with AUC (threshold-independent) gives a complete picture — how good the ranking is, and how well it translates into decisions at your chosen cutoff.

Interpretation closes the loop back to post 2's promise. We sort feature_importances_ by magnitude and print the top three feature names with their scores. The largest values — here features like worst radius and worst perimeter — are the signals the forest leaned on most heavily across all its trees.

These impurity-based importances are fast and convenient, but read them as 'the model uses this signal,' not as causation, and be aware they can be biased toward high-cardinality features. When the ranking really matters for a decision, the mistakes post shows permutation importance computed on held-out data, which is more trustworthy. For a quick gut check, though, this one-liner is invaluable.

Tuning is where you spend effort only after the basics are solid. This grid search explores the three knobs that matter most: n_estimators (more trees, more stability), max_depth (how deep each tree may grow, controlling per-tree bias), and max_features (how many features each split may consider, controlling tree de-correlation). GridSearchCV evaluates every combination with 5-fold cross-validation and reports the best.

Forests are mercifully insensitive to tuning compared with boosting or neural nets, so don't over-invest here — the default settings are usually close. The biggest practical lever is often max_features, since it directly controls the de-correlation that drives variance reduction. Start there if you tune at all.

This pipeline diagram is the whole post in four stages: split (stratified), fit the RandomForest, evaluate with OOB, AUC, and recall, and tune with GridSearchCV. It's the canonical supervised-learning workflow with a forest dropped into the model slot.

Internalizing this shape pays off far beyond forests. Swap the model box for logistic regression or a gradient booster and the surrounding stages are identical. The discipline — proper stratified split, honest cost-aware evaluation, deliberate tuning — is what separates a trustworthy result from a misleading notebook, regardless of which algorithm sits in the middle.

These pitfalls are the workflow's sharp edges, gathered in one place. Use stratify on imbalanced labels so the split is representative. Set n_jobs=-1 to train trees in parallel and save real time. Don't read importances as causation. Judge with AUC and recall, not accuracy alone. And remember that adding trees costs time and memory but never hurts accuracy — so there's no accuracy risk in raising n_estimators, only a compute cost.

Every one of these turns a 'working' notebook into a model that disappoints or misleads. Internalize them now, because the next post is entirely about mistakes — and several of these reappear there with deeper treatment and concrete fixes.

You now have a random forest working end to end and, thanks to the earlier posts, you understand every step rather than just running it. Split with stratification, fit without scaling, read the free OOB estimate, evaluate with metrics that match your costs, interpret importances with appropriate caution, and tune the few knobs that matter.

The final post is the field guide to what goes wrong. We'll catalog the six mistakes that most often quietly wreck a random forest — and the specific fix for each — so your reliable default stays reliable when it actually matters.