Random Forests

This is the engine-room post. The surprising claim a random forest makes is that adding randomness to your training process makes the final model more reliable, not less. That feels backwards until you see the mechanism. Three concepts carry the whole process: bootstrap sampling (bagging) to build varied trees, random feature selection to de-correlate them, and averaging to cancel their errors.

There's a bonus too — the out-of-bag samples give you a free, honest validation estimate. The payoff for understanding all this is that the code in the next post stops looking like magic: every parameter you set maps to one of the ideas developed here.

Step one is bootstrap sampling, the 'bagging' in random forests (bootstrap aggregating). For each tree you build a training set by drawing N rows at random from your N-row dataset with replacement. Because draws are with replacement, some rows show up multiple times in a given tree's sample and — it works out to roughly 37% — some rows never get drawn at all.

The effect is that every tree learns from a slightly different version of the data. That variation is the first ingredient of diversity: trees trained on different samples grow into different shapes and make different errors. Without this step, every tree would see the same data and you'd have hundreds of near-identical trees with nothing useful to average.

Step two is the feature-randomness trick, and it's what distinguishes a random forest from plain bagging. At every split, a tree is only allowed to consider a random subset of the features — commonly the square root of the total number for classification, or a third for regression. So even two trees trained on similar bootstrap samples will split on different features and diverge.

Why bother? Because if one feature is strongly predictive, plain bagged trees would all split on it first and end up highly correlated — and averaging correlated models barely reduces variance. Forcing each split to ignore most features breaks that lock-step behavior, producing genuinely de-correlated trees. This single idea is the reason forests outperform simple bagging.

This pipeline diagram lays the training procedure end to end. For each tree: take a bootstrap sample of rows, grow the tree while restricting each split to a random subset of features, and repeat the whole thing N times to build the forest. To predict, aggregate the trees by voting (classification) or averaging (regression).

Seeing it as a loop run N times is the right frame. Training a forest is just this single-tree recipe repeated independently many times — which is also why it parallelizes so well across CPU cores. The two randomization steps sit inside the loop; the aggregation step sits outside it, at prediction time.

Here is why averaging works mathematically. The variance of an average of many estimates shrinks as you add more estimates — but only to the extent that those estimates are independent. If the trees were identical, their average would equal any single tree and you'd gain nothing. The less correlated the trees, the more the averaging cancels their individual noise while preserving the shared signal.

This is precisely why the feature-randomness step matters so much. Bootstrap sampling alone leaves trees fairly correlated; the random-feature trick drives that correlation down, and lower correlation translates directly into a sharper drop in the forest's overall variance. Diversity isn't a nice-to-have — it's the lever that makes the whole method work.

This snippet builds a forest by hand to demystify it. The loop runs 300 times; each iteration draws a bootstrap sample of row indices with np.random.choice using replace=True, trains a single decision tree on that sample with max_features='sqrt' to enforce the feature-randomness, and stores it. That's the entire training procedure for a random forest, written explicitly.

The point of seeing it this way is that scikit-learn's RandomForestClassifier is just this, optimized and parallelized. There's no hidden sophistication — bootstrap the rows, restrict the features, grow a tree, repeat. Reading the library's fit as 'this loop, but fast' makes the next post far less mysterious.

This snippet completes the picture by showing aggregation. Each of the 300 trained trees predicts on the test set, giving a 300-by-n matrix of predictions. Taking the mode down the tree axis produces the majority-vote label for each example — the forest's final classification output.

For regression you'd swap the mode for a mean; that's the only change. Notice that aggregation happens entirely at prediction time and is trivially cheap. The expensive part is growing the trees, which is done once; after that, predicting is just polling them and summarizing, which is why forests serve predictions quickly even with hundreds of trees.

The out-of-bag (OOB) estimate is a clever free lunch. Recall that each tree's bootstrap sample misses about a third of the rows. For any given row, you can find the subset of trees that never saw it during training and let only those trees predict it. Aggregating those predictions across all rows gives an honest accuracy estimate on data each tree treated as unseen.

The beauty is that this is effectively built-in cross-validation at no extra cost — you don't need to carve out a separate validation set or run a separate CV loop. In scikit-learn you just pass oob_score=True and read oob_score_. It's especially handy when data is scarce and you'd rather not sacrifice rows to a holdout split.

This comparison names the bias-variance bargain a forest strikes. A single deep tree has low bias (it fits the training data well) but high variance (it's unstable and overfits). The forest of such trees keeps roughly the same low bias — averaging doesn't add bias — while dramatically reducing variance, so it generalizes far better and behaves stably across runs.

That asymmetry is the whole game. Forests attack the variance half of prediction error specifically, leaving bias largely untouched. It also explains a limitation: if your single trees are biased (for example, too shallow), the forest will be biased too — averaging biased trees gives a biased average. Forests fix variance, not bias.

This cycle diagram reframes prediction as the assembly of one answer. A new row arrives, every tree in the forest independently predicts on it, the predictions are tallied (voted for classification, averaged for regression), and a single output is produced.

The cyclic framing emphasizes that the same simple summarization runs for every prediction request. There's nothing adaptive or stateful here — each new input triggers the same poll-and-combine. That uniformity is what makes forest inference both easy to reason about and easy to parallelize: the trees never need to talk to each other.

These bullets are the entire engine in five lines: bootstrap the rows to get varied trees, restrict features per split to de-correlate them, average or vote to make variance collapse, note that bias stays low while variance drops, and remember the out-of-bag rows give you free validation.

If you can recite this list, you understand random forests more deeply than most people who use them daily. Every item here will reappear in the next post as a line of real scikit-learn code or a parameter you set — n_estimators, max_features, oob_score — now with the meaning attached rather than memorized.

This post traded intuition for mechanism: bagging, feature randomness, variance reduction through averaging, out-of-bag validation, and the bias-variance bargain. You now know not just what a forest does but why each piece is there and why injecting randomness produces a more dependable model.

The next post cashes all of this in. We'll build the full workflow in scikit-learn — split, fit, read the OOB score, evaluate properly, interpret importances, and tune the key knobs — and because you understand the engine, the code will read like narration of ideas you already hold.