Random Forests

This closing post is the field guide to failure. Random forests are so forgiving that their real-world failures rarely live inside the model — they live in how you interpret it and what you feed it. We catalog the six mistakes that show up most often and give a concrete fix for each.

Think of it as a pre-flight checklist. The model itself is robust and well-understood after the previous posts; these are the human errors that turn a great default into a misleading one. Dodge them and the reliability we've praised throughout actually shows up in production.

Mistake one: reading feature importances as causation. A high importance score tells you the forest leaned on that feature to make predictions — it's a statement of association within this particular model, nothing more. It does not mean the feature causes the outcome, nor that intervening on it would change anything.

The correct framing for a stakeholder is 'the model uses this signal,' not 'this drives the result.' Confusing the two leads to bad decisions — for instance, acting on a feature that's merely a proxy for the real cause. Treat importances as a guide to what the model attends to, and reserve causal claims for actual causal analysis.

Mistake two: trusting the default importances when features are correlated. The standard impurity-based (Gini) importance has known biases: it inflates high-cardinality features (those with many distinct values) and splits credit unpredictably between correlated predictors, so two features carrying the same signal can each look weak or one can hog the credit.

The fix is permutation importance, ideally measured on held-out data. It works by shuffling one feature at a time and measuring how much performance drops — a direct, model-agnostic measure of how much the model actually relies on that feature. It's slower but far more trustworthy when the ranking will inform a real decision.

This snippet shows the fix in code. permutation_importance refits nothing — it takes the trained forest and the test set, shuffles each feature ten times, and records the average drop in score. Sorting by importances_mean gives a ranking measured on data the model didn't train on, sidestepping the impurity measure's bias toward high-cardinality and correlated features.

Use this whenever the importance ranking will drive a decision — pruning features, explaining the model to stakeholders, or prioritizing data collection. The impurity-based importances from the previous post are fine for a quick gut check; permutation importance is what you reach for when you need to defend the ranking.

Mistake three: data leakage before the split. If you fit any transform that learns from data — imputation with the column mean, target encoding of a categorical, feature scaling, even selecting features by their correlation with the target — on the full dataset before splitting, information from the test rows leaks into training. Your notebook scores look fantastic, then collapse on genuinely unseen data.

The rule is absolute: split first, then fit every learned transform on the training fold only and apply the fitted version to validation and test. Wrapping preprocessing and the forest in a scikit-learn Pipeline makes this leak-free behavior automatic, including inside cross-validation, which is the easiest place to get it wrong by hand.

This comparison dramatizes the accuracy trap that the next slides address. On the left, accuracy reports a glowing 97% and says 'ship it' — one aggregate number that hides everything. On the right, recall on the rare class reveals the model catches only 40% of the positives it was built to find.

Same model, same data, two completely different stories. The lesson is that a single aggregate metric can actively conceal poor performance on the class that matters most. Always look at the per-class, cost-aware view before declaring victory — the comforting number is often the misleading one, especially on imbalanced data.

Mistake four: using too few trees. The entire mechanism of a forest is variance reduction through averaging, and averaging needs numbers. With only ten or twenty trees, the forest hasn't averaged away enough noise: its predictions and its feature importances wobble noticeably between runs, and you lose the stability that's the whole point.

The fix is simply to use a few hundred trees. Because adding trees never hurts accuracy — it only costs training time and memory — there's no downside to being generous beyond compute. A common pattern is to raise n_estimators until the OOB score stops improving, then stop. Don't starve the forest of the diversity it needs to do its job.

Mistake five: ignoring class imbalance. When one class dominates — fraud, rare disease, defaults — a forest can achieve high accuracy by mostly predicting the majority class, while missing the rare class that's the entire reason you built the model. Accuracy looks great and the model is useless.

The fixes mirror those for any classifier: set class_weight='balanced' so errors on the rare class cost more, resample the data, or move the decision threshold. Critically, change how you evaluate too — judge with recall, precision, F1, and AUC, picking the metric whose failure mode is most expensive for your problem. Doing nothing is rarely acceptable when the minority class is the point.

This snippet shows the lowest-effort imbalance fix: class_weight='balanced'. It automatically reweights the trees' splitting criterion so that errors on the rare class count proportionally more, nudging the forest to actually learn the minority class instead of ignoring it. No resampling pipeline, no manual weight math.

It's not a silver bullet — you'll still want to evaluate with recall and AUC and possibly tune the threshold — but it's the right first move and costs one keyword argument. Combined with the honest, per-class metrics emphasized throughout this post, it addresses the most common imbalance failure cheaply.

Mistake six: expecting a forest to extrapolate. A regression forest predicts by averaging the training target values that fall into each leaf, so its output can never exceed the range of values it saw during training. Hand it an input beyond that range — a future date, a value larger than any in the data — and it simply returns the nearest known average, flatlining instead of continuing a trend.

The fix is to recognize the limitation, not to fight it. Trees interpolate within the training distribution; they do not extrapolate beyond it. For genuinely trending data — time series with growth, physical relationships that continue past observed ranges — use a model that can extrapolate, like a linear or additive model, or engineer features that bring the problem back inside the training range.

This bar chart makes the extrapolation limit concrete. For inputs the forest saw during training — x at 2, 5, and 8 — the prediction rises sensibly with the input. But at x=20, far beyond the largest training value, the prediction doesn't keep climbing; it flatlines at the highest value the forest ever learned, around the x=8 level.

The visual reinforces the core constraint: a forest's output is bounded by its training data because it averages stored target values. When you see your inputs drifting outside the training range — a common failure in production as the world changes — that flatlining is the warning sign, and it means you've outgrown what a forest can responsibly predict.

This checklist is the whole post distilled into a pre-flight routine: read importances as association rather than cause, switch to permutation importance when the ranking matters, split before any learned transform, use a few hundred trees rather than a dozen, handle class imbalance and judge on recall and AUC, and never expect extrapolation beyond the training range.

Run through these six before trusting any random forest result. None of them is about the algorithm's internals — they're about disciplined practice around it. That discipline is precisely what turns a strong default into a model you can stake a real decision on.

That closes Day 38. Across five posts we've defined the random forest honestly as a committee of de-correlated trees, argued why it's the strong low-tuning default for tabular data, opened its engine of bagging and feature randomness, built it end to end in code, and catalogued the mistakes that quietly undermine it.

The meta-lesson is bigger than one model: reach for the robust, well-understood default first, understand exactly where it breaks — interpretation, leakage, imbalance, extrapolation — and escalate complexity only when the evidence demands it. The next topic continues that same approach in a new corner of machine learning — same depth, same no-fluff breakdown.