Decision Trees

This cover names the defining hazard of decision trees: by default they overfit, scoring perfectly on training data and failing in production. Framing it as expected behavior to fight, rather than an occasional bug, sets the right mindset for the whole post.

The 'common mistakes' angle works best as a failure manual — each slide is a specific, nameable error plus its fix. The cover lists the lineup so readers know they're getting six concrete traps, not vague cautions, which matches the depth standard of the series.

Unbounded growth is the number-one tree mistake, so it leads. The mechanism is worth stating precisely: with no stopping rule, the greedy recursion from post 3 continues until each leaf isolates a single training row, producing a model that has memorized noise rather than learned signal.

The fix is twofold and both halves matter: constrain the tree (max_depth, min_samples_leaf, or pruning) AND evaluate on held-out data. Many people do one without the other. Capping depth while still reporting training accuracy hides the very problem the cap is meant to solve.

The compare diagram quantifies the overfitting story with representative numbers so the abstract warning becomes concrete. The unbounded tree's perfect-or-near-perfect train score next to a much lower test score is the signature of memorization.

The depth-capped column shows the healthy pattern: train and test scores close together, train slightly higher. Teaching readers to read this gap as a diagnostic — large gap means overfit — gives them a tool they'll use on every model they build, not just trees.

Instability is a subtler failure than overfitting and often surprises people. Because the greedy algorithm commits to the single best split at the top, a small change in training data can flip that top split and cascade into a completely different tree. High variance like this makes a single tree an unreliable foundation for decisions.

The fix is the natural bridge to Day 38: ensembles. A random forest builds many trees on resampled data and averages them, canceling out the variance. Naming this here motivates the next topic while solving a real problem.

This code slide makes the fix tangible: swapping DecisionTreeClassifier for RandomForestClassifier is nearly a drop-in change, yet it directly addresses the instability just described. Showing n_estimators=300 conveys that a forest is literally many trees voting.

The comment 'variance averaged away' ties the code back to the concept. Readers leave with both the why (averaging cancels per-tree variance) and the how (one import, one constructor), which is the ideal pairing for a mistakes-and-fixes post.

Biased feature importances are a trap that fools even experienced practitioners because the default output looks authoritative. The mechanism: impurity-based importance rewards features that offer many possible split points, so high-cardinality columns (IDs, continuous values, timestamps) score high regardless of real predictive value.

The consequence is dangerous — you might drop a genuinely useful low-cardinality feature or trust a meaningless ID. The fix is permutation importance, which measures the actual drop in test performance when a feature's values are shuffled, reflecting real predictive contribution.

This code slide implements the permutation-importance fix on the test set, the correct place to measure real generalization impact. Running it with n_repeats=10 averages over several shuffles for a stable estimate, and sorting reveals the genuinely predictive features.

Pairing the warning slide with runnable code follows the post's pattern of never leaving a problem unsolved. The reader gets the diagnosis (default importances are biased) and the prescription (permutation importance on held-out data) in adjacent frames.

Target leakage is the most embarrassing failure because the model looks brilliant right up until production. The mechanism: a feature that secretly encodes the outcome — a closed_date when predicting account closure, a post-decision flag, an aggregate computed using the future — hands the tree the answer.

Trees are especially good at exploiting leaks because a single split on the leaky feature yields near-perfect purity. The fix is process, not code: audit every feature against the question 'would this value genuinely be available at prediction time?' and remove anything that fails.

The extrapolation limit is a structural property people forget. A regression tree outputs the mean of the training rows in each leaf, so its predictions are bounded by the range it saw in training — it physically cannot output a value above the highest or below the lowest training target.

This matters for any rising trend: forecasting demand, prices, or growth beyond historical range. A tree will flatline at the boundary leaf's average while reality keeps climbing. When extrapolation is required, a linear or other parametric model is the safer tool — knowing this limit is what separates correct tool selection from cargo-culting trees onto every problem.

The flow diagram makes target leakage's deceptive trajectory vivid: a sneaky feature leads to a great-looking test score that collapses the moment the feature is unavailable in production. Seeing it as a sequence helps readers recognize the pattern in their own projects.

The 'looks amazing' to 'fails live' progression is the emotional core of the leakage lesson. An unrealistically high score should trigger suspicion, not celebration — that instinct is the practical takeaway the diagram is meant to install.

Class imbalance closes the mistakes list because it's pervasive in real problems — fraud, disease, churn are all rare-positive tasks. The mechanism: on a 99%-negative dataset a tree can score 99% accuracy by always predicting the majority and learning essentially nothing useful about the minority.

The fix combines modeling and evaluation. class_weight='balanced' or resampling pushes the splits to care about the rare class, and judging with precision, recall, and F1 (not accuracy) exposes whether the model actually detects positives. This loops back to the classification_report habit modeled in post 4.

The recap consolidates all six fixes into a single screenshot-able checklist, turning the post into a pre-flight checic the reader can run before trusting any tree. Each bullet maps to a slide, condensing the failure manual into actionable rules.

Ordering them roughly by frequency — overfitting and instability first, then the subtler leakage and imbalance traps — helps readers prioritize. The list is intentionally portable: most of these (held-out evaluation, leakage audits, the right metrics) apply to every model, not just trees.

The CTA closes both this post and the day, pointing forward to Random Forests as the natural sequel. The framing is deliberate: post 5 catalogued the weaknesses of a single tree, and a forest is precisely the crowd-of-trees remedy for most of them — especially overfitting and instability.

This creates a clean narrative arc across the day and a strong hook into Day 38. The reader finishes understanding not just trees but why the next topic exists, which is the ideal state to carry into the following day's content.