Linear Regression

This is the failure-mode post, and its framing is deliberately stark: linear regression rarely crashes, it lies. The danger isn't an error message; it's a model that confidently produces wrong numbers while looking perfectly healthy in the notebook. Cataloging these traps is what turns someone who can run fit() into someone who can be trusted to ship a regression.

The cover sets that tone. Every trap in this post is silent — the code runs, the metrics look fine, and the problem only surfaces when reality diverges from your assumptions. Awareness is the only defense.

Forcing a line through a fundamentally curved relationship is the most common and most fundamental mistake, because it violates the model's core assumption. Diminishing returns, saturation thresholds, and U-shaped relationships all break linearity, and a straight line will systematically misfit them — too low here, too high there, in a predictable pattern.

The fix is not to abandon regression but to detect the problem (by inspecting residuals, covered next) and then extend the model with polynomial or interaction terms, which keep the linear machinery while capturing curvature. The lesson is that linear regression can model nonlinear relationships if you engineer the right features — but only if you first notice the line isn't fitting.

The residual plot is the single most important diagnostic in regression, which is why it gets a comparison diagram. When the fit is good, residuals scatter randomly around zero with no discernible shape — the model has captured the signal and only noise remains. When the fit is bad, the residuals show structure: a curve, a fan shape, or a trend, revealing that systematic information was left on the table.

This contrast is the practical skill the post most wants to instill. R² and other summary numbers can hide problems that a residual plot makes obvious in seconds. Teaching readers to always plot residuals — and to know what good versus bad looks like — is worth more than any single metric.

Multicollinearity is a subtle trap because it doesn't necessarily hurt predictions, but it destroys interpretability — and interpretability is the main reason to use linear regression in the first place. When two features are highly correlated, like square footage and room count, the model can't decide which deserves credit, so it splits the effect arbitrarily. The coefficients become unstable, sometimes flipping sign on a tiny data change.

The insidious part is that the model's predictions can still be fine, so nothing looks wrong on the surface. But any conclusion you draw from an individual coefficient is now unreliable. For a model whose whole value proposition is explainability, that's a serious failure that a naive user never notices.

Detecting collinearity early is cheap, so this slide makes it concrete. A correlation matrix of the features instantly flags pairs that move together; any off-diagonal value above roughly 0.8 is a warning sign. The remedies are practical: drop one of the correlated features, combine them into a single engineered feature, or switch to ridge regression, which adds a penalty that stabilizes the coefficients.

Mentioning ridge regression plants a seed for regularization, a topic the series will return to. For now, the actionable takeaway is the habit: glance at the correlation matrix before trusting any individual coefficient.

Confusing correlation with causation is the most consequential interpretive error, and it has caused real-world disasters in policy and business. A coefficient tells you that a feature moves together with the target in your data — nothing more. It does not promise that intervening on that feature will change the outcome.

The ice cream and drownings example makes the point unforgettable: both rise in summer heat, so ice cream sales 'predict' drownings, but banning ice cream won't save anyone. Treating regression weights as causal levers without a randomized experiment or careful causal design is a trap that even experienced analysts fall into, often with expensive consequences.

Outliers deserve special attention precisely because of the squared-error cost from post 3. Because MSE squares residuals, a single extreme point contributes enormously to the total error, and the optimizer will tilt the entire line toward it to reduce that one giant squared term. The result is distorted coefficients driven by one anomalous observation.

The practical danger is mundane: a data-entry typo, like a forty-million-dollar house, can wreck a model trained on normal homes. The defenses are to always plot the data, investigate any extreme points (is it a real outlier or an error?), and consider robust regression methods that down-weight outliers. This trap connects directly back to why we square errors — the strength becomes a vulnerability.

The vector diagram visualizes how a single outlier pulls the fitted line away from the true trend. The green line represents the relationship the bulk of the data supports; the red line shows how the fit tilts when one extreme point drags it. Seeing the two lines diverge makes the abstract warning concrete.

This picture reinforces the previous slide's lesson geometrically. Some readers will remember the image of a line being yanked off course by one rogue point long after they've forgotten the explanation of squared errors.

Extrapolation is a quieter trap that even careful modelers commit. A model fit on houses between 800 and 3000 square feet has learned the relationship only within that range. Asked about a 10,000 square foot mansion, it will dutifully extend the line and return a number — but that number is a pure guess, because nothing in the training data supports the relationship out there.

The relationship may change entirely outside the observed range: it could plateau, accelerate, or reverse. Predictions beyond the training distribution are guesses dressed as math, and they should either be avoided or flagged loudly. This is especially dangerous in time-series forecasting, where the future is by definition outside the observed range.

Over-trusting R² is the trap that closes the loop, because R² is the number beginners cling to. A high R² does not certify a good model. It can be high while the residuals plainly show nonlinearity, meaning the model is wrong in a structured way. Worse, R² mechanically increases every time you add a feature — even a column of random noise will nudge it up — so it rewards complexity for its own sake.

The defenses are concrete: use adjusted R², which penalizes useless features; always plot residuals; and judge performance on held-out test data rather than the training set. The recurring theme of this post is that no single summary number tells the whole story, and R² is the most over-trusted of them all.

The honest-evaluation code ties the post's lessons into a runnable routine. Predicting on the test set, plotting residuals against predictions, and printing the test R² combines the two most important checks: the residual plot (to catch structure and nonlinearity) and the held-out score (to catch overfitting). The instruction to look for 'a shapeless cloud, not a pattern' restates the residual-plot lesson as an actionable visual target.

Making this a copy-paste snippet means readers can drop it into any regression project. It operationalizes the whole post — instead of just warning about traps, it hands over the diagnostic code that detects them.

The final checklist distills every trap into a pre-trust ritual: plot residuals, check feature correlations, refuse to claim causation, hunt and explain outliers, never extrapolate blindly, and judge on test data rather than R² alone. Running through this list before trusting any linear model is the discipline that prevents shipping a confident lie.

Framing it as a 'pre-trust checklist' makes the point that trust must be earned through these checks, not assumed because the code ran. It's the screenshot-ready summary of the entire post.

The CTA closes both the post and the topic by pointing forward to logistic regression. The teaser highlights the elegant connection: logistic regression bends the same weighted-sum machinery to predict categories instead of numbers. This frames Day 36 as a natural extension rather than a fresh start, rewarding the reader's investment in understanding linear regression deeply.