Logistic Regression

This closing post is the field guide to failure. Logistic regression is so easy to run that most of its real-world failures happen around the model, not inside it — in how you prepare data, choose metrics, and read results. We catalog the six mistakes that show up most often and give a concrete fix for each.

Think of it as a pre-flight checklist. The model itself is trustworthy and well-understood after the previous posts; these are the human errors that turn a solid baseline into a misleading one. Dodge them and the reliability we've praised throughout actually shows up in production.

Mistake one: skipping feature scaling. Logistic regression sums weighted features and, in scikit-learn, applies L2 regularization by default. When features live on wildly different scales — income in thousands, a ratio in fractions — the penalty hits them unequally and the optimizer converges slowly or not at all, throwing the convergence warning you've probably seen.

The fix is to standardize before fitting, ideally inside a Pipeline so it's automatic and leak-free. This single habit eliminates a surprising fraction of 'my logistic regression won't converge' and 'my coefficients make no sense' problems in one move.

Mistake two: ignoring class imbalance. When one class dominates — fraud, rare disease, defaults — the loss is overwhelmingly about the majority, and the model can minimize it by essentially always predicting the majority class. You get high accuracy and zero useful detection of the thing you actually care about.

The fixes are to reweight the loss with class_weight='balanced', resample the data (oversample the minority or undersample the majority), or move the decision threshold so the rare class gets flagged. Which you choose depends on the problem, but doing nothing is rarely acceptable when the minority class is the point.

This snippet shows the lowest-effort fix for imbalance: class_weight='balanced'. It automatically reweights the loss so errors on the rare class cost proportionally more, pushing the model to actually learn it instead of ignoring it. No resampling pipeline, no manual weight math.

It's not a silver bullet — you'll still want to evaluate with recall and precision and possibly tune the threshold — but it's the right first move and costs one keyword argument. Combine it with the honest metrics from the next slide and you've addressed the most common imbalance failure cheaply.

Mistake three: trusting accuracy on skewed data. Accuracy is a single number that averages over all classes, so it happily reports 95% for a fraud model that catches no fraud. It hides exactly the failure you most need to see.

The fix is to use metrics that expose per-class behavior and match your error costs: precision (of the cases I flagged, how many were right), recall (of the real positives, how many did I catch), F1 (their balance), and ROC-AUC (ranking quality). Pick the one whose failure mode is most expensive for your problem and optimize for that, not for accuracy.

This comparison dramatizes the accuracy trap. On the left, accuracy reports a glowing 95% and says 'ship it' — one number that hides everything. On the right, recall on the fraud class reports 0%: the model catches nothing it was built to catch.

Same model, same data, two completely different stories. The lesson is that a single aggregate metric can actively conceal total failure on the class that matters. Always look at the per-class, cost-aware view before declaring victory — the comforting number is often the misleading one.

Mistake four: reading coefficients as causation. A large coefficient means a feature is predictive of the outcome given the other features in the model — a statement about association within this particular fit, nothing more. It does not mean the feature causes the outcome.

The danger sharpens with correlated features. Two collinear predictors can split the credit between them arbitrarily, swap which one looks important, or even flip signs when you add or remove a variable. Treat coefficients as evidence of association inside this model, investigate collinearity, and never present them as causal proof to a stakeholder.

Mistake five: expecting nonlinear magic. Logistic regression draws exactly one straight decision boundary in feature space. If the classes are arranged so that no straight line separates them — concentric rings, an XOR pattern — the model simply cannot do it, and no amount of extra iterations or data will change that.

The fix is not to train harder; it's to change the inputs or the model. Engineer features — interaction terms, polynomial features, domain transforms — that make the problem linearly separable, or switch to a model that learns nonlinear boundaries natively, like a tree ensemble or a neural network. Recognizing this limit is what tells you when to escalate.

This vector diagram contrasts the two situations. One region is cleanly separable by a straight line — logistic regression handles it perfectly. The other curves in a way that no single line can split, marking it as needing a curved boundary the base model can't produce.

The visual reinforces the core limitation from post 1: the boundary is always linear in the features you give it. So when you spot curvature in your data, your move is to add the features that straighten it out — or to choose a different tool — rather than blaming the optimizer.

Mistake six: data leakage through preprocessing. If you fit your scaler — or any transform that learns from data — on the entire dataset before splitting, the test set's statistics leak into training. Your validation scores look fantastic and then collapse on genuinely unseen production data.

The rule is absolute: fit every transform on the training fold only, then apply the already-fitted transform to validation and test. This is easy to get wrong by hand, especially with cross-validation, which is exactly why the next slide's Pipeline pattern exists — it makes leak-free preprocessing the default rather than something you have to remember.

This snippet is the durable fix for the leakage trap: wrap the scaler and the model in a single Pipeline. When you call pipe.fit on the training data, the scaler fits on that data only, and the same fitted scaler is applied automatically at predict time. There's no separate test-transform step to get wrong.

The bonus is that the Pipeline behaves correctly inside cross_val_score: on each fold, the scaler re-fits on just that fold's training portion, so there's no leakage even across folds. Making the Pipeline your default unit of work eliminates an entire category of subtle, score-inflating bugs.

This checklist is the whole post distilled into a pre-flight routine: scale inside a Pipeline, handle class imbalance, choose a metric that matches your error costs, read coefficients as association rather than cause, and respond to nonlinearity by engineering features or switching models.

Run through these five before trusting any logistic regression result. None of them is about the algorithm's internals — they're about disciplined practice around it. That discipline is precisely what turns a quick baseline into a model you can stake a real decision on.

That closes Day 36. Across five posts we've defined logistic regression honestly, argued why it remains the default baseline in serious production ML, opened its engine of log-odds and cross-entropy, built it end to end in code, and catalogued the mistakes that quietly undermine it.

The meta-lesson is bigger than one model: master the simple, interpretable, reliable tool first, understand exactly where it breaks, and escalate complexity only when the evidence demands it. The next topic continues that same approach in a new corner of machine learning — same depth, same no-fluff breakdown.