Logistic Regression

This is the engine-room post. Linear regression has a tidy closed-form solution; logistic regression does not, because the sigmoid makes the equations nonlinear in the weights. So instead of solving in one shot, we train iteratively. Three concepts carry the whole process: modeling the log-odds linearly, scoring error with cross-entropy, and minimizing it with gradient descent.

The payoff for understanding this is that the code in the next post stops looking like magic incantations. Every line — fit, the loss it minimizes, the convergence warning — maps to one of the ideas we develop here.

Start with what the linear part actually represents. z = w·x + b is not the probability; it is the log-odds of the positive class. Odds are P/(1−P), a ratio that runs from 0 to infinity, and taking the log stretches that to the full real line from −∞ to +∞.

That range is the key insight. A linear function can output any real number, so it can't directly produce a probability bounded in (0,1) — but it can perfectly well produce a log-odds. Sigmoid is precisely the inverse transformation that maps the log-odds back to a probability. So the model fits a line in log-odds space, and sigmoid translates that line into the S-curve we see in probability space.

This flow diagram lays the training intuition end to end in four nodes. We compute the linear score z, squash it to a probability p with sigmoid, measure how wrong p is with log loss, and take a gradient step to nudge the weights and bias. Then we repeat.

Seeing it as a loop is the right frame: training is just this cycle run thousands of times, each pass shrinking the loss a little. The diagram also previews the two pieces we'll dwell on — the loss function (why log loss, not squared error) and the gradient step (why it's so clean).

Here's a subtle but important point: you cannot simply reuse linear regression's squared-error loss. If you do, plugging the sigmoid inside squared error produces a non-convex loss surface riddled with local minima, so gradient descent can stall in a bad spot depending on where it started.

There's a second problem. Squared error produces vanishing gradients exactly when the model is confidently wrong — the sigmoid's flat tails kill the gradient signal — so the model learns painfully slowly from its worst mistakes. Both problems point to needing a loss tailored to probabilistic classification, which is cross-entropy.

Cross-entropy, also called log loss, is the right tool. For a single example its cost is −log(p) when the true label is 1, and −log(1−p) when the true label is 0. The combined form is −[y·log(p) + (1−y)·log(1−p)], averaged over the dataset.

The behavior is exactly what we want. Predicting close to the truth costs almost nothing; predicting confidently wrong — say 0.99 when the truth is 0 — costs an enormous amount, because −log of a tiny number blows up. This steep penalty on confident errors is what pressures the model toward honest, well-calibrated probabilities, and it pairs with sigmoid to give a convex surface.

This snippet implements log loss directly so you can feel its asymmetry. The np.clip guards against log(0), which would be infinite — a necessary numerical safety net. Then it computes the two cases shown: a true label of 1 predicted as 0.99 costs about 0.01, while the same true label predicted as 0.01 costs about 4.6.

That 460x difference in cost for the same true label is the whole point. The loss barely reacts to a confident correct prediction and reacts violently to a confident wrong one. Run it with predictions at 0.5 to see the 'maximally uncertain' cost of about 0.69 (= −log 0.5), the value you'd get from random guessing.

Now the elegant part. When you differentiate log loss with respect to the weights, with sigmoid as the activation, the algebra collapses dramatically. All the exponential and logarithm terms cancel, and the gradient for each weight reduces to the average over examples of (prediction − truth) × feature value.

This is not a coincidence — sigmoid and cross-entropy are a matched pair chosen precisely so this cancellation happens. The practical consequence is that each training step is cheap: compute predictions, subtract the labels, multiply by features, average. That simplicity is a big reason logistic regression trains so quickly even on large datasets.

This snippet is one full gradient descent step, the inner loop of training. It computes predictions p with the vectorized sigmoid, forms the error vector (p − y), and updates the weights by the average of X transposed times that error, scaled by the learning rate. The bias updates by the mean error.

Notice there is no autodiff and no library magic here — it's the clean gradient from the previous slide, written out. Loop this a few hundred times over your data and you have trained a logistic regression from scratch. Reading scikit-learn's fit as 'this, but optimized and with regularization' makes the next post far less mysterious.

This cycle diagram reframes training as a repeating four-step loop: predict the probability, compute the loss, derive the gradient (p−y)·x, and update the weights against it. Each lap reduces the loss slightly, and after enough laps the weights settle at the optimum.

The cyclic framing matters because it's literally what iterative optimization is — there is no single solve step, just disciplined repetition. It also makes hyperparameters like learning rate and number of iterations concrete: they control how big each step is and how many laps you run before stopping.

Why does this iterative process reliably reach the best answer? Because cross-entropy composed with sigmoid is convex in the weights. A convex loss surface is shaped like a single bowl with one lowest point and no other dips to get stuck in.

This is a genuinely valuable property. Unlike training a neural network — where you fight local minima, saddle points, and sensitivity to initialization — logistic regression converges to the one global optimum from any reasonable start. That's the mathematical source of the reliability and reproducibility we praised back in post 2: same data in, same model out, every time.

These bullets are the entire engine in five lines: the linear score models the log-odds, sigmoid maps it to a probability, log loss measures the error, the gradient (p−y)·x updates the weights, and convexity guarantees you land at the single global optimum.

If you can recite this list, you understand logistic regression more deeply than most people who use it daily. Every word here will reappear in the next post as a line of real scikit-learn code or a parameter you set — now with the meaning attached rather than memorized.

This post traded intuition for mechanism: log-odds, cross-entropy, gradient descent, convexity. You now know not just what logistic regression does but why each piece is there and why it trains so dependably.

The next post cashes all of this in. We'll build the full workflow in scikit-learn — split, scale, fit, predict probabilities, evaluate properly, and interpret coefficients — and because you understand the engine, the code will read like narration of ideas you already hold.