Logistic Regression

Logistic regression is one of the most widely deployed models in all of machine learning, yet its name actively misleads newcomers. This cover post resets that confusion before we go anywhere else. The goal here is a clean mental model: it is a classifier that produces a probability, built on a linear core wrapped in a single squashing function.

We deliberately separate the four moving parts — the linear score, the sigmoid, the probability, and the threshold — so that every later post (why it matters, how it works, code, mistakes) has a stable vocabulary to build on. If you only remember one thing, remember that logistic regression turns a straight-line score into a probability.

The single most important correction is that logistic regression does not output a class — it outputs a probability. The pipeline is: take the features, compute a weighted sum z = w·x + b, then pass z through the sigmoid to get a number strictly between 0 and 1. That number is an estimate of P(y=1 | x), the probability the example is in the positive class.

This distinction matters because the probability carries more information than a hard label. A prediction of 0.51 and one of 0.99 both become 'class 1' under a 0.5 threshold, but they represent very different levels of confidence. Keeping the probability around lets you tune thresholds, rank cases, and reason about risk — all things a bare label throws away.

The 'regression' in the name is historical and technically accurate, even though it confuses everyone. Logistic regression really does perform regression — on the log-odds. The log-odds (or logit) of the positive class is a continuous, unbounded quantity, and the model fits a linear function to it. So under the hood it is genuinely regressing a continuous target.

The twist is in how you use the result. Because that continuous target maps directly to a class probability, the practical application is classification. So the honest one-liner is: the math regresses the log-odds, and the application classifies the example. Holding both ideas at once is what makes the name stop being annoying.

The sigmoid (logistic) function is the hinge of the whole method. Defined as σ(z) = 1 / (1 + e^(−z)), it takes any real number and maps it smoothly into the open interval (0, 1). As z grows large and positive, σ(z) approaches 1; as z grows large and negative, it approaches 0; and at z = 0 it equals exactly 0.5.

That shape is exactly what we need to convert an unbounded linear score into a probability. It is also smooth and differentiable everywhere, which is what lets gradient descent train the model. The S-curve's steepness near the middle and flatness at the extremes also encodes a sensible behavior: the model is most uncertain near the boundary and increasingly confident as the score moves away from it.

This pipeline diagram captures the entire forward pass in four boxes. Features go in, they are combined into a single linear score z, sigmoid squashes z into a probability, and that probability is the model's output. Everything else — thresholds, loss, training — wraps around this core.

The value of seeing it laid out this way is that it demystifies the model. There is no hidden complexity in the prediction step: it is one dot product and one function call. That simplicity is precisely why logistic regression is so fast and so easy to deploy, points we expand on in the next post.

Here the sigmoid stops being abstract. The code defines it in one line and evaluates it at three points to make the behavior concrete: σ(0) = 0.5, σ(2) ≈ 0.88, and σ(−2) ≈ 0.12. Notice the symmetry — σ(−z) = 1 − σ(z) — which is why the values at +2 and −2 sum to 1.

Running this yourself is worth thirty seconds. Plug in larger magnitudes like 6 or −6 and watch the output saturate toward 1 or 0. That saturation is both a feature (confident predictions) and, as we'll see in the 'how it works' post, the reason squared-error loss behaves badly and cross-entropy is preferred.

There is a clean division of labor between the model and you. The model's job ends when it outputs a probability. Turning that probability into a yes/no decision is a separate, human choice: pick a threshold, and call everything above it class 1.

The default threshold of 0.5 is a convention, not a law. If false negatives are costly — say, missing a disease — you lower the threshold so the model flags more positives. If false positives are costly — say, blocking legitimate transactions — you raise it. Crucially, changing the threshold does not retrain the model; it only reinterprets the same probabilities. Confusing the threshold with the model is a common beginner error we revisit in post 5.

The decision boundary is where the model is perfectly undecided — where P(y=1) = 0.5. Since sigmoid equals 0.5 exactly when its input is 0, the boundary is the set of points where z = w·x + b = 0. That equation describes a line in two dimensions, a plane in three, and a hyperplane in general.

The takeaway is that logistic regression's boundary is always linear in the input features. This is its core strength (simple, interpretable, fast) and its core limitation (it cannot natively separate classes that curve around each other). That single fact explains both why it's such a reliable baseline and why post 5 warns against expecting nonlinear magic from it.

This bar chart traces the sigmoid curve as a sequence of probability values. At z = −4 the output is near 2%, climbing through 27% at z = −1, hitting the exact 50% midpoint at z = 0, rising to 73% at z = 1, and saturating near 98% at z = 4.

Reading it left to right shows the characteristic S-shape: steep change in the middle where small shifts in z move the probability a lot, and flattening at the extremes where the model is already confident. This visual reinforces why the region near z = 0 — the decision boundary — is where predictions are most sensitive, and why well-separated examples get pushed toward the flat, confident ends.

Putting linear and logistic regression side by side clears up the family relationship. Both fit a linear combination of features. The difference is in the output and the loss. Linear regression predicts an unbounded continuous number and is trained with squared error. Logistic regression squashes that number through sigmoid into a (0,1) probability and is trained with cross-entropy.

Understanding that they share a linear core but differ in the 'last step' is genuinely useful: it's why so much intuition transfers between them, and why logistic regression slots naturally into the same generalized-linear-model framework. It also foreshadows post 3, where we explain exactly why you cannot just reuse squared error here.

These four lines are the whole concept compressed into a portable summary you can recite. Weighted sum of features gives a score; sigmoid turns the score into a probability; a threshold turns the probability into a label; and the resulting boundary is linear in the features.

If this list feels obvious by now, the post did its job. Each line maps to a slide above, and each will be unpacked mechanically in the 'how it works' post. Carry these four sentences forward and the code in post 4 will read like narration rather than incantation.

This cover and CTA bookend the concept post. We started by dismantling the misleading name and end with a clean, shared vocabulary: score, sigmoid, probability, threshold, boundary.

The next post shifts from 'what it is' to 'why you should care.' Logistic regression isn't just a teaching example — it's the default baseline that quietly runs an enormous amount of production ML, precisely because it's interpretable, fast, and gives trustworthy probabilities. That's the case we make next.