Probability for ML

This cover frames the entire day. The core reframe is that machine learning is not a fortune-telling machine that outputs a single guaranteed future — it is a machine that assigns likelihoods across many possible futures and then acts on those likelihoods. Probability is the mathematical language that makes this honest reasoning possible.

The goal of this first post is conceptual, not computational. Before anyone can appreciate Bayes' rule, likelihood, or cross-entropy, they need a clean mental picture of what a probability is and why uncertainty is the natural state of a learning system. We build that picture here so the rest of the day has solid ground to stand on.

Probability as a number on the closed interval from 0 to 1 is the foundational abstraction. Zero means the event cannot happen and one means it is guaranteed; the entire interesting world of machine learning lives strictly between those endpoints, in the realm of partial belief. A model that only ever output 0 or 1 would be claiming perfect certainty about an uncertain world, which is almost never warranted.

The practical takeaway is that a good model should rarely be fully confident. When you see a model emit a 0.5, that is not a failure — it is the model correctly admitting it does not know. Treating every prediction as if it must collapse to a definite yes or no throws away exactly the information probability was designed to preserve.

A random variable is the bridge between messy real-world outcomes and the numbers that math and code can manipulate. By mapping outcomes to numbers — heads to 1, a die face to its value, a class to an index — we turn unpredictable events into quantities we can average, compare, and feed into models. The 'random' part captures the fact that we do not know in advance which value will appear.

In ML this is everywhere even when it is invisible. The class a classifier predicts, the next token a language model generates, the price a regressor estimates: each is a random variable whose distribution the model is trying to learn. Recognizing the output as a random variable, not a fixed answer, is what lets you reason about confidence and error later.

Sample space and events give probability its structure. The sample space is the exhaustive list of everything that could happen, and an event is simply any subset of that space you happen to care about. Probability is then mass — a fixed total amount of belief, equal to one — spread across the sample space, and the probability of an event is how much of that mass falls inside it.

This 'mass over a space' intuition pays off later. When you marginalize, you are summing mass over the dimensions you no longer care about. When you condition, you are zooming into a slice of the space and re-normalizing the mass inside it. Getting comfortable with the picture now makes those operations feel mechanical rather than mysterious.

The three Kolmogorov axioms are the entire rulebook, and they are refreshingly short. First, no probability is negative. Second, the probability of the whole sample space is exactly one — something in the space must happen. Third, for events that cannot co-occur (disjoint events), the probability of their union is the sum of their individual probabilities.

Everything else in probability theory is derived from these three statements. Bayes' rule, expectation, variance, the chain rule — all of it follows. The reason this matters for ML is that any quantity claiming to be a probability must obey these axioms; if a model's outputs do not sum to one or go negative, they are not probabilities, and you cannot reason about them as such.

Probability and statistics are often blurred together, but they run in opposite directions, and keeping them distinct prevents real confusion. Probability is forward reasoning: you assume a known model — say, a fair coin — and you deduce what data it will produce. Statistics is inverse reasoning: you observe data and infer what model could have generated it, such as asking whether the coin is actually fair.

Machine learning sits squarely at this intersection. Training is fundamentally a statistical act — you have data and you are inferring model parameters. But once trained, the model is used probabilistically — given parameters, predict outcomes. Knowing which direction you are reasoning in tells you whether you are estimating or predicting, and that distinction shapes how you evaluate results.

This slide delivers the punchline that ties probability directly to everyday ML. A classifier does not natively output a label; it outputs a probability distribution over labels, and the label you see is just the argmax of that distribution. The softmax layer exists precisely to convert raw scores into a valid probability distribution that sums to one.

Keeping the full distribution rather than only the top label is enormously valuable. It lets you measure confidence, decide when to defer to a human, combine multiple models sensibly, and calibrate the system. Throwing away everything but the argmax is like keeping only the winner of a vote and discarding the margin — you lose the information that tells you how much to trust the result.

The code makes the softmax abstraction concrete and verifiable. Raw model outputs, called logits, are unbounded real numbers with no probabilistic meaning. Exponentiating them makes them positive, and dividing by their sum normalizes them so the result is non-negative and sums to one — satisfying the axioms from earlier and producing a legitimate probability distribution.

The printed check that the probabilities sum to 1.0 is not a formality; it is the proof that softmax produces something you are allowed to call a probability. In practice you will almost never implement softmax by hand — frameworks do it — but seeing it once removes the mystery and grounds the claim that 'model outputs are probabilities.'

The frequentist versus Bayesian split is worth knowing because the two camps define probability differently, and ML borrows from both without apology. The frequentist sees probability as the long-run frequency of an event over many repetitions — the coin lands heads 50% of the time across infinite flips. The Bayesian sees probability as a degree of belief that can be updated as evidence arrives.

Neither view is 'correct'; they are tools suited to different problems. Maximum-likelihood training has a frequentist flavor, while Bayesian neural networks and probabilistic priors lean the other way. You do not need to pick a side. You only need to recognize which interpretation a given method assumes so you read its outputs correctly.

This mind map is a roadmap for the rest of the day and beyond. Probability surfaces in three big places in ML: in model outputs (softmax distributions and confidence scores), in training (likelihood and cross-entropy loss), and in entire model families that are probabilistic by design (Naive Bayes, Gaussian mixture models, Bayesian methods).

Seeing these grouped makes the field feel smaller. Rather than memorizing dozens of disconnected techniques, you can anchor each one to the same underlying idea: reasoning about uncertain quantities with the rules of probability. Each later topic becomes a specialization of the foundation laid in this post.

The closing mistake addresses the most common misreading of a probabilistic output. A prediction of 70% does not mean 'almost certainly right.' It means that if you collected all the cases the model labeled 70%, roughly 30% of them should turn out wrong. That spread is not a defect — it is the literal meaning of the number.

This connects directly to calibration, which the next post explores. A model that is never wrong on its 70% predictions is not impressive; it is miscalibrated, because its stated confidence does not match observed reality. Internalizing that a probability is a frequency-of-being-right, not a vibe, changes how you read every model output you will ever see.

This teaser points to the next post, which answers the 'so what' of probability in ML. We have established what probability is; next we make the case for why it is load-bearing in every loss function, every confidence estimate, and every decision a model-driven system makes.

Following along in order matters here, because the why post motivates the mechanics post after it. Without the motivation, the formulas feel arbitrary; with it, they feel inevitable.