Probability for ML

This cover narrows the whole field of probabilistic reasoning into one recurring question: given what I just observed, what is now more or less likely? That is conditional probability, and it is the move that underlies spam filtering, medical diagnosis, recommendation, and nearly every inference a model makes. Bayes' rule is the machine that lets you flip the question around when the direction you have is not the direction you need.

The post is the mechanical heart of the day. We cover conditional, joint, and marginal probability, the meaning of independence, and the handful of distributions you will meet constantly. By the end the goal is that spam filters and medical tests look like the same gear turning, not separate topics.

Conditional probability is the act of rescaling your view of the world to only the cases where some condition holds. P(A given B) asks: within the slice of reality where B is true, how often is A also true? The formula divides the joint probability of A and B by the probability of B, which is exactly this rescaling — restrict to B, then renormalize.

This is the single most-used operation in machine learning inference. Almost every prediction is some form of P(label given features): given these pixels, how likely is each digit; given these words, how likely is spam. Recognizing prediction as conditional probability connects the abstract formula to the everyday act of classifying.

Joint and marginal probability are the two other moves that combine with conditioning. The joint P(A, B) is the probability that both happen together. The marginal P(A) is the probability of A on its own, recovered by summing the joint over every possible value of B — a process literally called marginalizing because you collapse one variable out.

These three operations — conditioning, joining, and marginalizing — are the grammar of probabilistic modeling. Complex models are built by combining them: you specify some joints, marginalize out what you do not observe, and condition on what you do. Mastering the three primitives means you can read and build almost any probabilistic model.

Bayes' rule is the day's centerpiece because it solves a problem that comes up constantly: you can measure one conditional but you need the other. It states that the posterior P(A given B) equals the likelihood P(B given A) times the prior P(A), divided by the evidence P(B). In words, it updates a belief you held before seeing data into a belief after seeing it.

The genius of Bayes is that it makes belief-updating mechanical. You start with a prior, you see evidence, and the rule tells you exactly how much to revise. This is not just a formula for tests and filters — it is the conceptual core of Bayesian machine learning, where every observation nudges the model's beliefs about its parameters.

This labeled diagram names the three ingredients of Bayes so they stop blurring together. The prior P(A) is what you believed before any evidence. The likelihood P(B given A) measures how well the evidence fits each hypothesis. The posterior P(A given B) is the updated belief that results from combining them. The evidence term in the denominator simply normalizes so the posterior is a valid distribution.

Keeping these straight is what prevents the errors in the final slide. People most often forget the prior, treating the likelihood as if it were the answer. The diagram is a reminder that the likelihood is only half the story — without the prior, you cannot get to the posterior you actually want.

The worked medical-test example is the classic demonstration of Bayes in action, and the numbers are deliberately surprising. A disease affects 1% of people, the test catches 99% of true cases, and it has a 5% false-positive rate. The instinct is that a positive test means you almost certainly have the disease. Bayes says otherwise.

The code computes the total probability of a positive result by combining true positives and false positives, then divides the true-positive contribution by that total. The answer, about 0.167, means a positive test gives only a 17% chance of actually being sick. The reason is the next slide: the disease is rare, so most positives come from the large healthy population.

The base-rate surprise is one of the most important lessons in all of applied probability. Even a highly accurate test produces mostly false alarms when the condition is rare, because the prior — the base rate — dominates. With 99 healthy people for every sick one, the 5% false-positive rate generates a flood of false positives that swamps the true ones.

Machine learning inherits this directly from imbalanced data. Fraud, disease, and defects are all rare, so a model with great per-signal accuracy can still produce alerts that are mostly wrong. Ignoring the base rate when interpreting model outputs leads to wildly miscalibrated expectations and badly chosen decision thresholds.

Independence is the assumption that lets probability stay computationally tractable. Two events are independent when knowing one tells you nothing about the other, formally P(A given B) equals P(A). When events are independent, their joint probability is just the product of the individual probabilities, so you can multiply instead of tracking a full joint table that grows exponentially.

This is exactly the assumption that makes Naive Bayes both naive and fast. It pretends all features are independent given the class, which is usually false but computationally cheap and often good enough for ranking. Knowing when independence holds — and when assuming it is a convenient lie — is key to choosing and trusting models.

This comparison separates the two great families of distributions. Discrete distributions, described by a probability mass function, assign probability to countable individual outcomes — the chance of exactly three heads, or exactly five arrivals. Bernoulli, Binomial, and Poisson live here. Continuous distributions, described by a probability density function, deal in ranges; the probability of any single exact value is zero, and you integrate the density over a range to get a probability.

The distinction matters in code and in interpretation. For discrete variables you can read a probability straight off the mass function; for continuous ones you must integrate, and the density itself can exceed one. Confusing the two leads directly to the 'density greater than one' panic addressed in the mistakes post.

This mind map collects the distributions you will actually encounter so they are not intimidating when they appear. Bernoulli models a single yes/no trial. Binomial counts successes across many such trials. Gaussian, the bell curve, models continuous quantities clustered around a mean and shows up everywhere thanks to the central limit theorem. Categorical generalizes Bernoulli to many classes — exactly what a softmax output is.

Four distributions cover a remarkable amount of practical ML. A classifier output is categorical, a binary label is Bernoulli, measurement noise is often Gaussian, and counts of events are binomial or Poisson. Anchoring each to a concrete ML use makes the names stick rather than floating as abstract definitions.

The closing mistake is the prosecutor's fallacy, the dangerous habit of treating P(B given A) as if it were P(A given B). P(positive given disease) and P(disease given positive) are entirely different quantities that can differ by an order of magnitude, as the worked example showed. Swapping them silently is one of the most consequential reasoning errors in applied probability.

Bayes' rule exists precisely to convert between the two, and the conversion requires the prior — the very term people most often omit. The discipline to always ask 'which conditional do I actually have, and do I have the prior to flip it' prevents a huge fraction of real-world probability mistakes, in ML and beyond.

This teaser hands off to the code post, where every concept from this post gets implemented and verified in NumPy and SciPy. You will sample from distributions, estimate probabilities by simulation, and code the very Bayes update worked through here.

The pairing is intentional: the mechanics post builds the mental model, and the code post lets you compute with it. Intuition you have actually run on a machine sticks far better than intuition you have only read, so the next post turns every idea here into something you can execute and tweak.