Probability for ML

This cover sets the tone for a hands-on post. Probability becomes intuitive the moment you stop reading about it and start sampling from it, watching real numbers behave the way the theory predicts. Every snippet in this post is meant to be run, because the gap between understanding probability and feeling it closes only when you compute.

We use NumPy and SciPy throughout, the same tools you reach for in real ML work, so nothing here is a toy you will abandon later. By the end you will have sampled distributions, estimated probabilities by simulation, computed likelihoods, and implemented Bayes' rule — the practical toolkit of applied probability.

This first snippet shows how to summon randomness in a controlled, reproducible way. We create a seeded random generator and draw from three fundamental distributions: Bernoulli flips via binomial with n=1, uniform die rolls via integers, and Gaussian samples via normal. Each line corresponds to a distribution discussed conceptually in the earlier posts, now made concrete.

The seed is the key detail. Calling default_rng(42) makes the 'random' results identical every run, which is essential for debugging and reproducible experiments. In real ML pipelines, seeding your RNG is the difference between a result you can reproduce and a bug you can never catch again. Get in the habit now.

Monte Carlo estimation is the brute-force superpower of applied probability. Instead of deriving P(sum equals seven) analytically, we simulate a million pairs of dice rolls and simply count the fraction that sum to seven. The estimate, about 0.1667, matches the exact answer of six in thirty-six, demonstrating that simulation recovers theory.

This technique generalizes to any probability you can phrase as 'the fraction of trials where something happens,' including problems with no clean closed-form solution. It is the engine behind Bayesian inference via sampling, reinforcement-learning rollouts, and financial risk modeling. The price is computation: more samples give a tighter estimate but take longer, a trade-off you tune per problem.

Here we compute distributions directly rather than by simulation, using SciPy's stats module. The binomial PMF gives the exact probability of every possible number of heads in ten flips of a fair coin, and printing the array shows the familiar symmetric peak at five. Then we evaluate the Gaussian PDF at its center and get approximately 0.399, the well-known peak density of the standard normal.

The contrast with the previous slide is the point: simulation approximates, while these functions give exact values when a formula exists. SciPy's stats objects expose pmf, pdf, cdf, and sampling for dozens of distributions through a uniform interface, so once you know the pattern you can work with any of them. This is the everyday way to compute probabilities in real code.

This trace contrasts the simulated and exact answers side by side to drive home convergence. The Monte Carlo estimate of P(seven) reads 0.1667 and the exact value reads 0.1667 — they agree because a million samples is plenty for this simple problem. The comment notes the general law: simulation converges to truth as the number of samples grows.

Seeing the two numbers match builds trust in simulation as a legitimate tool, not a hack. When a problem has a known answer, you validate your simulator against it; once you trust the simulator, you point it at problems that have no closed-form answer and believe the results. This validate-then-extend workflow is how practitioners use Monte Carlo responsibly.

Likelihood is the quantity that connects data back to model parameters, and this snippet makes it concrete. Given five observations, we ask how probable that exact data is under two candidate Gaussian models — one centered at 2.0 and one at 5.0. Because the data clusters around 2, the log-likelihood under mu=2 is far higher than under mu=5, telling us which model better explains what we saw.

This is the seed of maximum-likelihood estimation, the backbone of most model training. Fitting a model is essentially searching for the parameters that maximize the likelihood of the observed data. Note that we sum logpdf values rather than multiplying pdf values — working in log-space, the recurring theme that prevents numerical disaster in any real likelihood computation.

Here Bayes' rule from the previous post becomes an executable function. We define a tiny bayes function that takes a prior, a likelihood, and the evidence, and returns the posterior. Then we reproduce the medical-test example: a 1% prior, 99% sensitivity, 5% false-positive rate, yielding the same 0.167 posterior computed by hand earlier.

Turning the formula into code does two things. It verifies the hand calculation, and it gives you a reusable building block you can apply to any prior-likelihood-evidence problem. The exercise also exposes the structure clearly: the evidence term in the denominator is just the total probability of the observation, assembled from the true-positive and false-positive paths.

This slide steps back to justify simulation as a first-class method, not a fallback. Many real distributions are too complicated to have a tidy formula — the posterior of a Bayesian model, the return distribution of an RL policy, the loss landscape of a complex system. For all of these, you cannot write down P; you can only sample and count.

Monte Carlo turns 'I cannot solve this integral' into 'I can estimate this fraction,' which is almost always achievable. Understanding that simulation is the universal escape hatch frees you from needing closed-form solutions for everything. It is why sampling-based methods dominate modern probabilistic ML, from variational inference to diffusion models.

The law of large numbers is the theoretical guarantee that makes simulation trustworthy, and this snippet lets you watch it operate. We estimate the probability of a biased coin (p=0.3) using 10, then 1000, then 100000 samples. The estimates tighten toward 0.3 as the sample count grows, exactly as the law promises.

Running this yourself builds an intuition for sample-size trade-offs that no proof can. With few samples the estimate is noisy and might be far off; with many it is reliable but slower to compute. Every Monte Carlo decision you make later — how many samples to draw — is a practical application of the convergence you can see happening in these three printed lines.

These gotchas are the practical wisdom that separates working code from subtly broken code. Always seed the RNG so results reproduce. Work in log-space to dodge the underflow covered in the next slide. Sum log-densities rather than multiplying raw densities, for the same reason. And remember that estimate quality scales with sample count, trading accuracy against runtime.

None of these is obvious from the theory, yet each one bites real practitioners constantly. They are the difference between probability code that runs and probability code that runs correctly. Keeping this short checklist in mind will save you from the most common implementation bugs in any probabilistic computation.

The closing mistake previews a trap that ruins naive probability code: multiplying many small probabilities together. Floating-point numbers cannot represent arbitrarily tiny values, so a product of hundreds of small probabilities underflows silently to exactly 0.0, destroying all information. The model then sees zero where it should see a very small positive number.

The universal fix, used throughout machine learning, is to sum the logarithms of the probabilities instead of multiplying the probabilities themselves. Addition in log-space is numerically stable, and you exponentiate only at the very end if you need a real probability back. This single habit prevents an entire class of bugs, which is why the mistakes post returns to it.

This teaser closes the code post and points toward the final post of the day, a field guide to the probability mistakes that quietly wreck real ML systems. Having built and run the tools, you are now equipped to recognize when they are being misused.

The progression is deliberate: concept, motivation, mechanics, code, and finally pitfalls. The mistakes post ties everything together by showing how each idea fails in practice, so you leave the day not just knowing probability but knowing how it goes wrong.