Probability for ML

This cover frames the final post as a field guide to failure. The crucial insight is that most probability bugs in ML are not arithmetic errors — the math runs perfectly and faithfully executes flawed reasoning. That is what makes them so dangerous: the code does not crash, the model still trains, and the problem only surfaces later as overconfident predictions or expensive bad decisions.

This post catalogs the traps that the previous four posts have hinted at, collected in one place so you can recognize them before they ship. Each mistake is subtle, common, and avoidable once you know its shape, which is exactly why a dedicated tour of the failure modes is worth your time.

Flipping the conditional is the most frequent and most damaging probability error. P(test positive given sick) and P(sick given test positive) are different quantities that the worked medical example showed can differ by nearly a factor of six. Confusing the direction you have with the direction you need produces conclusions that are not slightly off but qualitatively wrong.

The defense is a reflex: every time you have a conditional probability, pause and ask which way it points. Do you have the probability of the evidence given the hypothesis, or the hypothesis given the evidence? Bayes' rule converts between them, but only if you notice the mismatch in the first place. This single habit prevents a large share of real-world reasoning failures.

Ignoring base rates is the partner of the flipped-conditional error and just as costly. When a class is rare, even a strong predictive signal yields a low posterior probability, because the prior dominates. A fraud detector with 90% precision on its signal still drowns in false alarms when fraud is only 0.1% of transactions, simply because there are so many more legitimate cases to misflag.

Machine learning inherits this from imbalanced datasets, which are the norm in fraud, disease, and defect detection. If you set decision thresholds or interpret model confidence without folding in the base rate, your expectations and your operating points will be wildly miscalibrated. The base rate is not an optional refinement — it is half of Bayes' rule.

This decision tree encodes the practical guard against base-rate neglect. A high model score alone is not enough to trust a positive call; you must also confirm the base rate was accounted for. If it was, you can act on the prediction; if it was not, a high score on a rare event is most likely a false alarm. A low score, meanwhile, generally points to a negative.

The tree turns the abstract warning into a checklist you can apply at decision time. It captures the core lesson of the post in a flowchart: model confidence and base rate must be considered together, never the score in isolation. Walking a prediction through these branches before acting on it catches the most expensive class of mistakes.

Assuming independence that does not exist is a modeling trap with quiet consequences. Naive Bayes multiplies per-feature probabilities as though features were independent given the class, which is almost always false — in spam, words like 'free' and 'money' co-occur far more than chance. The model often still ranks well enough to be useful, which is exactly why the error hides.

The danger is in the reported probabilities. Because the independence assumption is violated, the numbers Naive Bayes outputs are distorted and should never be read as well-calibrated confidences. The lesson is to enjoy the speed and simplicity of independence assumptions for ranking while remaining deeply skeptical of the probabilities they produce.

This snippet demonstrates the underflow trap and its fix in a way you can run. Multiplying a thousand small probabilities with np.prod returns exactly 0.0 — the true product is too tiny for floating point to represent, so all information is lost. Computing the same thing as the sum of log-probabilities yields -4605.17, a perfectly representable number that preserves the magnitude.

The fix is the single most important numerical habit in probabilistic ML: work in log-space. Sums of logarithms replace products of probabilities, staying stable no matter how many terms you combine. You exponentiate only at the very end, and often not even then. Every serious likelihood, loss, and inference computation relies on this technique.

Trusting raw softmax as truth is the deep-learning-specific version of the calibration warning. Modern neural networks are systematically overconfident; a softmax output of 0.999 may correspond to a prediction that is wrong several percent of the time. Using these raw numbers as if they were honest probabilities for thresholds or decisions leads to confident, expensive errors.

The remedy is post-hoc calibration on held-out data. Temperature scaling, the standard technique, divides the logits by a single learned scalar before softmax, softening overconfident outputs to match observed accuracy. The rule of thumb is simple: never treat raw softmax as a calibrated probability for any decision that matters until you have verified or corrected its calibration.

This mind map organizes the post's traps into three families so they are easier to remember. Reasoning errors include flipping the conditional and neglecting base rates — failures of inference. Modeling errors include false independence assumptions and missing calibration — failures of how the model represents probability. Numerics errors include underflow and the panic over densities exceeding one — failures of computation.

Grouping the mistakes this way gives you a mental checklist with three drawers. When a probabilistic system misbehaves, you can ask in turn: is my reasoning sound, is my model's structure honest, and is my arithmetic numerically stable? Most real bugs fall into one of these three buckets, and naming the bucket speeds the fix.

Fearing a density above one is a harmless-looking confusion that signals a deeper misunderstanding worth fixing. A probability mass function value must never exceed one, because it is an actual probability. But a probability density function value can exceed one without any contradiction, because density is not probability — only the integral of density over a range is a probability.

A narrow, tall Gaussian can easily have a peak density of 1.6 or more; nothing is broken. The practical takeaway is to stop checking whether density values are at most one, and instead check that the area under the density integrates to one. Conflating density with probability is the root of this confusion and several related errors with continuous distributions.

Mistaking correlation for causation is the most consequential error when models drive interventions. A feature can predict a target probabilistically without causing it; models happily exploit any correlation that improves predictions. That is fine for passive prediction, but if you act on the model as though the relationship were causal, your intervention can backfire because you changed a symptom, not the cause.

Probability of co-occurrence is not a mechanism. A model might learn that a certain billing code predicts readmission, but forcing that code to change will not change patient outcomes. Whenever a prediction motivates an action that alters the world, you must ask whether the relationship is causal, because predictive probability alone cannot answer that question.

This checklist distills the entire post into five questions to ask before trusting any probabilistic output. Which conditional do I actually have? Did I include the prior or base rate? Are the features I am treating as independent really independent? Am I computing in log-space to avoid underflow? Are my probabilities calibrated against observed reality?

Running a system through these five questions catches the overwhelming majority of probability mistakes covered in this post and the whole day. It is a fast, repeatable defense you can apply during code review, model evaluation, or incident debugging. Internalize the checklist and you will avoid the errors that quietly wreck systems built by people who never learned to ask them.

This final teaser closes Day 10 and points forward. With probability understood as both a concept and a set of practical skills and pitfalls, you are ready to build on it toward statistics and inference, where these foundations turn into tools for learning from data.

The day's arc — concept, why, mechanics, code, mistakes — was designed to leave you not just informed but operational. You now have the language, the motivation, the machinery, the code, and the cautionary map. That is the full foundation that everything probabilistic in machine learning is built upon.