Statistics Essentials

The final post is a field guide to the statistical mistakes that survive precisely because the math still runs and the numbers still print. The cover makes the key point: most bad analysis isn't bad arithmetic — it's good arithmetic applied to the wrong question or interpreted wrongly.

These traps feel intuitive, which is why they're dangerous. Each section names a specific failure mode, shows why it's tempting, and gives the defense. Learning them once inoculates you against errors that derail real projects and published research alike.

Correlation-is-not-causation is the most famous statistical trap and still the most violated. Ice cream sales and drowning deaths rise and fall together across the year, but neither causes the other — summer heat drives both independently. The correlation is real; the causal story is fabricated.

The deeper lesson is the lurking variable: a third factor that influences both observed variables and creates a spurious link between them. Whenever you see a correlation, the disciplined response is to ask what else could produce it. Establishing causation requires a controlled experiment or careful causal inference, never a correlation alone.

This flow diagram makes the lurking-variable structure visible. Hot weather is the real driver. It independently increases ice cream sales and independently increases drownings. An observer who only watches the two effects sees them move together and is tempted to connect them directly.

Visualizing it as one cause branching to two effects is the antidote. Whenever two variables correlate, sketch the possible causal diagrams: maybe A causes B, maybe B causes A, or maybe a hidden C causes both. Most spurious correlations dissolve the moment you draw the third arrow.

Misreading p-values is so widespread that it shows up in peer-reviewed papers. A p-value below 0.05 does not mean there's a 95% chance the effect is real. What it actually means is narrow and conditional: if there were truly no effect, you'd observe data this extreme or more only 5% of the time.

It's a statement about the data under an assumption, not a statement about the probability your hypothesis is true. It also says nothing about how big the effect is — a microscopic, meaningless difference can be highly significant with enough data. Treat the p-value as one piece of evidence, always paired with effect size.

This code makes the fragility of the mean to outliers visceral. Six reasonable salaries average to about 51. Add a single founder earning 5000, and the mean leaps to 764 — a number that describes no one in the dataset. The median, by contrast, barely moves, staying at a representative 51.

The lesson connects back to the earlier post: the mean uses every value and so is hostage to extremes, while the median is robust. When data has outliers or heavy tails, leading with the mean paints a false picture. This is why robust statistics exist and why you should always check whether one extreme value is steering your summary.

Simpson's paradox is the most counterintuitive trap because it defies arithmetic intuition: a trend present in every subgroup can completely reverse when the groups are pooled. A treatment can outperform a control among men and separately among women, yet appear worse overall, simply because the groups have different sizes and base rates.

The cause is a confounding variable interacting with unequal group sizes. The defense is to always check whether a meaningful subgroup variable exists and whether the aggregate hides a reversal. Aggregating blindly can produce a conclusion that is the exact opposite of the truth in every actual subpopulation.

This comparison lays the paradox out side by side. Within each group — men and women — the drug wins. But when you combine the data, the drug appears to lose, purely because the group sizes and their baseline rates differ in a way that skews the pooled numbers.

The practical takeaway is to be suspicious of any aggregate statistic computed across heterogeneous groups. Before trusting a combined number, ask whether the groups differ in size or baseline, and whether breaking the data down by the relevant variable tells a different — possibly opposite — story.

Sampling bias is the trap where no amount of sophisticated math can save you, because the problem is upstream of the analysis. If your sample doesn't represent the population, every conclusion drawn from it is suspect, no matter how clean the statistics. A satisfaction survey answered only by your happiest customers will report glowing numbers that mean nothing.

In ML this shows up when test data is collected differently from production data, or when training data overrepresents certain groups. The model learns the bias and applies it confidently. The defense is to scrutinize how the sample was collected and ask who or what is systematically missing from it.

Survivorship bias is sampling bias's most elegant cousin, captured by the WWII bomber story. Engineers wanted to reinforce the parts of returning planes riddled with bullet holes. The statistician Abraham Wald pointed out the opposite: reinforce where the returning planes had no holes — because the planes hit there never made it back to be counted.

The insight is that you systematically observe only the survivors, and the missing data often carries the real signal. It appears everywhere: studying only successful startups, only published studies, only customers who didn't churn. Always ask what's absent from your data, because absence is itself information.

This code shows why small samples lie with confidence. A truly fair coin flipped only 10 times can easily land 7 heads, reading as a 70% bias that isn't there. Flip it 100 times and the proportion drifts toward 0.5; flip it 10,000 times and it converges tightly. The law of large numbers is doing the work.

The practical warning is that dramatic-looking effects in small samples are frequently noise. Before getting excited about a striking result, check the sample size — and remember from the central limit theorem that the spread of an estimate shrinks only as 1/√n, so genuine precision demands real volume of data.

These quick defenses distill the whole post into reflexes you can apply on the spot. Ask what else could cause an observed pattern, which guards against false causation. Report effect size and confidence intervals rather than bare p-values, which guards against significance theater. Plot the data before trusting any summary, which catches outliers and weird shapes.

Check subgroups for hidden reversals to defend against Simpson's paradox, and ask who is missing from the sample to defend against sampling and survivorship bias. None of these require advanced math — they're disciplined questions that catch the majority of real-world statistical errors.

The decision tree turns the post into an actionable checklist you can run on any claim, including your own. If someone claims X causes Y, demand a controlled experiment, not a correlation. If a result is reported as a single number, insist on a measure of spread and an interval around it. Otherwise, plot the data and check the subgroups before believing the headline.

Running analysis through this tree is how you move from producing numbers that look rigorous to producing conclusions you can actually defend. That habit of structured skepticism, applied as readily to your own work as to others', is what separates trustworthy data work from confident guessing.

That completes the Statistics Essentials theme. You now have the vocabulary, the reasons it matters, the mechanics of center, spread, shape, and the central limit theorem, a runnable code workflow, and a field guide to the traps that sink real projects.

The next theme builds directly on these foundations — every model you train and every result you report rests on the statistical thinking covered here. Keep these five posts handy; they're the reference you'll return to long after the formulas feel automatic.