Statistics Essentials

This is the code-heavy post, designed so you can copy each block, run it, and watch the abstractions from the previous posts turn into concrete numbers. The cover sets the contract: no black boxes. Every statistic is one function call, and we walk a small dataset end to end — describe, test, interval, correlate.

The whole point of this angle is muscle memory. Theory you've only read fades; theory you've executed and tweaked sticks. By the end you'll have a runnable template for the first pass on almost any tabular dataset.

The setup block builds a controlled dataset so the rest of the post has known ground truth. We create two groups, A and B, each with 50 scores. Group A is drawn from a normal distribution centered at 70, group B centered at 75, both with the same spread of 8. Because we know the true generating means differ by 5, we can later check whether our statistical tests correctly detect that.

The seeded random generator (default_rng(42)) makes the results reproducible, so anyone running this gets the same numbers. Building synthetic data with known parameters is a great habit for sanity-checking that your statistical tools behave as expected.

The describe() method is the single fastest way to understand a numeric column, and it should be your first move on any new dataset. In one call it returns the count, mean, standard deviation, minimum, the three quartiles (25%, 50%, 75%), and the maximum. That's center, spread, and a sense of shape all at once.

The groupby summary extends this to compare groups directly. Seeing group B's mean sit above group A's is the first hint of a difference — but a hint is all it is until we test whether the gap survives the noise, which is what the later slides do.

This slide pulls the key descriptive numbers out individually so each one is explicit. Mean and median together hint at skew: if they diverge sharply, the distribution is lopsided. The sample standard deviation uses ddof=1, the correct choice for a sample as established earlier.

The interquartile range — the 75th percentile minus the 25th — is a robust spread measure that ignores the extreme tails, complementing the outlier-sensitive standard deviation. The 95th percentile answers 'what value do all but the top 5% fall below,' a framing that's often more useful for decisions than the mean.

Before running the test, this slide states the question precisely, because a test you don't understand produces a number you'll misread. Groups A and B have different sample means, but each sample is noisy — some of that gap could be luck of the draw. The two-sample t-test formalizes the question: is the observed gap large enough that it would be unlikely to arise if the two groups truly had the same mean?

The output is a p-value: the probability of seeing a gap this large or larger under the assumption that the groups are actually identical. A small p-value means the data is hard to explain by chance alone, which is evidence — but not proof — of a real difference.

This is the test itself, and scipy makes it a single call. ttest_ind takes the two groups and returns a t-statistic and a p-value. The t-statistic measures the gap in units of standard error — bigger means more separation relative to noise. The p-value translates that into a probability under the null hypothesis of no difference.

The conventional threshold of 0.05 is exactly that — a convention, not a law of nature. Because we built the groups with truly different means (70 vs 75) and a decent sample size, we expect a small p-value here, confirming the test correctly detects the real difference we baked in.

The confidence interval slide turns a point estimate into a range, completing the inferential picture. We compute group B's mean, then its standard error — the standard deviation divided by the square root of the sample size, which is how spread translates into uncertainty about the mean. scipy's t.interval then builds the 95% interval using the t-distribution with the right degrees of freedom.

The t-distribution rather than the normal is the correct choice when estimating with a finite sample and unknown population standard deviation; it has slightly fatter tails to account for the extra uncertainty. The output gives the mean alongside a range you can actually defend, which is the proper form of a statistical answer.

Correlation measures whether two variables move together, and pearsonr quantifies it on a scale from −1 to +1. We construct an 'hours' variable deliberately related to score plus noise, so we expect a positive correlation. The function returns r, the correlation coefficient, and a p-value for whether that correlation differs significantly from zero.

An r near +1 or −1 indicates a strong linear relationship; near 0 means little linear association. The critical caveat — reinforced in the next post — is that even a strong, significant correlation says nothing about causation. It tells you the variables track each other, not that one drives the other.

This pipeline diagram is the reusable mental template the post is teaching. The first pass on almost any dataset follows the same four moves: describe it to get summaries, visualize it to see the distribution and spot outliers, test to check whether differences are real, and build an interval to express your estimate honestly.

Internalizing this sequence means you never face a new dataset blankly. You have a repeatable opening that surfaces problems early and produces defensible numbers, regardless of the specific domain or columns involved.

Reading the output correctly matters as much as computing it, and these tips guard against the most common misreadings. A p below 0.05 means the result is unlikely under the assumption of no difference — it is not the probability that you're right, a distinction the next post hammers. A narrow confidence interval signals a precise estimate; a wide one signals you need more data.

The final tip is the most important habit: always report effect size, not just significance. A statistically significant difference of 0.01 units may be real and utterly useless. Significance tells you a difference exists; effect size tells you whether it matters.

The closing mistake — p-hacking — is the most insidious because it's easy to do without realizing. If you run ten independent tests, then by pure chance you expect about one to clear the 5% threshold even when nothing is going on. Reporting only that 'winning' test manufactures a false discovery that looks rigorous.

The defenses are discipline and correction: decide your hypothesis and test before you look at the data, and when you genuinely run many comparisons, apply a correction like Bonferroni or control the false discovery rate. This is exactly how reproducibility crises happen in science, and the same trap catches ML teams running many experiments.

That's a full, runnable workflow: describe, test, interval, correlate. Save these blocks as your first-pass template for any tabular dataset, and you'll never stare blankly at a new table again.

The final post turns the lens on the traps — the statistical mistakes that feel intuitive, survive because the code still runs, and quietly sink real projects. Knowing them is what separates a defensible result from one that merely looks rigorous.