Statistics Essentials

This post opens the Statistics Essentials theme, and the goal of the cover is to reframe statistics as infrastructure rather than a school subject. Almost everyone arrives at machine learning thinking of statistics as the dusty prerequisite they tolerated. The truth is the opposite: statistics is the lens that lets you see whether your data and your results mean anything at all.

We set the tone here that the vocabulary is the hard part, not the arithmetic. Once the words population, sample, parameter, and statistic sit in the right slots in your head, the formulas that follow are almost mechanical.

Statistics, defined cleanly, is the science of reasoning about data under uncertainty. It splits into two jobs: summarizing what you actually observed, and using that summary to say something defensible about what you did not observe. Both jobs run constantly inside any ML workflow, often without being named.

The phrase 'under uncertainty' is the load-bearing part. If you could measure every entity perfectly, you wouldn't need statistics — you'd just count. Statistics exists precisely because you only ever see a slice of reality and have to reason carefully about the rest.

The population-versus-sample distinction is the single most important idea in the post, because nearly every later mistake traces back to confusing the two. The population is the complete set of things you care about: every customer, every transaction, every possible image of a cat. You almost never have access to it.

A sample is the subset you actually collected and measured. Because it's only a slice, any number you compute from it carries error. The entire machinery of inferential statistics is built to quantify and control that error rather than pretend it isn't there.

Descriptive and inferential statistics answer two different questions, and conflating them is a common source of overclaiming. Descriptive statistics is honest and modest: it summarizes the data in front of you with means, medians, and measures of spread, making no claim about anything beyond your sample.

Inferential statistics is ambitious: it uses the sample to make claims about the whole population, complete with quantified uncertainty like confidence intervals and significance tests. The danger zone is computing a descriptive number and then quietly treating it as an inferential truth about the world.

Parameters and statistics are easy to mix up because they're computed the same way but mean different things. A parameter is a fixed, usually unknown number that describes the entire population — the true mean μ or true standard deviation σ. It's a property of reality, not of your data.

A statistic is a number you compute from your sample, like the sample mean x̄, used as your best guess at the corresponding parameter. The Greek-versus-Latin lettering convention is not decoration: it's a constant visual reminder of whether you're talking about the truth or your estimate of it.

The inference loop is the mental model that ties the whole post together. You start with a population whose true parameters are unknown. You draw a sample of n observations from it. You compute statistics on that sample. You use those statistics to infer the population parameters, accepting some quantified error.

Drawing this as a cycle matters because in practice it repeats: you collect more data, refine estimates, and tighten your intervals. Every honest data claim is really a claim about where you are in this loop and how much uncertainty remains.

Data types are not pedantry — they determine which tools are even legal to use. Numerical data can be continuous, like height or temperature, where any value in a range is possible, or discrete, like counts, where only whole numbers make sense. Categorical data splits into nominal, where categories have no order (colors, countries), and ordinal, where they do (small, medium, large).

Getting the type wrong leads to nonsense: averaging zip codes, or computing a median of unordered categories. In ML this also drives encoding choices — one-hot for nominal, ordinal encoding for ordered categories — so the distinction propagates straight into your feature matrix.

This code slide makes the abstractions tangible by computing the core summaries on a tiny array. The key detail is ddof=1 in the standard deviation call. By default NumPy divides by n (treating the data as the whole population); passing ddof=1 divides by n−1, which is the correct, unbiased choice when your data is a sample.

Running this yourself anchors the vocabulary: n is the sample size, mean and median are measures of center, and std is a measure of spread. These four numbers are the minimal honest description of a one-dimensional dataset.

These vocabulary words recur in every later post and in every library's documentation, so it's worth internalizing them now. An observation is a single data point, usually one row in your table. A variable is one measured property, usually one column. A distribution describes how the values of a variable are spread across their possible range.

An estimator is a recipe — a formula or procedure — for guessing a parameter from a sample. The sample mean is an estimator for the population mean. Thinking in terms of estimators rather than fixed answers is the mindset shift that makes inferential statistics click.

The mind map gives you a one-glance map of the territory so the rest of the theme has a place to hang. Statistics has two branches sitting on a probability foundation. Descriptive statistics handles center, spread, and shape. Inferential statistics handles estimation and hypothesis testing.

Probability sits underneath both as the engine — it's what lets us attach numbers like '95% confident' to our claims. Day 10 covered that engine; this theme is about driving the car it powers. Keeping this map in mind prevents the common feeling that statistics is a random grab-bag of formulas.

The closing mistake is the one that quietly undermines beginners: treating the sample as if it were the population. Your sample mean is not the truth; it's one draw from a distribution of possible sample means. Collect a different sample and you'll get a different number, and that wobble is real and measurable.

The reason statistics is trustworthy is precisely that it refuses to hide this. Instead of reporting a single confident number, it reports an estimate plus a quantified uncertainty. Internalizing that every sample statistic is an estimate-with-error is the foundation everything else in this theme builds on.

That wraps the conceptual foundation. With population, sample, parameter, statistic, and the descriptive-versus-inferential split in place, you have the vocabulary to read any statistical claim critically.

The next post tackles the 'why it matters' angle — showing concretely how these dry-sounding ideas become load-bearing in a real ML pipeline, from catching data bugs to confirming a model improvement is real rather than noise.