Dropout & BatchNorm

This is the engine-room post. The previous two established what these layers are and why they matter; here we replace intuition with explicit formulas, tracing exactly what each layer does to a number on the way forward and how the gradient passes back.

The payoff is that the train/eval distinction and the running-statistics machinery stop being vocabulary and become things you have actually computed. Both layers are simpler than their reputation suggests once you write down the arithmetic, and doing the trace once is worth more than reading ten explanations.

Dropout's forward pass is two operations: mask, then rescale. You sample a binary mask in which each unit survives with probability 1-p, multiply the activations element-wise by that mask (zeroing the dropped units), and then divide the result by (1-p). That division is the 'inverted dropout' trick.

The reason for the rescale is to keep the expected value of each activation unchanged despite the zeros, so that the layer behaves consistently between training and inference. Because the scaling is folded into training, test time needs no adjustment at all — at eval the layer is simply the identity, which is exactly why the implementation branches on a training flag.

This from-scratch implementation makes the mechanics undeniable. When training is False, the function returns its input untouched — the eval-time identity behavior. When training is True, it draws a random mask the same shape as the input, keeps the units where the random value exceeds p, and divides by (1-p) to rescale.

Reading this you can see there are no learnable parameters anywhere; Dropout's only 'parameter' is the fixed hyperparameter p you choose. That absence of learned weights is what distinguishes it from BatchNorm and confirms it is purely a regularization mechanism rather than a transformation the network adapts.

BatchNorm's forward pass is four clean steps. First, compute the mean of each feature across the batch. Second, compute the variance across the batch. Third, normalize each value by subtracting that mean and dividing by the square root of the variance plus a small epsilon — that epsilon prevents division by zero and is why you always see it in the formula. Fourth, apply the learned affine transform: multiply by gamma and add beta.

Laying it out as four steps clarifies which parts are statistics computed from data (mean, variance) and which parts are learned parameters updated by gradient descent (gamma, beta). Keeping that division straight is essential to understanding both the backward pass and the train/eval behavior.

This from-scratch BatchNorm mirrors the four steps exactly. It computes the per-feature mean and variance over axis 0 (the batch dimension), normalizes to produce x_hat, and returns gamma * x_hat + beta. The eps inside the square root guards against a zero-variance feature blowing up the computation.

The code makes visible that mean and var are derived from the current batch — not learned — while gamma and beta are the parameters the optimizer trains. This is the crux of why BatchNorm behaves differently at eval: the batch-derived statistics are unavailable for a single example, so they must be replaced by stored running averages, which the next slides address.

The pipeline diagram traces a batch of pre-activations through BatchNorm's stages: the raw batch enters, gets centered by subtracting the mean, gets scaled by dividing by the standard deviation, and finally passes through the learned affine transform parameterized by gamma and beta.

Visualizing it as a pipeline reinforces that normalization and the affine step are separate concerns. The first two stages are fixed, data-driven standardization; the last stage is where the network regains the freedom to undo or reshape that standardization if doing so lowers the loss. That separation is exactly why gamma and beta exist.

Why bother with gamma and beta at all? Because forcing every layer's output to have mean 0 and variance 1 would be too rigid — it could strip away representations the network actually needs. The learned scale gamma and shift beta give the network an escape hatch: it can rescale and recenter the normalized values however the task demands.

In the limit, if the original unnormalized activations were ideal, the network can learn gamma equal to the batch standard deviation and beta equal to the batch mean, exactly recovering the input. So BatchNorm never reduces the network's expressive power; it only changes the optimization landscape while leaving the identity reachable.

Running statistics are the bridge from training to inference. At evaluation time you frequently process a single example or a small batch where a meaningful batch mean and variance simply cannot be computed. BatchNorm solves this by maintaining an exponential moving average of the per-feature mean and variance throughout training, updating these running estimates on every batch.

At inference it freezes and uses those running statistics instead of any live batch statistics. This is the concrete reason eval mode behaves differently from training mode — and why a model evaluated without switching to eval mode produces inconsistent, batch-dependent outputs. The running stats are stored on the layer, not learned, which is a distinction worth keeping clear.

This comparison puts the train and eval behaviors of both layers side by side so the differences are unmistakable. In training, Dropout zeros random units, BatchNorm uses the live batch mean and variance and updates its running averages, and the overall output is stochastic — it changes from run to run. In eval, Dropout passes everything through, BatchNorm uses its stored running statistics and updates nothing, and the output is fully deterministic.

This table is essentially the contract these layers expect you to honor by calling train() and eval() at the right times. Internalize it and the single most common bug with these layers — covered in the final post — becomes easy to avoid.

This snippet confirms that, in practice, you never implement any of the backward math yourself. Autograd records the operations of both the mask-and-rescale of Dropout and the normalize-and-affine of BatchNorm, then computes all the gradients automatically when you call backward().

The printed shape of bn.weight.grad — the gradient with respect to gamma — is the tangible proof that gamma is a learned parameter receiving a learning signal, just like any weight. Seeing the hand-derived forward passes first is what lets you trust and debug this framework version: you now know exactly what operations autograd is differentiating through.

This recap pins down the mechanics: Dropout is mask * x / (1-p) and is active in training only; BatchNorm normalizes then applies gamma * x_hat + beta; gamma and beta restore expressiveness so normalization never costs you representational power; the stored running mean and variance are what BatchNorm uses at eval; and autograd handles backpropagation through both for you.

With the math traced by hand, you are ready to build the real thing. The next post assembles a working network that uses both layers and trains it end to end.

The teaser points to the hands-on build. Having traced the forward passes and understood the train/eval split, the next post wires both layers into a real PyTorch network — including the all-important model.train() and model.eval() calls — so the mechanics become a working, runnable artifact.