Backpropagation, Step by Step

This cover names the defining hazard: backprop will faithfully compute a correct gradient of a broken setup and never warn you. The math is rarely the problem; the traps surround it. Framing these as expected dangers to guard against, rather than rare bugs, sets the right defensive mindset for the whole post.

The common-mistakes angle works best as a failure manual — each slide is a specific, nameable error plus its fix. The cover lists the lineup so the reader knows they are getting six concrete traps, not vague cautions, matching the depth standard the series holds to.

Vanishing and exploding gradients lead because they are the failure mode most intrinsic to backprop's design. The algorithm multiplies many local derivatives together as the signal flows back; if those factors are consistently below one, their product shrinks toward zero and early layers stop learning, and if consistently above one, it blows up to NaN.

Naming saturating activations like sigmoid as a classic cause connects directly to post 3, where the activation derivative a(1-a) is tiny across most of its range. This is the concrete mechanism behind the abstract problem: each saturated layer multiplies the signal by a small number, and a deep stack of small numbers vanishes. The fix slide that follows addresses all three remedies.

This code slide implements the standard defenses against vanishing and exploding gradients. ReLU activations are non-saturating, so they do not crush the gradient the way sigmoid does, which addresses vanishing. Good initialization keeps the initial signal scale sane. And gradient clipping caps the gradient norm during training, which is the direct guard against explosion to NaN.

Showing clip_grad_norm_ explicitly is deliberate because explosion is often what produces sudden NaN losses that beginners cannot explain. Capping the norm before the optimizer step bounds the update size regardless of how large the raw gradient got, which stabilizes training of deep or recurrent networks where exploding gradients are common.

Forgetting optimizer.zero_grad() is the most common PyTorch-specific backprop bug and it is invisible until the model misbehaves. The mechanism is precise: PyTorch accumulates gradients into .grad by default, adding to whatever is already there. Without clearing each iteration, you sum gradients across batches, producing wrong updates and degrading training.

The fix is a single line before backward, every iteration, and naming exactly where it goes matters because the order is what makes it correct. This connects to the training loops in the neural-network posts, where zero_grad appeared in its proper place; here the focus is on what silently breaks when it is omitted.

This code slide makes the zero_grad fix tangible by showing the correct loop with the easy-to-forget line called out. Seeing zero_grad as the first statement inside the batch loop, before the loss is even computed, cements the placement that prevents accumulated-gradient bugs.

Pairing the warning with runnable code follows the post's pattern of never leaving a problem unsolved: the reader gets both the diagnosis and the exact prescription. The comment 'fills fresh gradients' on the backward line reinforces why clearing first is necessary — backward writes correct gradients only into a zeroed buffer.

Accidentally breaking the computation graph is a subtle trap because the code runs fine and only some weights silently stop updating. Operations like .detach(), .item(), .numpy(), and wrapping code in torch.no_grad() all sever the tensor from the graph that post 2 described. Do any of them mid-network and the gradient cannot flow past that point.

The fix is discipline about where these operations appear: keep tensors attached throughout the forward pass, and detach only for logging, metrics, or genuine constants. This is the failure mode that most often produces a model that trains but plateaus early, because a whole subset of its weights never received a gradient. The diagram that follows lists the offenders side by side.

The compare diagram turns the graph-breaking warning into a quick reference the reader can recognize by sight. The left column lists what breaks backprop — detach mid-graph, item or numpy conversions, no_grad around training code, and non-differentiable ops. The right column lists the safe equivalents — attached tensors, differentiable ops, no_grad only at evaluation, and detach only for logging.

Presenting it as a two-column contrast makes the rule memorable: the same operations are fine in one context and harmful in another. no_grad belongs at evaluation time, not around a training step; detach belongs around a logging line, not in the middle of the forward pass. This visual consolidates the two graph-related slides into one decision aid.

In-place operations are dangerous because they overwrite the very values backprop cached for the chain rule. Operations like x += 1, relu_(), or assigning into a tensor slice modify data that the backward pass still needs. PyTorch frequently catches this and raises 'a variable needed for gradient computation has been modified by an inplace operation,' but subtler cases pass silently with wrong gradients.

The fix is to prefer out-of-place operations inside the forward pass, accepting a small memory cost for correctness. This ties back to the caching requirement from post 1 and post 3: backprop depends on the forward pass's stored values being intact, and an in-place op breaks that contract. When the error does appear, it is pointing at exactly this problem.

Non-differentiable operations are a trap because they look like ordinary computation but have zero or undefined derivatives, so backprop cannot pass a useful signal through them. argmax, rounding, hard thresholds, and most sampling fall in this category. Place one in the forward path and every weight upstream of it receives no gradient and never learns.

The fix depends on intent: where possible, use a differentiable relaxation such as softmax instead of argmax, which gives a smooth gradient. When a genuinely hard choice is required, a straight-through estimator passes the gradient as if the hard operation were the identity. Naming both options gives the reader a path forward rather than just a prohibition, consistent with the post's solve-every-problem pattern.

Not gradient-checking hand-written backprop is the mistake that the post 4 build directly guards against. An off-by-a-transpose or a missing scaling factor produces gradients that look plausible — the right shape, reasonable magnitudes — but are subtly wrong, so training merely underperforms rather than crashing. There is no error to catch.

The fix is the numerical gradient check: compare your analytic gradient to a finite-difference estimate (loss(w+ε) - loss(w-ε)) / 2ε, which should agree to several decimal places. This is the same verification idea as post 4's allclose check against autograd, generalized to the case where you have no framework reference and must construct the reference yourself.

This code slide implements the gradient check concretely. A small helper computes the numerical gradient by perturbing the weight up and down by epsilon and measuring the loss difference, then an assertion confirms the analytic backprop result agrees to within a tight tolerance. If the assertion fails, the hand-written gradient has a bug.

The central-difference form (using both w+eps and w-eps) is chosen deliberately because it is far more accurate than a one-sided difference, making the check sensitive enough to catch small errors. This is the standard tool for validating any custom backward implementation, and running it once on a new layer saves hours of debugging a model that trains but quietly underperforms.

The recap consolidates all six fixes into a single screenshot-able checklist, turning the post into a pre-flight check before trusting any backprop: ReLU plus good init plus clipping for stability, zero_grad before every backward, keep tensors attached, avoid in-place ops in the forward pass, use differentiable operations or relaxations, and gradient-check anything hand-written.

The list is intentionally portable. The stability and graph items apply to any framework use, while zero_grad and the in-place caution are PyTorch-specific habits that prevent the most silent damage. The gradient-check item ties the failure manual back to the verification discipline modeled in post 4.

The CTA closes both the post and the day, pointing forward to gradient descent variants as the natural sequel. The framing is deliberate: post 1 separated backprop (compute the gradient) from gradient descent (take the step), and the next topic dives into how SGD, momentum, and Adam each use the gradients backprop produces.

This creates a clean narrative arc across the day and a strong hook into Day 44. The reader finishes understanding not just what backprop is and how it works, but how easily it fails silently and why the optimizer that consumes its gradients — the next topic — deserves its own deep dive.