Backpropagation, Step by Step

This cover signals a gear change: the previous post traced backprop on paper, this one implements it in runnable code. The promise is a complete numpy build — forward, backward, one update — plus a verification step against PyTorch's autograd, which is exactly what the code-example angle demands.

Using a tiny two-layer network with hand-written gradients is a conscious choice. It is small enough to read in full and to gradient-check, yet large enough to show backprop flowing through more than one layer, including the transpose mechanics that a single neuron never reveals. The verification against autograd is what turns 'I think this is right' into proof.

The setup slide front-loads the network and data so the rest of the build runs without surprises. Two input features, a three-unit hidden layer, and a one-unit output give a network with real structure but few enough parameters to inspect by hand. Seeding the random number generator makes the run reproducible, which matters for the gradient check later.

Initializing weights small and random (scaled by 0.1) and biases to zero is standard practice that also previews post 5's note that initialization affects gradient stability. Laying out the exact shapes here means the matrix multiplications in the forward and backward passes have clear, traceable dimensions.

This slide implements the forward pass and, crucially, caches every value backprop will need: z1, a1, z2, and a2. The structure mirrors post 3 exactly — linear, then ReLU, then linear — now in matrix form. The MSE loss collapses the output to a single scalar, which is the starting point for the backward pass.

Using a linear output (a2 = z2) keeps the example focused on the backprop mechanics rather than a softmax or sigmoid at the end. Every variable computed here is reused in the next slide, making the dependency from post 1 and post 3 literal: without these cached values, the backward pass has nothing to multiply against.

This is the payoff slide: the backward pass in numpy, computed layer by layer from the output back. The output-layer gradients come first — dL_da2 from the loss, then dL_dW2 by multiplying the signal with the cached activation. Then the signal flows back through W2 to the hidden layer, gets gated by the ReLU derivative, and produces dL_dW1.

Each line is one of post 3's four steps in matrix form. dL_dz1 = dL_da1 * (z1 > 0) is the ReLU derivative as a boolean mask; dL_dW1 = dL_dz1 @ x.T is the dL/dw = dL/dz · x rule with the required transpose. Computing the output layer before the hidden layer makes the propagation direction visible: later layers' gradients feed earlier ones.

This slide performs one gradient-descent step, deliberately kept separate from the backward pass to reinforce post 1's measurement-versus-action distinction. Each weight and bias is moved a small step against its gradient, scaled by the learning rate. Backprop produced the gradients; this is where the weights actually change.

The comment noting that a re-run forward pass should show a lower loss gives the reader a concrete way to confirm the step worked. Seeing the update as four simple subtractions, fully separate from the gradient computation, makes clear that optimizer.step() in a framework is doing exactly this — nothing more mysterious than w -= lr * grad.

This explanatory slide addresses the part of the backward code most likely to confuse: the transposes. dL_dW2 = dL_da2 @ a1.T is the matrix form of the scalar rule dL/dw = dL/dz · x from post 3 — each weight's gradient is its incoming signal times the input that flowed through it. The transpose is what makes the shapes align with the weight matrix.

Explaining the transpose in terms of the post 3 rule, rather than as a mysterious linear-algebra requirement, keeps the reader grounded. The shapes have to produce a matrix the same size as the weight matrix being updated, and the transpose is simply the bookkeeping that guarantees it. This is the detail that trips up most first attempts at hand-written backprop.

The verification slide is what elevates the build from a demonstration to a proof. It rebuilds the same tiny network in PyTorch with requires_grad tensors, runs loss.backward() to let autograd compute the gradients, and compares them to the hand-written results with torch.allclose. If they match, the hand-written backprop is correct.

This closes the loop on the whole day: the by-hand chain rule from post 3, the numpy implementation here, and the autograd from post 2 all agree on the same numbers. The verification habit also previews post 5's gradient-check fix — comparing a hand-written gradient against a trusted reference is the standard way to catch backprop bugs.

The pipeline diagram distills the build into four stages — forward, loss, backward, step — independent of the specific numpy. It reinforces that implementing backprop is a disciplined sequence, and that the forward stage's job is to cache the values the backward stage consumes.

The stage details echo the code: cache z1, a1, z2 on the way in; produce an MSE scalar; apply the chain rule per layer on the way back; then step the weights. This portable four-stage template is the thing a reader carries to any network, swapping the layer math while keeping the structure identical, which is exactly how frameworks organize a training step.

This slide situates the single forward-backward-step example inside a real training run, which repeats the four stages over many batches and epochs. Each iteration predicts and caches, measures loss, fills gradients, and steps the weights — the same loop from the neural-network training posts, now with the backward pass made explicit.

The key payoff is the connection to framework code: the hand-written version makes visible exactly what model(x), loss.backward(), and optimizer.step() do under the hood. A reader who has written the loop manually reads framework training code with full understanding rather than treating those three calls as black boxes.

The recap orders the six build steps so a reader can reproduce the workflow from memory: initialize small random weights, forward pass while caching activations, backward pass starting from the output layer, transpose to match weight shapes, step each weight against its gradient, and verify against autograd with allclose. It is a portable template, not just code for one example.

The emphasis on computing the output layer before the hidden layer, and on the transposes, deliberately bakes in the two things most likely to go wrong in a hand-written backward pass — the same kind of error the gradient check in post 5 is designed to catch.

The CTA pivots from the working build to the cautionary post. Having implemented backprop and verified it, the reader is ready to learn how it goes wrong — vanishing gradients, a forgotten zero_grad, a detached tensor, in-place ops, non-differentiable operations, and unchecked hand-written gradients.

Naming the specific traps in the teaser creates anticipation and signals that the day does not stop at 'it works in a notebook.' Real competence is knowing the failure modes backprop will not warn you about, which is exactly the promise of post 5.