Backpropagation, Step by Step

This cover frames post 3 as the engine room, signaling a shift from motivation to mechanism. The promise — one tiny network, real numbers, the gradient flowing backward step by step — tells the reader the payoff is a concrete trace they can follow, not more abstraction.

The key reframing is that backprop is a sequence of a few repeated steps, not a single mysterious event. Doing the bookkeeping by hand on a one-neuron-then-one-layer example is the fastest way to kill the mystery, because every number is visible and every multiplication is the chain rule in plain sight.

Step 0 establishes the precondition that everything else depends on: the forward pass must run and cache its intermediate values. For a single neuron, that means computing z = w·x + b, then a = f(z), then the loss, and crucially saving z, a, and x for the trip back. Nothing is discarded because backprop reuses all of it.

Emphasizing the cache here pays off twice. It explains the memory cost flagged in post 2, and it sets up the literal cached variables in the post 4 numpy build. Beginners who skip this step are confused later about where the values in the backward equations come from; grounding it first prevents that.

Step 1 is where backprop actually begins: at the output, where the error is known. Differentiating the loss with respect to the output gives the first gradient — for a squared error, dL/da = 2(a - y). Calling this the error signal gives it a name the reader can track as it moves backward.

The framing that everything after this just routes the signal backward is the central intuition of the whole trace. Backprop does not compute each weight's gradient from scratch; it takes this one starting signal and transforms it step by step. Seeing the loss gradient as the source makes the layer-by-layer propagation that follows feel inevitable rather than arbitrary.

Step 2 passes the signal through the activation, the first application of the chain rule in the trace. The incoming gradient dL/da is multiplied by the activation's local derivative f'(z) to produce dL/dz, the gradient with respect to the pre-activation. The concrete derivatives — a(1-a) for sigmoid, the 0-or-1 step for ReLU — make it actionable.

This step is where the activation choice connects to gradient flow. A saturating activation with a tiny derivative shrinks the signal here, which is the seed of the vanishing-gradient problem in post 5. Doing the multiplication explicitly shows the reader exactly where in the trace that shrinkage happens.

The cycle diagram visualizes the signal's journey through one layer: start with dL/da from the loss, multiply by f'(z) through the activation to get dL/dz, then multiply by the input x to get dL/dw. Showing it as a cycle of multiplications reinforces that backprop through a layer is just a short chain of products.

Visualizing it this way makes the chain rule tangible — each arrow is one local derivative being multiplied in. It is the portable mental image the reader can carry into the code, where each node becomes a single line, and into deeper networks, where the same cycle repeats per layer.

Step 3 reaches the gradients the reader actually wanted: dL/dw and dL/db. Applying the chain rule to z = w·x + b gives dL/dw = dL/dz · x and dL/db = dL/dz · 1. The cached input x from step 0 is exactly what makes the first multiplication possible, closing the loop on why the forward pass saved it.

The bias gradient being dL/dz · 1 is a small but clarifying detail: the bias adds directly to z with a coefficient of one, so its gradient is just the incoming signal unchanged. Seeing weight and bias gradients side by side shows they come from the same step, differing only in what they multiply.

Step 4 generalizes the trace from one layer to a whole network. To reach an earlier layer, the signal passes through the weights: dL/dx = dL/dz · w. That dL/dx becomes the incoming gradient for the previous layer, which then repeats steps 2 and 3 with its own cached values. The same four moves cascade backward to the input.

This is the slide that turns a single-neuron example into real backpropagation. The recursion — each layer's input-gradient is the next layer back's output-gradient — is precisely why the algorithm is called propagation. It also previews the matrix form in post 4, where dL/dx = dL/dz · w becomes a transposed weight multiply.

This code slide implements step 0 concretely: a forward pass through one sigmoid neuron with real numbers, caching z, a, and the loss. Running it gives the reader actual values to carry into the backward pass, making the trace tangible rather than symbolic.

Using concrete inputs (x = 1.5, w = 0.8, and so on) is deliberate. The reader can change a number and re-run to see how the cached values shift, which builds intuition for how the forward pass feeds the backward one. The explicit sigmoid implementation also sets up its derivative a(1-a) in the next slide.

This code slide implements the backward pass by hand, mapping each line to a step from the slides. dL_da is step 1, da_dz is the sigmoid derivative, dL_dz is step 2 through the activation, and dL_dw and dL_db are step 3. The reader sees the chain rule as a short sequence of multiplications producing the gradients.

Pairing this with the forward-pass code completes a runnable mini-example: forward to get values, backward to get gradients. The fact that it is plain arithmetic with no framework involved is the point — backprop on this neuron really is just five multiplications, and seeing it demystifies the loss.backward() call from post 2.

The compare diagram puts the forward and backward passes side by side so their symmetry is unmistakable. The left column caches z, a, and the loss on the way in; the right column applies the chain rule on the way out, each line undoing one step of the forward computation in reverse order.

This mirrored structure is the deepest intuition of the post: the backward pass retraces the forward pass in reverse, multiplying by local derivatives at each step. Presenting both columns together gives the reader a single image that captures the entire algorithm, which the recap then distills into the four steps.

The recap orders the four backward steps into the exact sequence the trace followed: cache in the forward pass, start with dL/da, multiply by f'(z) through the activation, multiply by x to get dL/dw, and multiply by w to flow to earlier layers. Reciting this is enough to backprop through any simple network by hand.

This ordered list doubles as a study aid for post 4, where these steps appear as numpy lines. A reader who can recite the four moves can read the from-scratch backward pass and recognize each multiplication, rather than treating the matrix operations as opaque.

The CTA transitions from a by-hand trace to a full runnable build. Having seen the four steps on a single neuron, the reader is primed to implement them in numpy for a two-layer network and verify the result against PyTorch's autograd.

Naming the deliverable — backprop you wrote yourself, checked against the framework — sets a concrete expectation that post 4 is buildable and verifiable, which is the right reward after working through the math.