Backpropagation, Step by Step

This cover sets the frame for the whole topic: backpropagation is not arcane machinery, it is the chain rule from first-year calculus combined with careful bookkeeping. Most learners arrive believing backprop is the hardest, most mysterious part of deep learning, which makes them avoid understanding it and treat loss.backward() as a spell.

The post that follows is deliberately the 'map' angle. We are not yet justifying why backprop matters historically, nor tracing the math through a concrete network — we are nailing down what the algorithm IS and the vocabulary around it, so the later posts have solid ground to build on.

The definition slide reframes backprop as a procedure with a precise job: compute the gradient of the loss with respect to every weight. The key insight to internalize is that it works backward, from the output where the error is known, toward the input, applying the chain rule at each layer. That direction is not arbitrary — it is the only place the error is actually measured.

The phrase 'how to change each weight to make the prediction less wrong' grounds the abstraction. A gradient is a direction in weight space, and backprop's entire output is that direction. Everything else in training is built on top of this one computation, so getting the definition exact matters.

This slide corrects the single most common misconception about backpropagation: that it trains the network. It does not. Backprop computes gradients and stops there. A separate component — the optimizer running gradient descent — consumes those gradients and actually moves the weights. Conflating the two leads to endless confusion about what loss.backward() versus optimizer.step() do.

The measurement-versus-action framing is worth holding onto for the rest of the day. The whole comparison slide later in the post returns to it, and the from-scratch code in post 4 separates the backward pass from the weight update precisely to make this boundary visible in code.

The vocabulary slide front-loads the five terms that recur in every later slide and every piece of backprop material the reader will encounter. Defining loss, gradient, partial derivative, chain rule, and cache together gives the reader a glossary to anchor on before any math arrives.

The term doing the most quiet work is 'cache.' Beginners rarely realize that backprop depends on values saved during the forward pass; naming it here primes the dependency that post 3 traces concretely and that the from-scratch code in post 4 makes literal. The partial-derivative entry also clarifies that the gradient is just the collection of these per-weight derivatives.

The flow diagram makes the forward-then-backward structure concrete by walking left to right: input enters, the forward pass computes the output, the loss measures error, and only then does the backward pass apply the chain rule to produce one gradient per weight. Seeing it as a single left-to-right-then-back pipeline turns the algorithm into a process the reader can trace.

The final node — 'gradients, one per weight' — previews the payoff. Backprop's entire deliverable is that collection of numbers, and visualizing it as the endpoint of the flow clarifies why the forward pass and loss are prerequisites rather than separate topics.

This slide answers the question hiding in the algorithm's name: why backward? The reason is that error is only observable at the output. A weight buried three layers deep contributed to the final error only indirectly, through everything downstream of it. To fairly assign that weight its share of the blame, the error signal must be propagated back through every layer it influenced.

The 'each layer hands the next-earlier layer its share of responsibility' framing is the intuition behind the chain rule applied layer by layer. It also previews the step-by-step trace in post 3, where dL/dx from one layer literally becomes the incoming signal for the previous one.

The dependency slide makes explicit that backprop is one half of a matched pair. The forward pass must run first and store its intermediate values; the backward pass then reuses exactly those cached values to evaluate the chain rule. Without the cache, there is nothing to differentiate against.

The practical implication is large and shows up in post 5: storing the whole forward pass costs memory, which is why large-model training uses tricks like gradient checkpointing. Stating the dependency plainly here prevents the common confusion where learners think backprop somehow re-derives everything on its own.

This code slide makes the chain rule tangible in the smallest possible form. For a loss that depends on an output that depends on a weight, the gradient is just the product of two local derivatives: dL/da times da/dw. Seeing it as multiplying the links of a chain demystifies the rule that the entire algorithm rests on.

The concrete derivatives — 2(a - y) for a squared error and x for a linear unit — are deliberately the same ones that reappear in post 3's hand trace and post 4's numpy build. Introducing them here in two lines means the reader meets them as simple arithmetic before they appear inside a network.

The comparison slide cements the measurement-versus-action distinction from earlier, now side by side. Backpropagation computes the gradient using the chain rule, running from output to input; gradient descent consumes that gradient and takes the actual step. They are sequential collaborators, not synonyms.

This matters because the two are so often blurred in casual explanations that say 'backprop trains the network.' Separating them cleanly here sets up the next topic teaser — Day 44's gradient descent variants — and prevents the reader from misattributing learning-rate or optimizer behavior to backprop itself.

This slide explains why backprop, specifically, was the breakthrough rather than the general idea of gradients. Gradients were well understood mathematically long before deep learning. The bottleneck was computing them at scale: the naive approach perturbs each weight and re-runs the network, costing one pass per weight.

Backprop's contribution is efficiency — it reuses shared intermediate computation to produce every gradient in roughly one extra pass, independent of the parameter count. That single property is what made training networks with millions of weights feasible, and it is the entire argument of post 2, previewed here.

This recap consolidates the five core ideas into a single screenshot-able reference. Spaced repetition of 'chain rule output to input,' 'computes gradients not updates,' 'needs cached values,' 'gradient descent does the stepping,' and 'efficient' is intentional — these exact points anchor the next four posts.

The item doing extra duty is 'computes gradients, not updates,' since it is the distinction most likely to be misremembered. Listing it alongside the efficiency point primes both the why-it-matters argument in post 2 and the clean separation of backward from step in the post 4 code.

The CTA closes the loop and points to post 2's argument. Framing the next post around why this one algorithm is the reason deep learning exists gives the reader a concrete reason to continue rather than a generic prompt to follow.

It also sets expectations for the day's arc: post 1 was the what, post 2 is the why. Naming that progression helps the reader see the day as a structured lesson rather than disconnected tips, consistent with the standard of the whole series.