PyTorch in 8 Slides

This is the mechanics post, and the cover sets the tone: backward() is not magic, it's bookkeeping plus the chain rule. Demystifying it is valuable because autograd is the piece most learners treat as an impenetrable black box, which leaves them unable to debug gradient problems.

The through-line is the computation graph: built silently during the forward pass, replayed in reverse during the backward pass. Once you see the graph, every gradient behavior — including the bugs in post 5 — becomes explainable.

The forward pass doing double duty is the key idea. When you flag a tensor with requires_grad=True, PyTorch begins recording. Every operation you apply produces a new tensor that carries a grad_fn: a reference to the operation that created it, which knows both how to compute its local derivative and which parent tensors it came from.

String these together and you have a directed acyclic graph running from your inputs to the final scalar loss. Crucially, you didn't write any of this graph-building code — it's a side effect of doing ordinary math on tracked tensors. The graph is the recipe that backward() will later run in reverse.

This snippet exposes the otherwise invisible graph. We create a leaf tensor x with requires_grad, compute y = x**2 + 3x, and print y. Notice the output isn't just a number — it's a tensor that also reports grad_fn=<AddBackward0>, naming the last operation that produced it.

That grad_fn is the entry point to the recorded graph. Following it backward would reach the PowBackward node for x**2 and the multiplication for 3x, down to the leaf x. Seeing this printed makes the abstract 'graph' concrete: it's literally attached to the tensors you're working with.

The flow diagram traces the tiny example as an explicit graph. The leaf x feeds two branches — one squares it (PowBackward), one scales it (the 3x term) — and an addition node combines them into the output y. Each non-leaf node stores the function needed to compute its local gradient.

Real models have millions of such nodes, but the structure is identical: a graph from leaves (parameters and inputs) up to a single scalar loss. Visualizing the four-node toy version builds the intuition that scales directly to a hundred-layer network.

This is the heart of autograd. backward() starts at the output with an implicit gradient of 1 and walks the graph in reverse. At each node it multiplies the gradient flowing in by that node's local derivative and passes the result to the node's parents — this is precisely the chain rule executed mechanically.

Because it sweeps from the single output back to all inputs in one pass, it's called reverse-mode automatic differentiation. The beauty is that one backward call computes the gradient of the loss with respect to every parameter simultaneously, which is exactly what gradient descent needs.

The payoff snippet: after y.backward(), x.grad holds dy/dx. For y = x^2 + 3x the derivative is 2x + 3, which at x=2 equals 7 — and that's exactly what prints. Verifying autograd against hand calculus on a simple function is the single best way to trust it.

This tiny check is worth doing yourself once. It converts belief into knowledge: you've seen the framework reproduce a derivative you computed by hand, so you know the mechanism is just calculus, not magic.

Where gradients land matters for the training loop. After backward(), every leaf tensor that had requires_grad=True now carries its gradient in .grad. Model parameters are precisely such leaves — when you build an nn.Module, its weights are leaf tensors with requires_grad on.

So after backward(), param.grad for each parameter tells the optimizer which way to nudge that weight to reduce the loss. By default the graph is freed after the backward pass to save memory, which is why a second backward() on the same graph errors unless you pass retain_graph=True.

This slide explains the design choice behind backpropagation. Neural networks have a peculiar shape: a single scalar loss output, but millions of parameter inputs. Reverse-mode autodiff computes the derivative of one output with respect to all inputs in a single backward sweep, costing roughly the same as one forward pass.

Forward-mode autodiff does the opposite — one input with respect to all outputs — which would require a separate pass per parameter, millions of them. That asymmetry is the entire reason every deep-learning framework backpropagates rather than forward-propagates gradients. It's not arbitrary; it's the only tractable choice at this scale.

This is the conceptual root of the most common PyTorch bug, so it earns a slide here and a fuller treatment in post 5. PyTorch accumulates gradients: each backward() adds to whatever is already in .grad rather than overwriting it.

That's a deliberate feature — it lets you split a large batch across several backward passes and sum the gradients, simulating a bigger batch than fits in memory. But it means that in a normal loop you must explicitly clear .grad each step, or last step's gradients contaminate this step's. The mechanism and the discipline it demands go hand in hand.

This snippet shows the canonical four-line ordering that every correct PyTorch loop follows. First zero_grad() clears stale gradients. Then the forward pass and loss computation build a fresh graph. Then backward() populates .grad. Finally step() applies the optimizer's update using those gradients.

The order is not arbitrary: you must clear before backward (or you accumulate), and you must backward before step (or you'd update with empty gradients). Memorizing this sequence is the practical distillation of everything in this post, and it's the skeleton the next post fleshes out into a full script.

The CTA hands off to the hands-on post. Having seen how gradients are computed in isolation, readers are ready to see them embedded in a complete, runnable training loop with real data, a real model, and an optimizer.