Calculus & Derivatives

This is the hands-on post, and the framing is that there are three distinct ways to obtain a derivative in code, each with different trade-offs. Numeric differentiation approximates the slope with finite differences. Symbolic differentiation manipulates the formula to produce an exact derivative expression. Automatic differentiation records the operations and applies the chain rule mechanically.

The punchline you'll arrive at is that automatic differentiation — autograd — is the method that powers all of deep learning, because it is exact like symbolic but cheap enough to scale to millions of parameters. Running all three and watching them agree is the most convincing way to understand why.

Numeric differentiation is the most intuitive method: approximate the derivative directly from its definition as a slope. This snippet uses the central difference formula, (f(x+h) − f(x−h)) / (2h), which is more accurate than the one-sided forward difference because it cancels first-order error terms.

Applied to f(x) = x³ − 2x at x = 2, it returns ~10.0, matching the true derivative 3x² − 2 = 10. The method's appeal is that it needs nothing but the ability to evaluate the function — no formula, no framework. Its weakness, covered in the mistakes post, is that it's approximate and becomes prohibitively expensive when you have many inputs, since you'd perturb each one separately.

Symbolic differentiation manipulates the mathematical expression itself to produce an exact derivative formula, the way you would by hand. SymPy does this: we declare x as a symbol, write f = x³ − 2x, and sp.diff returns the literal expression 3x² − 2. We can then substitute x = 2 to evaluate it as exactly 10.

This is wonderful for understanding and for deriving clean closed-form results. Its drawback for large models is 'expression swell' — the symbolic formulas for a deep network's derivatives explode in size and become impractical to build and evaluate. Symbolic differentiation is a learning and analysis tool, not the engine of a billion-parameter model.

Automatic differentiation is the method deep learning actually runs on, and PyTorch's autograd is the canonical implementation. You mark a tensor with requires_grad=True, perform ordinary operations to compute f, then call f.backward(). PyTorch has silently recorded every operation in a graph and now applies the chain rule backward through it to fill in x.grad.

The result here, 10.0, is exact to machine precision — no formula was written and no finite-difference approximation was made. The crucial property is efficiency: autograd computes gradients for all inputs in roughly the cost of one backward pass, which is what makes training networks with millions of parameters feasible. This single mechanism is the workhorse behind every framework.

The comparison crystallizes why autograd won. Numeric differentiation is easy and needs only function evaluations, but it's approximate (subject to rounding and truncation error) and slow when there are many inputs, since each input requires its own perturbation. Automatic differentiation is exact to machine precision and computes the full gradient in roughly a single backward pass regardless of how many parameters there are.

That scaling property is decisive. A modern network has millions to billions of parameters; perturbing each one numerically would be hopeless, and symbolic expressions would explode. Autograd sidesteps both problems, which is exactly why it underpins all of deep learning.

This snippet shows autograd handling multiple variables at once, which is the realistic case. We create a weight vector w = [1, 2] with requires_grad=True and compute f = w0² + 3·w0·w1. After backward(), w.grad holds the full gradient.

The result [8, 3] matches the hand-derived partials 2w0 + 3w1 = 8 and 3w0 = 3 — exactly the calculation done manually in the previous post, now automatic. This is the payoff of autograd: whether the function has two variables or two billion, the same backward() call produces every partial derivative simultaneously, with no manual differentiation required.

This is a complete, real gradient-descent loop using autograd, tying together everything in the day. We minimize (w − 4)², whose minimum is at w = 4. Each iteration: compute the loss, call loss.backward() to get dloss/dw, then update w against its gradient inside torch.no_grad() (so the update itself isn't tracked), and finally zero the gradient.

After 50 steps w converges to ~4.0. This is, in miniature, exactly what training a neural network does — only the loss is more complex and the weights number in the millions. Three details matter and recur in every real training loop: backward() to get gradients, no_grad() around the update, and zeroing gradients each step.

The flow diagram explains how autograd actually works under the hood, demystifying the 'magic.' During the forward pass, PyTorch records every operation into a computation graph, where each node stores its local derivative. When you call backward(), it traverses that graph in reverse, multiplying local derivatives via the chain rule, and deposits the accumulated result into each tensor's .grad attribute.

This is precisely the chain rule from the mechanics post, automated and organized so that shared sub-results are computed once. Understanding that autograd is 'just' a bookkeeping system for the chain rule removes the mystery and helps you reason about edge cases like detached tensors or non-differentiable operations.

These practical notes are the habits that separate working training loops from broken ones. Use central differences rather than forward differences for numeric checks, since they're markedly more accurate. Always zero gradients each step, because PyTorch accumulates them by default and stale sums will corrupt your updates. Wrap weight updates in torch.no_grad() so the optimizer step isn't itself recorded into the graph. And remember symbolic differentiation is for insight, not for scaling to large networks.

Each of these maps to a real bug people hit. They're small disciplines, but skipping them produces failures that are hard to diagnose because nothing crashes — the numbers just quietly go wrong.

Gradient checking is a professional habit worth adopting: verify an autograd (or hand-coded) derivative against a numeric approximation. Here we compute the derivative of f(x) = x³ − 2x at x = 2 both via autograd and via a central finite difference, and confirm both read 10.0.

Whenever you implement a custom layer or a non-trivial loss with a hand-derived backward pass, this comparison is how you catch errors. If the autograd value and the numeric value disagree beyond a small tolerance, you've almost certainly made a chain-rule mistake. It's cheap insurance against silently incorrect gradients, which are among the hardest bugs to find.

The closing mistake is the single most common autograd bug: forgetting to zero gradients. PyTorch accumulates gradients — each call to backward() adds into the existing .grad rather than overwriting it. This design supports advanced patterns like gradient accumulation across mini-batches, but it means that in an ordinary loop you must call w.grad.zero_() (or optimizer.zero_grad()) every iteration.

If you skip it, each step uses a running sum of all previous gradients, so your updates are based on stale, inflated slopes and training quietly destabilizes. The symptom — loss behaving erratically with no error thrown — is exactly the kind of silent failure that wastes hours, which is why this habit is drilled into every PyTorch tutorial.

That closes the code post. You've now seen the three ways to differentiate — numeric, symbolic, and automatic — and why autograd is the one that scales to real models. You've also run a complete gradient-descent loop, which is training reduced to its essentials.

The final post of the day is the field guide to calculus mistakes: the derivative errors that silently break models, from dropped chain-rule factors to vanishing gradients, and the concrete fixes for each.