Calculus & Derivatives

This post is the mechanics. Having established what a derivative is and why it drives learning, we now get the tools to actually compute one. The good news is that you almost never go back to the limit definition — derivatives obey a small set of mechanical rules, and you compose them.

The centerpiece is the chain rule, because a neural network is just a deep nesting of functions, and backpropagation is the chain rule applied systematically through that nesting. Master the rules here and you can differentiate any network on paper, which makes the framework's autograd far less mysterious.

The power rule is the one you'll use most: d/dx[xⁿ] = n·xⁿ⁻¹. Bring the exponent down as a multiplier and reduce the exponent by one. So x² becomes 2x, x³ becomes 3x², x¹ becomes 1, and any constant becomes 0 (a flat function has zero slope).

Combined with linearity — the derivative of a sum is the sum of the derivatives, and constant factors pass straight through — the power rule alone lets you differentiate any polynomial term by term. That covers a surprising amount of the functions you meet in basic ML math, and it's the foundation everything else builds on.

The product and quotient rules handle functions multiplied or divided together, and the key warning is that you cannot simply combine the separate derivatives. For a product, (f·g)' = f'·g + f·g' — note the two cross terms. For a quotient, (f/g)' = (f'·g − f·g') / g².

The intuition for the product rule is that when both factors change, the total change has two contributions: one from f changing while g stays, one from g changing while f stays. Naively writing (f·g)' = f'·g' drops these cross-terms and is simply wrong — a classic source of subtly broken hand-derived gradients.

The chain rule is the most important rule for machine learning. For a composed function y = f(g(x)), the derivative is dy/dx = f'(g(x)) · g'(x): differentiate the outer function evaluated at the inner one, then multiply by the derivative of the inner function. You work outside-in, multiplying as you go.

This is the entire basis of backpropagation. A neural network is a long composition — linear layer, activation, linear layer, activation, …, loss — and to get the gradient with respect to an early weight you multiply the local derivatives along the whole path back to it. Backprop is just an efficient, organized application of the chain rule, reusing shared sub-results instead of recomputing them.

The flow diagram traces the chain rule as an actual chain: the input x feeds the inner function g(x), whose output feeds the outer function f, and the final derivative dy/dx is the product f'(g)·g'(x). Reading it left to right shows how information flows forward; reading the derivative right to left mirrors how gradients flow backward in backprop.

This directional symmetry is exactly why the algorithm is called backpropagation. The forward pass computes the function; the backward pass multiplies local derivatives in reverse to assemble the gradient. The chain rule is the rule that makes the two passes line up.

This snippet verifies a chain-rule derivation against a numeric ground truth, which is a habit worth forming. We differentiate y = (3x + 1)² by setting g = 3x + 1 and f = g², giving dy/dx = f'(g)·g'(x) = 2(3x+1)·3. At x = 2 the analytic answer is 42.

The code then approximates the same derivative numerically with a tiny h and prints both, confirming they agree at ~42.0. This pattern — derive by hand, then gradient-check numerically — is exactly how practitioners catch chain-rule errors in custom layers. If the analytic and numeric values disagree, you almost certainly dropped or mis-multiplied a factor.

Partial derivatives extend differentiation to functions of several variables, which is the realistic case in ML. The partial ∂f/∂x measures how f changes as you vary x alone, treating every other input as a frozen constant. You differentiate normally, just pretending the other variables are numbers.

Collect all the partial derivatives of a function into a vector and you get the gradient, written ∇f. This is the object gradient descent actually operates on: it tells you, for each variable simultaneously, the direction and rate of steepest increase. Understanding the gradient as 'a vector of partials' demystifies the jump from single-variable calculus to the multi-dimensional optimization of real models.

This snippet computes a gradient by hand-deriving each partial and packaging them into a NumPy array. For f(x, y) = x² + 3xy, holding y fixed gives ∂f/∂x = 2x + 3y, and holding x fixed gives ∂f/∂y = 3x. At (1, 2) the gradient is [8, 3].

The practical point is that gradient descent would step this point in the direction −[8, 3] to reduce f. This is the same calculation autograd performs automatically inside a deep-learning framework, just for millions of variables at once. Doing it manually on a two-variable example builds the intuition for what the framework is doing under the hood.

This table collects the derivatives of functions you'll meet constantly in ML, worth committing to memory. The sigmoid's derivative is the elegant σ(x)(1 − σ(x)) — expressible in terms of the sigmoid itself, which makes it cheap to compute during backprop. Tanh's derivative is 1 − tanh²(x). ReLU's is a clean 1 for positive inputs and 0 otherwise. The natural log differentiates to 1/x.

These reappear everywhere because they are the activation and loss components that networks are built from. Knowing them by sight lets you reason about gradient flow — for instance, immediately seeing why sigmoid's derivative, capped at 0.25, contributes to vanishing gradients.

The pipeline diagram shows how the individual rules assemble into backpropagation. First, each node in the computation graph knows its own local derivative — how its output responds to its inputs. Second, the chain rule multiplies these local derivatives along each path from the loss back to a weight. Third, the result is the gradient for every weight in the network.

The elegance of backprop is that it computes all these gradients in a single backward sweep, reusing intermediate products rather than recomputing them for each weight. So 'training a billion-parameter model' is, at its core, this three-step pattern executed efficiently at scale.

The signature chain-rule mistake is differentiating the outer function and forgetting the inner derivative. For (3x + 1)², the outer power rule gives 2(3x + 1), but you must still multiply by the derivative of the inside, which is 3 — yielding 6(3x + 1). Stop early and your answer is wrong by a factor of 3.

In a neural network this kind of dropped factor doesn't crash anything; it silently scales a gradient incorrectly, so the affected weights learn at the wrong rate and the model underperforms in a way that's maddening to debug. Numeric gradient checking, shown earlier, is the standard defense — it catches exactly these missing-factor errors.

That closes the mechanics post. You now have the full toolkit: the power rule for polynomials, product and quotient rules for combined functions, the chain rule for nesting, and partial derivatives that assemble into the gradient. Together they let you differentiate essentially any function a model is built from.

The next post puts all of this into code three different ways — numeric, symbolic, and automatic differentiation — and shows why autograd is the method that actually scales to real neural networks.