Calculus & Derivatives

This final post is a field guide to the calculus mistakes that actually bite practitioners. The framing matters: because derivatives are mechanical, the errors are mechanical and predictable too. A surprising fraction of 'my model won't train' problems trace back to a slope that was dropped, vanished, assumed, or accumulated incorrectly.

Knowing these failure modes in advance turns days of confused debugging into quick recognition. Each slide pairs a specific mistake with why it happens and how to fix it, so you can pattern-match against your own stuck training runs.

The first and most common hand-derivation mistake is dropping the chain-rule's inner factor. When differentiating a composed function, you must multiply by the derivative of the inside. d/dx[sin(2x)] is 2·cos(2x), not cos(2x); the factor of 2 comes from the inner function 2x.

In backpropagation, this kind of dropped factor doesn't raise an error — it silently scales an entire layer's gradient incorrectly. The affected weights then learn at the wrong effective rate, and the model underperforms in a way that's frustrating to track down. This is precisely why gradient checking exists: a numeric comparison instantly flags a derivative that's off by a constant factor.

The second mistake is conflating the function's value with its slope. A large loss does not imply a large gradient, and a small loss does not mean training is done. You can be parked on a high but flat plateau where the value is big yet the derivative is nearly zero, so the optimizer barely moves.

The discipline is to reason about value and slope as separate quantities. When the loss is high but not decreasing, the right question is 'what is the gradient doing?' rather than 'why is the loss still high?' Keeping the two concepts distinct is what lets you correctly diagnose plateaus, saddle points, and vanishing gradients instead of misreading them.

This comparison nails down the value-versus-slope distinction. The value f(x) tells you how high you are — it's what the loss readout shows — but on its own it says nothing about which direction to move. The slope f'(x) tells you which way and how fast loss changes, and critically it's zero at both peaks and valleys.

That last point is a frequent trap: a zero gradient does not mean you've found a minimum. It could be a maximum, a saddle point, or a flat region. Optimizers and practitioners alike have to use more than 'the gradient is small' to conclude that training has actually converged to something good.

The third mistake is assuming every function is differentiable everywhere. ReLU has a sharp kink at zero where no single tangent slope exists. Absolute value, max, and step functions share this problem of non-smooth or undefined points. Frameworks paper over it by choosing a subgradient — PyTorch defines ReLU'(0) = 0 by convention — so code runs fine.

But if you derive gradients by hand, you can mistakenly assign a slope at a point where the derivative genuinely doesn't exist, or assume smoothness that isn't there. Knowing which of your activations and operations have kinks tells you where to be careful and why the framework's chosen subgradient is a pragmatic convention rather than a true derivative.

This snippet makes the ReLU kink concrete. We evaluate ReLU at exactly x = 0 and call backward(); PyTorch reports a gradient of 0.0. That's not the 'true' derivative — there isn't one at the kink — but a chosen subgradient that keeps computation well-defined.

The comment summarizes the actual situation: the slope is 0 for negative inputs, 1 for positive inputs, and undefined exactly at zero. In practice this convention rarely causes problems because inputs land exactly on zero with negligible probability, but it's worth understanding that the framework is making a deliberate choice at the non-differentiable point rather than computing a real derivative.

The fourth mistake is using saturating activations in deep networks and being surprised when learning stalls. Sigmoid and tanh flatten out at their extremes, so their derivatives approach zero there. Backpropagation multiplies the local derivatives along the path from loss to weight, so stacking several saturating layers multiplies many small numbers together.

The product collapses toward zero before it reaches the early layers, which therefore receive essentially no learning signal and stop improving — the vanishing gradient problem. This is the core calculus reason ReLU and its relatives largely replaced sigmoid and tanh in hidden layers: ReLU's derivative is 1 for active units, so it doesn't shrink the gradient as it flows back.

This flow diagram traces the mechanism behind vanishing gradients step by step. Each saturating layer contributes a small local slope, say around 0.2. The chain rule multiplies these across layers, so the product becomes 0.2 × 0.2 × 0.2 and so on. After enough layers the gradient reaching the early layers is effectively zero.

The diagram's value is showing that vanishing gradients are not a mysterious training quirk but a direct, predictable consequence of multiplying many sub-one slopes. Once you see it as a product collapsing geometrically, the fixes — non-saturating activations, residual connections that add a shortcut path, normalization — all make intuitive sense as ways to keep that product from shrinking.

This snippet quantifies the vanishing problem in one line. The sigmoid's derivative peaks at 0.25, so as a best case each layer multiplies the backpropagated gradient by at most 0.25. Raised to the tenth power for a ten-layer stack, that's about 9.5e-07 — roughly a millionth.

The interpretation is stark: the gradient reaching the earliest layer is a millionth of the signal at the output, and that's the optimistic case using the maximum slope. In reality it's worse. This is why, before residual connections and better activations, very deep networks were nearly impossible to train — the calculus simply starved the early layers of any learning signal.

The fifth mistake concerns numeric differentiation's sensitivity to step size, which is a Goldilocks problem. If h is too large, you're measuring the slope of a secant over a wide gap rather than the tangent, introducing truncation error. If h is too small, the subtraction f(x+h) − f(x−h) cancels almost all the significant digits and floating-point rounding error dominates.

The sweet spot for central differences is typically around 1e-6 to 1e-8. This matters whenever you gradient-check by hand or implement numeric derivatives directly. Picking h carelessly can make a perfectly correct analytic gradient appear wrong, or mask a real bug, simply because the numeric reference itself was inaccurate.

These are the fixes that actually resolve the mistakes above. Use ReLU or GELU to keep slopes from saturating and vanishing. Apply batch or layer normalization to stabilize the scale of activations and gradients. Use gradient clipping to cap exploding slopes. Always zero gradients before backward() so they don't accumulate. And gradient-check any hand-coded derivative numerically to catch dropped chain-rule factors.

The unifying theme is that healthy training is largely a matter of keeping gradients well-behaved — finite, non-vanishing, non-exploding, and correctly computed. Most of these techniques are, at heart, interventions on the calculus to keep the slopes in a usable range.

The final mistake is the most seductive: treating a flat loss curve as proof of convergence. A loss that stops moving can indeed mean a true minimum — but it can equally mean a saddle point, a wide plateau, or vanished gradients that have starved learning. The flat curve alone doesn't distinguish these.

The practical defense is to inspect the gradient norm rather than trusting the loss curve in isolation. In a deep network, gradients that are near zero everywhere are suspicious — more likely a sign of vanishing gradients or a bad initialization than of genuine success. Reading the slopes, not just the value, is the recurring lesson of this entire day, and it's where good debugging starts.

That wraps Day 12. You now have the full picture of derivatives in machine learning: what they are, why they drive learning, how to compute them by rule and in code, and the mistakes that quietly break models. The recurring thread across all five posts is the same: pay attention to the slope, not just the value.

Next in the Math for ML track, we move from slopes to areas — integrals and the area-under-the-curve intuition that complements everything you just learned about derivatives.