Gradient Descent, Visually

This closing post is a field guide to how gradient descent fails. The good news is that the failures are not random — there's a small, well-known catalog, each with a characteristic shape on the loss curve and a standard fix. Once you can match a symptom to a cause, debugging a stalled or diverging training run stops being guesswork and becomes a quick diagnostic checklist.

We walk through the big five: learning rate too high, too low, getting trapped, unscaled features, and vanishing/exploding gradients — then close with a checklist that catches most of them.

A learning rate that's too high is the most common and most dramatic failure. Each step overshoots the minimum, landing further up the opposite wall of the valley, so corrections grow instead of shrink. The loss oscillates wildly and then blows up to infinity or NaN, often within the first handful of steps.

The visual tell is unmistakable: a loss curve that spikes upward or goes erratic almost immediately. The fix is equally simple — divide the learning rate by ten and try again. If it still diverges, divide again. Tuning η downward is the first move whenever loss refuses to behave at the start.

The opposite failure is subtler and easy to miss. A learning rate that's too small makes correct but glacial progress — loss inches downward, and you can burn hours of compute making microscopic improvement. Nothing looks 'wrong,' which is exactly why it wastes so much time.

The tell is a nearly flat curve with a barely perceptible downward slope. The fixes: raise the learning rate, use a schedule that warms up then decays it, or switch to an adaptive optimizer like Adam that effectively tunes the step size per parameter. The art is finding the largest η that still descends smoothly.

These bars contrast the three learning-rate regimes on one axis: how fast loss falls. Too low barely moves the needle — it descends, but at a crawl. The good rate produces a strong, smooth drop. Too high actually scores worst because the run diverges and loss goes up, not down.

The shape of this chart — a sweet spot flanked by failure on both sides — is why learning-rate tuning is usually a log-scale sweep (try 1e-1, 1e-2, 1e-3, 1e-4) rather than fine adjustments. You're hunting for the regime, then refining within it.

Real loss surfaces for non-convex models aren't single clean bowls; they have multiple valleys of differing depth (local minima) and saddle points that are flat in some directions but slope down in others. Plain gradient descent, seeing a near-zero gradient, can stall in a local minimum or linger on a saddle, mistaking it for the bottom.

In high-dimensional deep networks, saddle points turn out to be far more common than bad local minima, and the practical escape mechanisms are the noise from mini-batch gradients plus momentum, which together provide enough jitter and inertia to roll off flat spots and continue downhill. This is one reason pure full-batch descent is rarely used.

This flow shows the escape mechanism concretely. You land in a local minimum where the gradient is roughly zero but the loss isn't actually the lowest achievable. Adding noise — via the randomness of mini-batch sampling — plus momentum's accumulated velocity perturbs you out of that shallow basin, letting you roll toward a deeper, better valley.

The practical implication is reassuring: you usually don't need exotic global-optimization methods. The everyday combination of mini-batch SGD with momentum handles most of the trapping problems that worry beginners, which is why it remains the default recipe.

Unscaled features deform the loss surface into a long, narrow ravine. When one feature ranges 0–1 and another ranges 0–100000, the loss changes steeply along one parameter axis and gently along another. Descent then zig-zags across the steep walls of the ravine while making painfully little progress along its length.

The fix isn't a clever optimizer — it's preprocessing. Standardizing each feature to mean zero and unit variance rounds the surface out, so the gradient points more directly at the minimum and a single learning rate works for all parameters. This is the highest-leverage, lowest-effort fix in the whole list.

This snippet is the standardization fix in two lines: subtract each column's mean and divide by its standard deviation, giving every feature mean zero and unit variance. The comment spells out the payoff: a round loss surface where descent goes straight toward the minimum instead of zig-zagging, and a single learning rate that works across all weights.

Do this before training, every time you have features on different scales. It's such a reliable fix that 'did you standardize?' should be your first question whenever a model trains slowly or refuses to converge, right alongside 'is your learning rate sane?'

Vanishing and exploding gradients are the deep-network version of the learning-rate problem, arising from the chain rule. Backpropagation multiplies gradients layer by layer, so if those per-layer factors are consistently below one, the product shrinks toward zero and the early layers stop learning; if they're consistently above one, the product blows up and the loss goes to NaN.

The standard fixes are architectural rather than just tuning η: careful weight initialization (Xavier, He), activation functions like ReLU that don't squash gradients, normalization layers (batch/layer norm) that keep activations well-scaled, and gradient clipping to cap the size of any single update. Together these keep gradients in a healthy range through very deep stacks.

This decision tree is the diagnostic flow to run when training misbehaves. First question: is the loss rising or hitting NaN? If yes, you're almost certainly diverging — the learning rate is too high or gradients are exploding, so lower η and clip gradients. If no, ask whether the loss is barely moving; if so, the rate is too low or features are unscaled, so raise η and standardize.

If neither — loss is dropping smoothly — you're healthy, keep training. Following this tree turns a vague 'training isn't working' into a specific, actionable diagnosis in seconds, which is exactly the payoff of understanding the failure modes.

This checklist condenses the whole post into habits. Always plot the loss curve first — it tells you which failure mode you're in before you change anything. Standardize features before training to round out the surface. Sweep learning rates by factors of ten to find the right regime quickly. Clip gradients if they spike to prevent blowups. And check your data and loss for NaNs, since a single bad value can poison an entire run.

Run through this list and you'll catch the overwhelming majority of training problems without resorting to guesswork. It's the practical distillation of everything in this five-post arc.

That completes the gradient-descent arc: the picture, the why, the mechanics, the code, and the failure modes. You now have both the intuition to reason about training and the checklist to debug it. Next week builds on this foundation with the math that makes every model trainable in the first place.

Save this field guide so the symptom-to-fix map is one tap away the next time a loss curve does something alarming.