Gradient Descent, Visually

This is the mechanics post. Stripped of metaphor, gradient descent is a short loop run thousands of times, and once you see the loop the famous variants stop being intimidating — they're just choices about how much data you consume before each update. We'll make the four steps precise, show where the gradient comes from, and lay out batch, stochastic, and mini-batch descent side by side.

The goal is that you could implement this from memory, because in the next post you essentially will.

Every iteration of training performs the same four operations. Forward: feed inputs through the model to get predictions and compute the loss. Backward: differentiate the loss to get the gradient with respect to each parameter. Step: update every parameter by subtracting the learning rate times its gradient. Then move to the next batch and repeat.

That's the complete training loop — there is no fifth secret step. Whether you're training a linear model or a transformer, this forward-backward-step-repeat rhythm is identical; only the model and the data change.

This cycle diagram makes the loop's circularity explicit. Forward produces predictions and loss; backward turns that loss into gradients; the step applies those gradients to the parameters; and then you grab the next batch and close the loop. Round and round, thousands of times.

Drawing it as a cycle rather than a list emphasizes that no single pass matters much — it's the repetition that drives the parameters downhill. One iteration barely moves anything; ten thousand iterations train a model.

The gradient is the partial derivative of the loss with respect to each parameter, and backpropagation is just the chain rule applied systematically through the model's operations. For mean squared error on a line, the derivative works out to dL/dw = mean(2·(ŷ−y)·x) — plain calculus you could do by hand.

In practice you almost never differentiate manually; autograd engines in PyTorch, JAX, or TensorFlow build a computation graph and apply the chain rule for you. But knowing it's 'just derivatives' demystifies backprop and helps you reason about why gradients vanish or explode in deep stacks.

This is the loop made concrete. The outer loop runs over epochs; the inner loop runs over mini-batches of the data. For each batch we do a forward pass to get predictions and loss, a backward pass to get the gradients, and then a step that subtracts η·gradient from every parameter. That's it.

Real frameworks wrap this in objects — an optimizer, a loss function, autograd — but the skeleton is exactly these lines. If you can read this, you can read a PyTorch training loop, because it's the same five lines with nicer names.

The three flavors differ only in how many examples you use to estimate the gradient per step. Batch gradient descent uses the entire dataset, giving the exact gradient and a smooth descent path, but it's slow and memory-hungry. Stochastic gradient descent (SGD) uses a single random example per step, giving a noisy, jittery gradient that's cheap to compute and whose randomness can actually help escape shallow traps.

Neither extreme is ideal in practice, which sets up the sweet spot in the next slide. The key insight is that all three run the identical update rule — they just disagree on how much data informs each gradient.

Mini-batch descent uses a small chunk of data — commonly 32 to 256 examples — per step, and it's what nearly all real training uses. The batch is large enough that the gradient estimate is reasonably stable, but small enough to compute quickly and to parallelize efficiently on a GPU, which processes a batch's worth of math at once.

It captures the best of both extremes: more stability than single-sample SGD, far less cost than full-batch. When people say 'SGD' in deep learning today, they almost always mean mini-batch SGD. The batch size becomes another hyperparameter you tune.

This vectors diagram contrasts the descent paths. Full-batch descent (blue) takes a smooth, direct route toward the minimum because each gradient is exact. Stochastic descent (orange) takes a noisier, more erratic route because each gradient is estimated from very little data.

The noise isn't purely bad — it's a feature. The jitter lets SGD bounce out of shallow local minima and saddle points that a perfectly smooth path might settle into. Mini-batch descent lives between these two paths, inheriting a useful amount of helpful noise.

Plain descent struggles in long narrow valleys, oscillating across the steep walls while creeping slowly along the floor. Momentum fixes this by accumulating a running average of past gradients, so consistent directions build up speed while oscillations cancel out — like a heavier ball that doesn't get deflected by every bump.

Adam goes further by maintaining a per-parameter adaptive learning rate based on recent gradient statistics, so parameters with small or noisy gradients still make progress. Both are popular defaults, and crucially both are still 'step downhill' — momentum and adaptivity are refinements layered on top of the same core update.

Knowing when to stop is part of the mechanics. Convergence shows up as the training loss ceasing to decrease meaningfully, the gradient magnitude shrinking toward zero, and — most importantly — the validation loss flattening out. A common practical stopping rule is when updates fall below a tolerance or validation loss stops improving for several checks (early stopping).

Note that minimizing training loss isn't the real goal; generalization is. So you watch validation loss, not just training loss, to decide when further descent is helping the model versus just memorizing the training set.

A frequent source of confusion is conflating a step with an epoch. A step (or iteration) is a single parameter update on one batch. An epoch is one complete pass over the entire dataset, which contains many steps. People expect the loss to fall monotonically and get alarmed by the jitter between steps within an epoch.

The right habit is to watch the trend across many steps, not to react to every individual point. Mini-batch gradients are noisy by design, so per-step loss bounces around even when training is perfectly healthy. Smooth the curve or look at per-epoch averages to see the real signal.

That's the engine: a four-step loop, gradients from the chain rule, and three flavors that trade off data per step. With the mechanics clear, the next post gets your hands dirty — a from-scratch implementation in NumPy that you can run and watch the loss actually fall.

Save this so the loop and the batch/SGD/mini-batch distinction are one tap away when you're setting up your next training run.