Gradient Descent, Visually

This post is deliberately code-heavy, because gradient descent only truly clicks when you've written one and watched the numbers move. We use nothing but NumPy and Matplotlib — no framework hiding the mechanics — so every line of the algorithm is visible. We'll minimize a simple function, fit a real regression, plot the loss, visualize the descent path, and compare learning rates.

The instruction is simple: don't just read it, run it. Change the numbers, break it on purpose, and watch what happens. That experimentation is where durable intuition comes from.

This first example minimizes f(x)=x², the simplest possible bowl, whose minimum sits at x=0. The gradient is 2x, so the update x -= eta * grad(x) repeatedly nudges x toward zero. We also append each x to a history list so we can plot the path later.

Starting at x=5 with η=0.1, after thirty steps x is about 0.0008 — essentially at the minimum. Watch how the steps shrink as x approaches zero: the slope 2x gets smaller, so each update gets smaller, and the descent gently settles rather than overshooting.

Here we fit a real linear regression y = wx + b by descent alone. We generate data from the true relationship 3x + 2 with a little noise, initialize both parameters to zero, and loop a thousand times. Each iteration computes the squared-error loss (logged for plotting), then the gradients for w and b, then steps both downhill.

The gradients are the calculus from post 3: dL/dw = mean(2·err·x) and dL/db = mean(2·err). After a thousand steps w and b land near the true 3.0 and 2.0. This is the canonical 'gradient descent actually fits models' demonstration — small enough to verify, real enough to matter.

Plotting the recorded losses against step number gives you the loss curve, the single most important diagnostic in training. We use a logarithmic y-axis because loss often drops fast early then slows dramatically; a log scale makes that late, fine-grained progress visible instead of hugging the x-axis.

When you run this you should see a curve that falls steeply at first and then levels off as the parameters approach their optimum. Recognizing that classic shape — fast then flattening — is exactly the skill you'll use to judge whether a real training run is behaving.

This visualization plots the x² bowl and overlays the actual sequence of points the optimizer visited, colored by step order so you can literally watch the ball roll into the valley. Early points sit high on the walls of the bowl; later points cluster at the bottom near x=0.

Seeing the discrete steps on the continuous curve makes the algorithm tangible. You can see them bunch up near the minimum as the slope flattens. This is the picture from post 1 — the ball on the hill — rendered from your own code rather than imagined.

This pipeline summarizes what every one of the code examples is doing under the hood: initialize the parameters and learning rate, compute the gradient (the slope at the current parameters), update by stepping against that gradient, and record the loss so you can inspect the run afterward. Then loop.

It's the same four-step rhythm from post 3, now mapped onto the concrete code you've been writing. Whether the surface is x² or a regression loss, the implementation follows this identical pattern — which is exactly why understanding one transfers to all of them.

This experiment runs the same descent on x² with three different learning rates and prints where each ends up after twenty steps. With η=0.01 the value is still around 3.3 — correct direction but far too slow. With η=0.1 it reaches about 0.057, close to the minimum. With η=1.01 it diverges, each step overshooting further than the last until the number explodes.

The lesson is visceral: same algorithm, same starting point, one knob changed, three completely different outcomes. This single experiment teaches more about learning rates than any rule of thumb, and it's why post 5 treats a bad learning rate as the number-one failure mode.

Reading the printed output ties the numbers to the behavior. η=0.1 glides smoothly to zero — the well-tuned case. η=0.01 barely moves in twenty steps; it would get there eventually but wastes enormous time, which is exactly what 'too slow' feels like on a real model that takes hours per run. η=1.01 overshoots and blows up because each step's correction is larger than the distance to the minimum.

The boundary case is instructive: for f(x)=x², descent converges only when η is below 1, oscillates at exactly 1, and diverges above it. Most real surfaces have an analogous stability threshold you discover by experiment.

These tweaks turn the examples into a playground. Changing the start point shows that for a simple bowl you reach the same minimum regardless of where you begin — a property that breaks on non-convex surfaces, which is a great thing to discover yourself. Adding momentum (v = 0.9·v − η·g, then x += v) shows how the heavier-ball idea accelerates descent.

Swapping mean squared error for absolute error changes the gradient and the behavior near the minimum. And printing the gradient each step lets you watch it shrink toward zero as you converge — a direct, numeric view of 'the ground flattening out' that the metaphor describes.

The most common real-world bug in hand-rolled descent is forgetting to scale features. If your inputs span the thousands while the target sits near zero, the gradients for different parameters differ by orders of magnitude, and any single learning rate is simultaneously too big for one weight and too small for another. The result is divergence or painfully slow progress.

The instinctive fix — shrinking the learning rate until nothing explodes — treats the symptom and leaves training crawling. The real fix is to standardize features first (subtract the mean, divide by the standard deviation) so every parameter sees a comparable slope. A huge fraction of 'my model won't converge' problems are exactly this, which is why post 5 returns to it in depth.

That's gradient descent you can actually run: minimize a function, fit a model, plot the curve, watch the path, and feel what the learning rate does. Code makes it real in a way prose can't. The final post turns to the failure modes — the specific ways this breaks and the visual tells that let you diagnose each one fast.

Save this so the runnable snippets are handy the next time you want to sanity-check the optimizer with your own eyes.