How Models Learn (Intuition)

Diagrams build intuition, but nothing locks it in like watching a model learn in real code. This post is deliberately hands-on: we fit a straight line to noisy data using the exact guess-check-adjust loop from the earlier posts, first by hand so every moving part is visible, then with PyTorch so you see what the framework automates.

Everything here is runnable. Paste it, run it, change a number, run it again. That cycle of poking at live code is where the intuition stops being abstract.

The plan is intentionally tiny so the mechanism, not the model, is the star. We fit y = w·x + b — a single slope and intercept — to data that came from a known line plus noise. Because we know the true line is y = 3x + 2, we can check whether learning actually recovers it.

Doing it by hand first means no framework magic hides the gradients. Then the PyTorch version does the identical job in fewer lines, and the contrast shows you exactly which parts the library is handling for you.

First we manufacture data with a known answer, which is the best way to learn because you can verify the result. x is 100 evenly spaced points; y is the true line 3x + 2 with Gaussian noise added so it's not trivially clean.

Setting the seed makes the run reproducible. Because we built the data from a known line, success has a clear definition: after training, w should land near 3 and b near 2. Synthetic data like this is a standard sanity check before trusting a setup on real data.

This is the heart of the post: gradient descent with zero libraries doing the thinking. pred is the model's current guess; err is the raw mistake; loss is the mean squared error. dw and db are the gradients — the slope of the loss with respect to each parameter, derived by hand from the squared-error formula.

The two subtraction lines are the learning itself: each parameter steps a little against its gradient, scaled by lr. Run this 200 times and the dials walk from zero toward the true values. There is genuinely nothing more to gradient descent than these few lines.

Printing every 50 steps turns the abstract loop into something you can watch. The sample output tells the whole story: loss starts at 13.1 with w and b near zero, then collapses to roughly 0.1 as the parameters home in on 3.00 and 2.00.

That residual loss isn't failure — it's the noise we deliberately added; the model correctly refuses to overfit it. Seeing the numbers march like this is the "aha" moment. Always log your loss during training; a number that doesn't drop is your first and clearest signal that something is wrong.

This trace narrates the run as a story. We start with a flat line because w and b are both zero, giving a large loss of 13.1 — the model is very wrong. Two hundred small downhill nudges later, w and b have settled at 3.00 and 2.00.

The punchline in the comment line is the whole point: the loop recovered the true line we hid in the data, using nothing but error feedback. No one told the model the slope was 3; it discovered it by repeatedly reducing its own mistakes.

Now we rebuild the same thing in PyTorch to see what a real framework provides. We wrap the data in tensors, define a Linear(1, 1) layer — which is just our w and b with nicer plumbing — pick SGD as the optimizer with the same learning rate, and choose MSELoss as the error measure.

Notice we haven't written any gradient math. That's the trade: a few lines of setup in exchange for the framework taking over the derivative bookkeeping we did by hand above. For one parameter it's overkill; for real models it's essential.

This is the PyTorch training loop, and it's worth comparing line-for-line with the hand-written version. pred = model(X) is the guess; loss_fn computes the error; loss.backward() is where autograd computes the gradients we previously derived ourselves; opt.step() applies the w -= lr * grad nudge.

The only unfamiliar line is opt.zero_grad(), which clears gradients from the previous step so they don't accumulate. Strip that detail away and this is the exact same guess-check-adjust loop — just with the calculus delegated to the library.

This slide names the single most important thing PyTorch did for you. In the from-scratch version we personally worked out dw and db with calculus. loss.backward() computes those gradients automatically, for any model, by tracing the operations that produced the loss and applying the chain rule.

That's the feature that makes deep learning practical. Hand-deriving gradients for a two-parameter line is easy; doing it for a network with millions of parameters is hopeless. Autograd does it for free, and opt.step() then applies the same simple update rule you wrote by hand.

The fastest way to truly understand training is to deliberately break it, so this slide invites experiments. Crank the learning rate to 2.0 and watch the loss explode as each step overshoots. Drop it to 0.0001 and watch it barely budge. Remove the noise and the fit becomes near-perfect. Add more steps and w, b lock precisely onto 3 and 2.

Each experiment maps a concept from the previous post onto a number you can change. This is the cheapest, highest-value learning you can do: break the toy, observe the failure, and connect it back to the mechanism.

This final comparison summarizes when to use which approach. Writing gradient descent from scratch is unbeatable for understanding — you see every nudge — but it becomes tedious and error-prone past a handful of parameters. PyTorch hands the gradient math to autograd and scales effortlessly to enormous models.

The key insight is that they are the same loop underneath. Frameworks don't change what learning is; they automate the bookkeeping. Knowing the from-scratch version means you can read, debug, and trust the framework version instead of treating it as a black box.

You've now watched a model learn end to end, by hand and with a framework. The last post in this set covers the traps: the ways this simple loop quietly learns the wrong thing while looking perfectly healthy in your notebook — and the quick checks that catch each one.