What is a Neural Network?

This cover frames post 3 as the engine room, signaling a shift from motivation to mechanism. The one-line promise — two passes, one forward to predict and one backward to learn — names the core loop of the whole post upfront so readers know the payoff before the details arrive.

The key reframing is that training is a repeated loop of four steps, not a single mysterious event. Most learners imagine 'the network learns' as a black box; naming the forward pass, loss, backprop, and update as concrete, separable stages turns that black box into a procedure they can follow and eventually code, which is exactly what post 4 delivers.

Starting with random weights is the right first mechanical step because it establishes where training begins: from noise. The detail worth dwelling on is why the initialization is random rather than zero — identical weights would make every neuron in a layer compute the same thing and receive the same gradient, so they could never differentiate. Randomness breaks that symmetry.

The reason to ground the story here is that everything afterward is defined as movement away from this starting point. Each training step nudges these random numbers toward useful values, so understanding that the network begins as pure noise makes the gradual improvement over the loop concrete rather than magical.

The forward-pass slide explains how the network turns inputs into a prediction, and the key insight is that it is just function composition. Each layer runs the same operation — multiply by weights, add bias, apply activation — on the output of the layer before it, until the final layer emits a result. There is nothing different about it at training versus serving time; it is the same computation.

Emphasizing that the forward pass is identical during training and inference clears up a common confusion. Training adds the backward pass and weight updates around the forward pass, but the prediction step itself never changes. This connects directly back to the neuron and layer definitions from post 1, now assembled into an end-to-end computation.

The loss slide introduces the objective that gives training its direction. The crucial reframe is that 'how wrong is the model' must become a single number before it can be optimized — and that number is the loss. Cross-entropy handles classification by penalizing confident wrong answers heavily; mean squared error handles regression by penalizing large deviations.

The practical point is that training is literally the minimization of this number. Every subsequent step — backprop, gradient descent — exists only to push the loss down. Framing the loss as the thing the entire machine is trying to shrink gives readers a north star for understanding why the later steps do what they do.

This slide demystifies backpropagation, the step most beginners find intimidating. The reframe is that backprop is nothing more than the chain rule of calculus applied systematically from the output back toward the input. It computes, for every weight, how much a small change in that weight would change the loss — the gradient.

The efficiency point is worth stating: by reusing the values computed during the forward pass and working backward layer by layer, backprop calculates all these derivatives in roughly the same cost as a single forward pass, rather than perturbing each weight independently. That efficiency is precisely what makes training networks with millions of weights feasible, and it is why backprop, not the idea of gradients, was the breakthrough.

This from-scratch code slide makes the forward pass tangible by implementing a two-layer network in a few lines of numpy. Readers can see the hidden layer compute W1 @ x + b1 followed by a ReLU, then the output layer produce raw logits. Seeing the entire prediction with no framework magic is what makes the mechanics finally feel real.

Returning both the output and the hidden activation h is deliberate foreshadowing: backpropagation needs those stored intermediate values to compute gradients via the chain rule. Even without writing the backward pass here, the slide hints at why frameworks cache activations during the forward pass, connecting the code directly to the previous slide's explanation.

The gradient-descent slide explains what to do with the gradient once backprop produces it. The core rule — w = w - lr · gradient — is simple but easy to get backwards, so the slide states the logic explicitly: the gradient points in the direction of increasing loss, so stepping in the opposite direction decreases it. Repeat for every weight and the loss falls.

The practical framing is that this update is the moment learning actually happens. Forward and backward passes only gather information; the weight update is where the model changes. Doing it over many batches and epochs is what slowly transforms the random initial weights into a trained model, which is the loop the next diagram visualizes.

The learning-rate slide isolates the single most important hyperparameter. The trade-off to internalize is sharp: too high and updates overshoot the minimum or diverge entirely, too low and training crawls or stalls in a poor spot. The learning rate scales every weight update, so it governs the whole optimization's stability and speed.

The concrete range — roughly 0.1 down to 0.0001, often decayed over training — gives readers an actionable starting point, and the note about decay previews real-world practice where the rate shrinks as the model converges. This is the same shrinkage intuition that governed the previous day's gradient boosting, reinforcing a transferable principle: small, well-sized steps generalize better than big greedy ones.

The cycle diagram captures the training loop in four repeating steps: forward to predict, compute the loss, backprop to get gradients, and update the weights. Visualizing it as a cycle reinforces that there is one procedure repeated thousands of times, not a different operation per stage of training.

Seeing the loop as a closed cycle also clarifies why training takes time and data. Each pass through the cycle improves the weights only slightly, so the model must traverse the loop over many batches and epochs to converge. This visual is the portable mental model a reader can carry into the code in post 4, where each node becomes a line of PyTorch.

This code slide maps the four-step loop onto real PyTorch, making the abstract cycle executable. Each line corresponds to a node in the diagram: model(x) is the forward pass, loss_fn measures error, loss.backward() runs backprop to fill the gradients, and optimizer.step() applies the gradient-descent update.

The zero_grad() call is highlighted on purpose because it is the step with no diagram node and the one beginners forget — PyTorch accumulates gradients by default, so they must be cleared each iteration. Seeing the canonical five-line loop here both cements post 3's mechanics and sets up post 4's full pipeline and post 5's most common mistake.

The recap orders the six mechanics into the exact sequence training follows, giving readers a single screenshot-able summary of how a network learns end to end: initialize randomly, forward pass, measure loss, backprop, step the weights, repeat. Reciting this sequence is enough to follow any training code.

This ordered list doubles as a study aid for post 4, where these mechanics appear as runnable PyTorch lines. A reader who can recite the loop can read the build line by line, recognizing model(x) as the forward pass and optimizer.step() as the weight update, rather than treating the framework calls as opaque incantations.

The CTA transitions from mechanism to hands-on practice. Having understood how the forward pass, loss, and backpropagation fit together, the reader is primed to watch it happen in real code — defining layers, choosing a loss and optimizer, and running the loop end to end.

Naming the next post's deliverable — a full runnable build — sets a concrete expectation that post 4 is a buildable pipeline, not more theory, which is the right reward after a math-heavy post.