What is a Neural Network?

This cover names the defining hazard of neural networks: they fail silently. A network can train to a flat loss, memorize its data, or learn nothing at all without ever throwing an error. Framing these as expected dangers to guard against, rather than rare bugs, sets the right defensive mindset for the whole post.

The 'common mistakes' angle works best as a failure manual — each slide is a specific, nameable error plus its fix. The cover lists the lineup so readers know they are getting six concrete traps, not vague cautions, matching the depth standard the series holds to.

Forgetting optimizer.zero_grad() leads because it is the single most common PyTorch-specific bug and it is invisible until your model misbehaves. The mechanism is precise and worth stating: PyTorch accumulates gradients into .grad by default, adding to whatever is already there. Without clearing them each iteration, you sum gradients across batches, producing updates that explode or wander.

The fix is a single line placed before backward, every iteration. Naming exactly where it goes — before loss.backward() — matters because the order is what makes it correct. This connects straight to post 3's loop and post 4's training code, where zero_grad appeared in its proper place; here we explain what breaks when it is missing.

This code slide makes the zero_grad fix tangible by showing the correct loop with the easy-to-forget line called out in a comment. Seeing zero_grad as the first statement inside the batch loop, before the loss is even computed, cements the placement that prevents the bug.

Pairing the warning slide with runnable code follows the post's pattern of never leaving a problem unsolved: the reader gets both the diagnosis and the exact prescription. The comment 'fills fresh gradients' on the backward line reinforces why clearing first is necessary — backward writes a clean gradient only into a zeroed buffer.

A wildly wrong learning rate is the second mistake because it masquerades as 'the model is broken' when it is really one number out of range. The mechanism splits two ways: too high and the loss diverges to NaN or oscillates forever, too low and it barely moves across many epochs. Both produce a useless model with no error message.

The fix is a diagnostic habit, not a fixed value: start around 1e-3 with Adam, watch the loss curve, and adjust by factors of ten. A diverging loss says lower it; a stubbornly flat loss says raise it or train longer. This teaches readers to read the loss curve as the primary instrument for tuning, which is the skill the next diagram develops.

The compare diagram turns the learning-rate warning into something readers can recognize by sight. A bad rate shows up as a loss that goes to NaN, bounces erratically, or barely descends — all signatures of trouble. A good rate produces a smooth, steady descent that plateaus near a minimum with training and validation curves tracking together.

Teaching readers to read the shape of the loss curve gives them a tool they will use on every model they ever train. The curve is the single richest diagnostic in deep learning: its slope, smoothness, and the gap between train and validation lines reveal learning rate problems, convergence, and overfitting at a glance.

Feeding unscaled inputs is a subtle mistake that stalls training without any obvious symptom. The mechanism is that when features live on wildly different scales — one ranging 0 to 1, another 0 to 100000 — the large-scale feature dominates the gradient, so the optimizer effectively ignores the others and training wobbles or stalls. Neural nets implicitly assume roughly standardized inputs.

The fix is to standardize features to zero mean and unit variance, or normalize to a 0-to-1 range, before training. The critical detail, expanded in the code slide, is to compute the scaling statistics on the training set only and reuse them on test data — otherwise you leak information from the test set into preprocessing, a quieter cousin of the leakage problems covered earlier in the series.

This code slide implements the input-scaling fix with scikit-learn's StandardScaler and emphasizes the one detail people get wrong: fitting on the training set only. The scaler learns the mean and variance from X_train, then transforms both train and test with those same statistics.

The comment 'reuse train stats' on the test transform is the load-bearing instruction. Fitting a fresh scaler on the test set, or on the full dataset before splitting, leaks information and inflates your reported performance. Showing the correct pattern in code prevents a mistake that is easy to make and hard to notice, since the model will still appear to train fine.

Overfitting with no regularization is the classic capacity failure, and on a network with thousands of weights it is the default outcome without countermeasures. The mechanism is that the model has more than enough parameters to memorize the training set, driving training loss toward zero while validation loss climbs — it has learned the noise, not the signal.

The fixes are a toolkit rather than a single switch: dropout randomly disables units to prevent co-adaptation, weight decay (L2) penalizes large weights, early stopping halts training at the validation minimum, and more data is the most reliable remedy of all. The diagnostic is the widening gap between training and validation loss, which the learning-rate diagram already trained the reader to watch.

This code slide implements two of the regularization fixes concretely: a Dropout layer that zeros 30% of units during training, and weight_decay in the Adam optimizer for L2 regularization. Seeing both in a few lines shows how cheap regularization is to add relative to its benefit.

The inline comments tie the code to the concepts — 'randomly drop 30% of units' and 'L2 regularization' — so the reader connects the API arguments to the ideas from the previous slide. Note that dropout's behavior depends on the eval() switch from post 4: it is active during training and disabled at test time, which is exactly why forgetting eval() (the next mistake) is dangerous.

Trusting training accuracy is a mistake of evaluation rather than modeling, and it is insidious because a memorizing model looks spectacular on the data it trained on. A 99% training score tells you nothing about generalization; the only number that matters is performance on data the model never saw.

The fix has two parts. First, always hold out validation and test sets and judge on those, as post 4 modeled. Second — the part beginners miss — switch to model.eval() with torch.no_grad() at test time so dropout is disabled and batchnorm uses its running statistics. Skipping eval() leaves dropout active during evaluation, silently corrupting the very numbers you are trying to trust, which links this mistake directly to the regularization slide.

Vanishing gradients and dead ReLUs close out the post as the failure modes specific to depth, tying back to post 2's note that better activations and initialization unlocked deep learning. The mechanism for vanishing gradients is that sigmoid and tanh squash their inputs into a narrow range, so their derivatives are tiny, and multiplying many tiny derivatives through the chain rule shrinks the gradient toward zero before it reaches the early layers — which therefore barely learn.

Dead ReLUs are the flip side: a too-large update can push a neuron into a state where it always outputs zero, so its gradient is always zero and it never recovers. The fixes are the standard modern toolkit — ReLU and its variants instead of saturating activations, sane weight initialization, normalization layers, and a moderate learning rate. This previews the next day's deep dive on activation functions.

The recap consolidates all six fixes into a single screenshot-able checklist, turning the post into a pre-flight check the reader can run before trusting any neural network: clear gradients, sane learning rate, standardized inputs, regularization, held-out evaluation, and good activations plus initialization.

The list is intentionally portable. Several items — held-out evaluation, input scaling with train-only statistics, regularization — apply to nearly every model, not just neural nets, while the framework-specific ones like zero_grad and eval()/no_grad are the habits that prevent the most silent damage in PyTorch specifically.

The CTA closes both this post and the day, pointing forward to Activation Functions as the natural sequel. The framing is deliberate: post 1 established that the nonlinearity is the whole trick, post 5 showed how the wrong activations cause vanishing and dead-neuron problems, and the next topic examines ReLU, sigmoid, and softmax in depth and when to use each.

This creates a clean narrative arc across the day and a strong hook into Day 42. The reader finishes understanding not just what a neural network is and how it learns, but how easily it goes wrong and why the choice of activation function — the next topic — matters so much.