PyTorch in 8 Slides

The closing post is a field guide to silent PyTorch bugs. The cover names the cruelty of the genre: these mistakes don't throw exceptions. The code runs, loss just won't drop, or memory creeps up until an out-of-memory crash hours in. That makes them far harder than ordinary syntax errors.

The organizing insight is that almost all of these are conceptual, not syntactic — and each has a one-line fix once you recognize the symptom. The post pairs every bug with its tell and its remedy.

Bug one is the most universal: forgetting optimizer.zero_grad(). Because PyTorch accumulates gradients (as post 3 explained), skipping the clear means each step's gradients add on top of all previous steps'. The accumulated gradient grows, your effective step size balloons, and loss diverges or oscillates wildly.

The symptom is training that explodes or never settles even with a reasonable learning rate. The fix is literally one line — zero_grad() at the top of each step — which is why it's so frustrating to lose hours to it. Recognizing the divergence pattern is the skill worth building.

This snippet shows the fix in context: zero_grad() is the first statement inside the per-batch loop, before the forward pass. Placing it at the top makes the four-beat rhythm self-correcting — clear, forward, backward, step — so stale gradients can never carry over.

The comment marks it as 'the missing line' because in buggy code everything else looks correct; the loop runs and produces numbers. Only the clearing step is absent, and nothing flags its absence. Internalizing its position in the rhythm is the durable defense.

Bug two is a memory leak that masquerades as normal logging. When you write total_loss += loss, loss is still a tensor attached to the computation graph. Adding it keeps a reference to that entire graph alive, so the graphs from every iteration accumulate and memory climbs until you run out.

The symptom is GPU or RAM usage that grows linearly with iterations and eventually OOMs, often well into a run. The fix is to extract the plain number with .item() (or break the graph link with .detach()) before accumulating, so you store a float and let each step's graph be freed.

This snippet shows the correct logging pattern: accumulate loss.item(), a Python float, into a running total. Because .item() pulls the scalar value out of the tensor and severs all ties to the graph, nothing holds the graph alive and memory stays flat.

The rule generalizes: any time you only need a number for printing or metrics, call .item() or .detach() so you don't accidentally pin the computation graph. This single habit prevents a whole class of mysterious out-of-memory failures during long training runs.

Bug three is wrong mode at evaluation. Skip model.eval() before testing and two things go wrong silently. Dropout stays active, randomly zeroing activations so predictions become noisy. BatchNorm keeps updating its running statistics from your test data, contaminating the model and leaking information.

The symptom is test accuracy that looks worse and more variable than it should, or subtly shifts between runs. The fix is to call model.eval() before inference and wrap it in torch.no_grad(); just remember to switch back to model.train() before continuing to train, or your dropout regularization disappears.

This snippet shows the full correct evaluation block and, importantly, the switch back to train mode afterward. eval() puts layers in inference behavior, no_grad() skips graph construction for speed and memory, and the explicit model.train() at the end restores training behavior.

That final line is easy to forget and causes the inverse bug: you evaluate mid-training, then continue training with dropout and batchnorm stuck in eval mode, quietly degrading learning. Pairing eval() with a matching train() is the disciplined pattern.

Bug four is the device mismatch, and unlike the others it usually does throw — 'Expected all tensors to be on the same device'. It's included because it's so common and the message, while explicit, confuses beginners who don't yet hold the device-match rule.

It happens the instant one tensor is on a different device than another in the same operation: model on GPU, a batch still on CPU. The fix is the consistency discipline from post 4 — send model and every batch to the same chosen device, every step. The bug is really just a forgotten .to(device) somewhere.

This compare diagram puts the broken and correct versions side by side. On the left: the model is moved to CUDA but the input batch is left on the CPU, producing the device RuntimeError the moment they meet in a layer. On the right: both model and batch are moved to the same device variable, so the operation runs cleanly.

The visual reinforces the mental checklist — for every op, ask 'are both operands on the same device?'. In practice this collapses to one rule: whatever touches the model lives on the model's device.

Bug five is subtle and purely conceptual: pairing the wrong loss with the wrong final activation. nn.CrossEntropyLoss applies log-softmax internally for numerical stability, so it expects raw, unnormalized logits. If you add your own softmax before it, you effectively apply softmax twice, which flattens the signal and trains poorly.

The symptom is a model that learns slowly or to mediocre accuracy with no error message. The same trap exists for binary classification: BCEWithLogitsLoss wants raw logits, not a sigmoid output. The rule is to know which losses include their activation and to feed those raw logits.

This snippet contrasts the wrong and right calls directly. The commented WRONG line wraps logits in softmax before CrossEntropyLoss; the RIGHT line feeds logits straight in. Because the loss does its own log-softmax, the correct version is both simpler and numerically more stable.

Keeping a one-line comment like 'logits!' near your loss is a cheap guard against a bug that's invisible at runtime. The broader lesson: read the loss function's docs to learn whether it expects logits or probabilities, because that contract is where this entire class of error lives.

The recap is the 2am checklist — the five fixes condensed so you can scan it when training misbehaves. zero_grad every step; .item() for logging; eval() plus no_grad() to test; same device for model and data; logits into CrossEntropyLoss. Each maps to one bug above.

Keeping this list nearby converts hours of debugging into seconds of recognition. That's the practical payoff of the whole day: not just writing PyTorch that runs, but writing PyTorch that trains correctly.

The CTA closes the day and points beyond PyTorch. With the framework's concepts, motivation, mechanics, code, and pitfalls covered, the series moves on to the next deep-learning building block — armed now with a tool you can actually drive.