Recurrent Neural Networks

This closing post is a field guide to the mistakes that quietly break RNN training. The earlier posts gave you understanding and a working model; this one inoculates you against the failures that do not announce themselves with a clean error message.

The theme is that RNNs fail silently. A wrong setting rarely crashes — it gives you a model that trains at a crawl, forgets everything, or never improves. Each mistake here comes with a one-line fix, so the value is in being able to NAME the problem the moment you see its symptom.

Exploding gradients are the loudest of the quiet failures. Because BPTT multiplies gradients across many time steps, they can compound into enormous values, turn into NaNs, and obliterate your weights in a single optimizer step — leaving a loss that suddenly jumps to infinity.

The fix is so standard that nearly every working RNN training loop includes it: clip the global gradient norm before stepping. This caps the update size without changing its direction, keeping training stable even when an occasional step would otherwise blow up. If you see NaN loss in an RNN, gradient clipping is the first thing to check.

This snippet is the fix in its proper place: clip_grad_norm_ goes after loss.backward() (so the gradients exist) and before optimizer.step() (so the step uses the clipped values). A max_norm around 5.0 is a common starting point.

The operation rescales the entire gradient vector so its norm does not exceed the cap, preserving direction while limiting magnitude. This is cheap insurance — it costs nothing when gradients are well-behaved and saves the run when they are not, which is why it belongs in essentially every RNN loop.

Expecting a vanilla RNN to remember the distant past is a conceptual mistake with no error message. As the mechanics post showed, gradients vanish through repeated multiplication by W_h, so the learning signal barely reaches inputs from many steps ago. The model simply never learns long-range dependencies.

The symptom is a model that handles local patterns fine but fails on anything requiring memory across a long span — matching brackets, tracking a subject across a long sentence, long-document context. The fix is architectural: switch to an LSTM or GRU, whose gating gives gradients a stable path over long distances.

The decision tree turns 'which cell should I use' into a simple flowchart. If your task needs long-range memory, choose a gated cell: an LSTM when robustness matters most, or a GRU when you want fewer parameters and faster training at a small cost in capacity. If you do not need long-range memory, a vanilla RNN is perfectly adequate and lighter.

The broader lesson is to match the cell to the task rather than defaulting to the most complex option. Reaching for an LSTM on a problem with only short dependencies wastes compute; using a vanilla RNN where long memory is essential guarantees underperformance.

Hidden-state mishandling covers two opposite errors that both stem from misunderstanding what the state is. Carrying it across batches without calling .detach() keeps it attached to the autograd graph, so the graph grows with every batch until memory is exhausted — a crash that confuses people because the model code looks fine.

The opposite error is re-zeroing the state mid-sequence, which erases context you actually needed and quietly hurts accuracy. The right discipline is to detach between truncated chunks (keeping the values but cutting the graph) and to reset the state only at genuine sequence boundaries.

This snippet shows truncated BPTT done correctly. You process the sequence in chunks, carrying the hidden state h from one chunk to the next so context is preserved, but calling h.detach() after each backward pass so the autograd graph does not extend across chunks.

The detach is the key move: it keeps the numerical value of the hidden state (so memory carries forward) while severing the gradient history (so the graph stays bounded). This is how you train on sequences too long to backpropagate through in full, and it is the standard pattern for streaming or very long inputs.

Wrong tensor shapes are the most insidious bug because the model often trains on garbage without complaining. nn.RNN defaults to (time, batch, features); if your data is (batch, time, features) and you forget batch_first=True, PyTorch happily interprets the batch dimension as time and trains on nonsense.

Because there is no error — just a loss curve that does not improve or improves strangely — this can cost hours. The fix is disciplined: set batch_first=True whenever your data is batch-first, and verify the dimension order explicitly before trusting any results.

This comparison sorts RNN bugs into loud and silent, and makes the counterintuitive point that the silent ones are worse. Loud bugs — shape mismatches that crash, NaN loss from exploding gradients, out-of-memory from a growing graph — at least tell you something is wrong immediately.

The silent bugs are the dangerous ones: a wrong batch dimension that trains on noise, a double softmax that quietly weakens gradients, missing masking that wastes capacity on padding. They produce a model that looks like it is training but is subtly broken. Learning to suspect silent bugs when results are merely mediocre is a mark of experience.

The softmax and padding traps round out the catalog. CrossEntropyLoss applies log-softmax internally, so passing it softmax probabilities double-counts the operation, flattening the loss landscape and weakening gradients — always feed it raw logits. Separately, when you pad variable-length sequences to a common length, the pad tokens are meaningless, and an unmasked RNN wastes capacity trying to model them.

The fix for padding is to mask it, typically with pack_padded_sequence, so the RNN skips the pad steps entirely. Both mistakes share a flavor: the code runs and produces numbers, but the numbers are subtly wrong in a way that hurts learning.

This snippet shows the padding fix in practice. pack_padded_sequence takes the embeddings and the real length of each sequence and produces a packed representation that the RNN processes without ever touching the pad steps. Setting enforce_sorted=False lets you pass sequences in any order.

The result is that gradients and hidden states reflect only real data, not padding artifacts. For any batch of variable-length sequences, packing is the standard, correct way to handle padding, and skipping it is a common reason a model underperforms on real-world batched data.

The final checklist consolidates the five fixes into a pre-flight you can run before trusting any RNN: clip the gradient norm every step; use an LSTM or GRU when you need long-range memory; detach the hidden state between chunks; set batch_first and verify your shapes; and pass raw logits while packing padded sequences.

Run through this list and you avoid the failures that cost most people their first few days with RNNs. With understanding, mechanics, a working build, and now the traps all covered, you have the full picture of recurrent networks.

The teaser closes the day and opens the next topic. Having mastered recurrence — including its central weakness, the difficulty of long-range memory — the natural next step is the architecture built to overcome exactly that: attention, and the Transformer it powers. The series continues there.