LSTMs & GRUs

This closing post is a field guide to the mistakes that quietly break LSTM and GRU training. The earlier posts gave you understanding and a working model; this one inoculates you against the failures that don't announce themselves with a clean error message.

The theme is that gated cells fail silently. A wrong setting rarely crashes — it gives you a model that trains slowly, overfits, mishandles its own memory, or never improves. And a meta-mistake threads through all of them: reaching for an LSTM by reflex when a GRU, a plain RNN, or a Transformer would serve better. Each trap here comes with a one-line fix, so the value is being able to NAME the problem the moment you see its symptom.

Mishandling the (h, c) state tuple is the most common LSTM-specific bug. nn.LSTM returns its state as a tuple (h_n, c_n) because it carries two memories, while nn.GRU and nn.RNN return a single tensor. Code written for one cell breaks on the other: index the tuple as if it were a tensor and you crash; pass a bare tensor where a tuple is expected and you error or silently misbehave.

The symptom usually appears the moment you swap cells to compare them, or when you copy a snippet written for a GRU into an LSTM project. The fix is to unpack explicitly and keep the difference front of mind: LSTM gives you (h, c), everything else gives you one tensor.

This snippet shows correct state unpacking for both cells side by side. For an LSTM you write out, (h_n, c_n) = lstm(x) and then index h_n[-1] to get the final layer's hidden state. For a GRU you write out, h_n = gru(x) — a single tensor — and index it the same way.

The h_n[-1] indexing is itself worth noting: the state's first dimension is the layer (and direction) count, so for a multi-layer or bidirectional model you must index deliberately to grab the right slice. Getting both the tuple-versus-tensor distinction and the layer indexing right is what makes cell-swapping experiments painless rather than a source of confusing crashes.

Assuming LSTMs need no gradient clipping is a conceptual trap with an expensive payoff. People reason that since gating solves the vanishing-gradient problem, recurrent training is now safe — but gating does nothing for the EXPLODING gradient. A long-sequence batch can still produce gradients that blow up to NaN and obliterate the weights in a single optimizer step.

The fix is identical to the vanilla-RNN case: clip the global gradient norm before stepping. Nearly every robust LSTM or GRU training loop includes it for exactly this reason. If you see a loss that suddenly jumps to NaN partway through training, missing or too-loose gradient clipping is the first thing to check.

This snippet places the clip in its correct position: after loss.backward() so the gradients exist, and before optimizer.step() so the step uses the clipped values, with a max_norm around 5.0 as a common starting point. The comment is the lesson — gating fixes vanishing, not exploding — because that misconception is exactly what leads people to omit this line.

The operation rescales the whole gradient vector so its norm stays under the cap, preserving direction while limiting magnitude. It costs nothing when gradients are well-behaved and saves the run when they aren't, which is why it belongs in essentially every recurrent training loop regardless of whether the cell is gated.

Defaulting to an LSTM over a GRU by reflex is a quiet efficiency mistake. The LSTM has three gates, a separate cell state, and roughly a third more parameters, so it trains slower and has more capacity to overfit. On small and medium datasets a GRU frequently matches or beats it while training faster and generalizing better, because it has fewer parameters to misuse.

The better default is to try a GRU first and escalate to an LSTM only if it clearly underperforms on your task. Reaching for the 'bigger' cell automatically wastes compute and can actively hurt results when data is limited. This is the same match-the-tool-to-the-task discipline that the why-it-matters post emphasized.

The decision tree turns 'which recurrent cell should I use' into a simple flowchart. If your task needs long-range memory, choose a gated cell: an LSTM when you have lots of data and compute and need maximum robustness, or a GRU when you want a lean, fast model. If you don't need long-range memory at all, a vanilla RNN is perfectly adequate and lighter still.

The broader lesson is to match the cell to the task rather than defaulting to the most complex option. Reaching for an LSTM on a problem with only short dependencies wastes capacity; using a vanilla RNN where long memory is essential guarantees underperformance. And for very long, parallelizable workloads, the right answer may be no recurrence at all — a Transformer.

Ignoring the forget-gate bias is a subtle initialization mistake with an outsized effect. When an LSTM's forget-gate bias starts at zero, the sigmoid outputs around 0.5, so the cell discards roughly half its memory every step before training has taught it otherwise. Learning long dependencies from that starting point is slow because the model must first unlearn its tendency to forget.

Initializing the forget-gate bias to a positive value — 1.0 is the standard choice — makes the gate start mostly open, so the cell remembers by default and only learns to forget where the data demands it. It's a one-time change that measurably speeds convergence on tasks requiring long memory, and it's a well-known trick that surprisingly many implementations omit.

This snippet implements the forget-gate-bias fix in PyTorch, and it requires knowing the framework's gate layout. PyTorch packs the four gate biases per layer in the order [input, forget, candidate, output], concatenated into one bias vector. To set just the forget-gate portion you slice the second quarter — indices from n/4 to n/2 — and fill it with 1.0.

The loop iterates over named parameters, finds the bias tensors, and applies the fill. Getting the slice right depends on knowing that layout; fill the wrong quarter and you've initialized the input or output gate instead, with no error to warn you. This is a good example of how a beneficial trick still demands care about framework internals.

This comparison sorts gated-RNN bugs into loud and silent, and makes the counterintuitive point that the silent ones are worse. Loud bugs — a tuple-versus-tensor state mismatch that crashes, NaN loss from exploding gradients, out-of-memory from a graph that grows because you forgot to detach — at least announce themselves immediately.

The silent bugs are the dangerous ones: a wrong batch dimension that trains on noise, a missing forget-gate bias that makes the model learn slowly without any error, missing sequence packing that wastes capacity on padding. They produce a model that looks like it's training but is subtly broken. Learning to suspect silent bugs when results are merely mediocre is a mark of experience with recurrent nets.

This snippet shows the padding fix in practice, which is the same for LSTMs and GRUs as for vanilla RNNs. pack_padded_sequence takes the embeddings and the true length of each sequence and produces a packed representation that the cell processes without ever touching the pad steps. Setting enforce_sorted=False lets you pass sequences in any order.

The result is that gradients and hidden (and for an LSTM, cell) states reflect only real data, not padding artifacts. For any batch of variable-length sequences, packing is the standard, correct way to handle padding, and skipping it is a common reason a model underperforms on real-world batched data despite looking fine in a single-sequence test.

The final checklist consolidates the fixes into a pre-flight you can run before trusting any LSTM or GRU: unpack the LSTM (h, c) tuple versus the GRU's single tensor; clip gradients because gating only fixes vanishing, not exploding; try a GRU before defaulting to an LSTM; initialize the forget-gate bias positive; and use proper dropout while packing padded sequences.

Run through this list and you avoid the failures that cost most people their first days with gated cells. With understanding, mechanics, a working build, and now the traps all covered, you have the full picture of LSTMs and GRUs.

The teaser closes the day and opens the next topic. Having mastered gated recurrence — and seen that it eases but never fully removes the limits of processing sequences one step at a time — the natural next step is the mechanism built to escape recurrence entirely: attention, and the Transformer it powers. The series continues there.