LSTMs & GRUs

This post is the on-ramp for the whole day on gated recurrent cells. Day 45 established what a vanilla RNN is and why its memory fails over long spans. This day is about the fix: LSTMs and GRUs, recurrent cells that add learned gates so memory can survive across many steps.

Before arguing why they mattered or tracing the equations, the goal here is a clean mental model: a gated cell is the same recurrence you already know, with valves bolted on that decide what to keep and what to drop. Everything later in the day is a consequence of that one idea.

The defining feature of an LSTM or GRU is the gate. A vanilla RNN is forced to overwrite its entire hidden state at every step, which is why it loses old information. A gated cell instead computes small valves — numbers between 0 and 1 — that control how much of the old memory survives and how much new information is written.

The recurrence itself is unchanged: the new state still depends on the previous state and the current input. What changes is HOW that update happens. The gates make the update selective and partial rather than total, and that selectivity is the entire conceptual leap of this day.

A gate is mechanically simple: a linear layer followed by a sigmoid, producing a vector of values in [0, 1], which is then multiplied elementwise against some other vector. A value near 1 lets that dimension pass through fully; a value near 0 blocks it. There is one gate per controllable flow in the cell.

The important point is that the gate weights are learned. The network is not told when to remember or forget — it discovers, from the loss signal, which information is worth keeping and for how long. That is the difference between a vanilla RNN, which must dump and rewrite its memory every step, and a gated cell, which can choose to hold a value steady for hundreds of steps.

The LSTM's signature design is carrying two state vectors instead of one. The cell state c_t is the long-term memory: a highway along which information can travel across many steps with minimal interference. The hidden state h_t is the short-term, exposed output — a filtered view of the cell state used to make predictions and passed to the next step.

This separation is what people often miss. The cell state is protected storage; the hidden state is the working register you actually read from. Keeping them distinct is what lets an LSTM both preserve a fact for a long time AND emit a different, context-appropriate output at each step. A GRU, as we'll see, merges these two roles back into one vector.

The four named components of an LSTM are worth memorizing as a set. The forget gate decides how much of the previous cell state c_{t-1} to retain. The input gate decides how much of the new candidate content to write. The candidate (often written g or C-tilde) is the new content itself, proposed via a tanh. The output gate decides how much of the updated cell state to expose as the hidden state h_t.

Notice the division of labor: three sigmoid gates act as valves controlling flow, and one tanh produces actual content. If you can recite 'forget, input, candidate, output' and what each controls, the equations in the mechanics post will read as labels on things you already understand rather than as new symbols.

The flow diagram traces information through one LSTM cell. Old memory arrives, the forget gate scales it down (dropping what's no longer needed), the input gate scales the new candidate content, the two are added to form the updated cell state, and finally the output gate filters that cell state through a tanh to produce the hidden state.

The single most important arrow is the one carrying the cell state straight across with only a multiply-and-add. That nearly-uninterrupted path is what later lets gradients flow backward across long spans. If you remember one picture from this day, remember the cell-state highway running left to right with gates tapping into it rather than blocking it.

The GRU is the leaner cousin that captures most of the LSTM's benefit with fewer parts. It has two gates instead of three: an update gate that interpolates between the old state and a new candidate, and a reset gate that decides how much of the past to ignore when computing that candidate. Crucially, there is no separate cell state — the single hidden state h_t does both the long-term-storage and the working-output jobs.

The practical consequence is roughly 25% fewer parameters than an LSTM of the same width, faster training, and accuracy that is often comparable. The GRU is the answer to the question 'do I really need all of the LSTM's machinery?' — and for many tasks the honest answer is no.

This code makes the day's headline concrete: in PyTorch, swapping between a vanilla RNN, an LSTM, and a GRU is almost a no-op because they share a call signature. You construct the module with input and hidden sizes, call it on a sequence, and read back outputs and state.

The one difference to internalize early is the return type of the state. nn.RNN and nn.GRU return a single state tensor, while nn.LSTM returns a (h_n, c_n) tuple because it carries two memories. That tuple-versus-tensor distinction is the source of a surprising number of bugs, which is why it appears here on the conceptual post and again, with fixes, in the mistakes post.

Comparing directly against a vanilla RNN sharpens what gating buys you. A plain RNN keeps a single state that is fully overwritten each step, offers no control over what is kept, suffers vanishing gradients over long spans, and consequently forgets distant context. A gated cell makes updates selective, learns what to keep versus forget, preserves gradients through an additive path, and holds long-range context.

The takeaway is that gated cells are not a different family of model — they are RNNs with a better update rule. Choosing a gated cell over a vanilla RNN is the default for almost any task where information from more than a few steps back matters, which in practice is most sequence tasks.

This slide names the structural reason gating actually works, and it is worth pausing on. In a vanilla RNN, the state is pushed through a weight matrix and a squashing nonlinearity at every single step, so any signal is repeatedly scaled — which mathematically drives gradients toward zero (vanishing) or infinity (exploding) over many steps.

The gated cell's cell state is instead updated mainly by addition: c_t = f_t * c_{t-1} + i_t * g_t. When the forget gate is open, c_{t-1} passes through nearly unchanged, creating an almost-uninterrupted path for both information and gradient to travel across time. This additive 'constant error carousel' is the single structural change that defeats the vanishing-gradient problem, and the mechanics post derives exactly why.

This recap consolidates the mental model into five durable points: a gate is a learned valve in [0, 1]; an LSTM carries a long-term cell state plus a short-term hidden state; its three gates are forget, input, and output; a GRU uses just update and reset gates with no separate cell state; and the additive memory path is what beats the vanishing gradient.

If you can recite these five, you're ready for the rest of the day. Each subsequent post assumes this foundation and builds on it rather than re-explaining it.

The teaser points forward to the 'why it matters' post. Having established what LSTMs and GRUs are, the natural next question is what this gating actually bought the field — which problems in translation, speech, and language modeling it suddenly made tractable, and why these cells dominated sequence modeling for years before Transformers arrived.