LSTMs & GRUs

This is the engine-room post. The previous two established what gated cells are and why they mattered; here we replace intuition with explicit mechanics, tracing all four LSTM equations and the GRU's two-gate version on examples small enough to follow by hand.

The payoff is that two concepts — the additive cell-state update and the reason gradients stop vanishing — stop being slogans and become things you have actually watched happen. Doing the trace once is worth more than reading ten explanations of 'gates control the flow.'

The four LSTM equations are the entire forward computation. From the current input x_t and previous hidden state h_{t-1}, you compute: the forget gate f_t = sigmoid(W_f · [x_t, h_{t-1}] + b_f), the input gate i_t similarly, the output gate o_t similarly, and the candidate g_t = tanh(W_g · [x_t, h_{t-1}] + b_g). The three sigmoid gates live in [0, 1] and act as valves; the single tanh produces new content in [-1, 1].

Each is just a linear layer plus a nonlinearity over the same concatenated input. That symmetry is why implementations compute all four with one big matrix multiply and then split the result — a detail the code slide makes explicit.

Listing one LSTM step in order makes the loop concrete. First compute the four quantities f, i, o, g. Then form the new cell state: c_t = f_t * c_{t-1} + i_t * g_t, where the forget gate scales down the old memory and the input gate scales the new candidate before they are added. Finally produce the hidden state h_t = o_t * tanh(c_t), and pass both c_t and h_t forward to the next step.

The critical line is the cell-state update. It is an elementwise scale-and-add, not a matrix multiply through a nonlinearity. That structural choice — addition rather than repeated transformation — is the whole reason the LSTM can carry memory across long spans, and it is what the next slides examine from the gradient's point of view.

The flow diagram visualizes the cell-state update as a pipeline. The old memory c_{t-1} enters, gets scaled by the forget gate, has the gated new content i_t * g_t added to it, and emerges as c_t. A separate tap applies the output gate and a tanh to produce the hidden state h_t.

The shape to internalize is that c_t flows straight through with only a multiply and an add along the way — no matrix squashing it at each step. Contrast that with a vanilla RNN, where the state is pushed through W_h and a tanh every step. The diagram is essentially a picture of why gradients survive here and die there.

This from-scratch implementation collapses the four equations into clean code, and it reveals an efficiency detail real frameworks use. Rather than four separate matrix multiplies, you do one matmul on the concatenated [x, h] and then np.split the result into the four gate pre-activations. Apply sigmoid to the three gates and tanh to the candidate.

The line c = f * c + i * g is the additive update — the heart of the LSTM — and h = o * np.tanh(c) is the gated output. Seeing the whole cell in a dozen lines makes clear how little there is to it: the LSTM's power comes not from complexity but from one well-chosen update rule applied with learned valves.

This slide states the central insight of the entire day. In a vanilla RNN, the backpropagated gradient is multiplied by the recurrent weight matrix at every step, so over many steps it shrinks geometrically toward zero (vanishing) or grows without bound (exploding). That is why distant context can't be learned.

In an LSTM, the cell-state update c_t = f_t * c_{t-1} + ... is nearly additive. When the forget gate is open (near 1), the derivative of c_t with respect to c_{t-1} is approximately 1, so the gradient passes backward almost unchanged. This is the 'constant error carousel' the original authors described: an uninterrupted path that carries error signal far back in time. The gates regulate it, but the additive backbone is what keeps it alive.

The bar chart makes the gradient argument visual. In a vanilla RNN, the gradient reaching a step 20 positions back has collapsed to near zero — the model effectively cannot learn from that far. In an LSTM, the same distant gradient remains large, and even at 50 steps back it is still substantial.

The cause is exactly the additive cell-state path from the previous slide. Where the RNN's repeated matrix multiplication compounds toward zero, the LSTM's near-identity update preserves magnitude. This chart is the empirical face of the theory: it is why an LSTM can match a bracket fifty tokens earlier while a vanilla RNN has already forgotten it.

The GRU achieves the same gradient-preserving behavior with fewer parts, and this slide derives how. It drops the separate cell state and uses two gates over the single hidden state. The reset gate r_t controls how much of the previous state feeds into the candidate; the update gate z_t then interpolates: h_t = (1 - z_t) * h_{t-1} + z_t * h_tilde.

The key is that interpolation. When z_t is near 0, the new hidden state is almost exactly the old one — the past is carried verbatim, which is the same additive, near-identity path that protects gradients in the LSTM. The GRU proves you don't need three gates and two states to get the benefit; two gates and one state suffice for most tasks, at lower cost.

This from-scratch GRU mirrors the LSTM implementation so the contrast is clear. You compute the update gate z and reset gate r from the concatenated input. The reset gate multiplies the previous hidden state before it enters the candidate computation — r * h — which is how the GRU decides to 'ignore' part of the past. The candidate h_tilde is a tanh of that gated input.

The final line h = (1 - z) * h + z * h_tilde is the interpolation: a convex blend of old and new state controlled by the update gate. Compared to the LSTM's six-ish lines this is slightly shorter, and it uses one fewer gate and no cell state — a concrete look at where the GRU's parameter savings come from.

The side-by-side comparison consolidates the structural differences. The LSTM has three gates (forget, input, output), a separate cell state c_t, more parameters, and the strongest performance on the longest dependencies. The GRU has two gates (update, reset), a single hidden state, roughly 25% fewer parameters, and trains faster while often matching the LSTM's accuracy.

The practical reading is that these are points on a complexity-versus-capacity curve, not right-versus-wrong choices. More gates and a dedicated memory give the LSTM an edge in the hardest, data-rich regimes; the GRU's leanness wins when data or compute is limited. Knowing the mechanics is what lets you reason about which edge your task actually needs.

This trace makes the gates tangible with concrete numbers. The forget gate is [0.95, 0.10]: it keeps almost all of dimension 0's old memory but discards 90% of dimension 1's. The input gate [0.20, 0.90] writes little new into dimension 0 but a lot into dimension 1. Applying c_t = f * c_prev + i * g shows dimension 0 staying near its old value while dimension 1 is essentially rewritten.

This is selective memory in action: per dimension, the cell decides independently whether to preserve or replace. Watching specific numbers flow through the update rule is what converts 'gates control the flow' from a phrase into an understood mechanism — and it shows why an LSTM can hold one fact steady while updating another.

This recap pins down the mechanics: the LSTM computes f, i, g, o, then updates c_t = f*c + i*g and emits h_t = o * tanh(c_t); the additive cell-state path is what preserves gradients; and the GRU uses update gate z and reset gate r to interpolate between old and new state, achieving similar power with fewer parameters.

With the math traced by hand, you are ready to build the real thing. The next post assembles a working, trainable LSTM in PyTorch and shows the one-line swap to a GRU.

The teaser points to the hands-on build. Having traced both cells by hand, the next post puts it together as complete PyTorch models — data preparation, model definition, the training loop, generation, and the one-line GRU swap — so the mechanics become a working artifact you can run and modify.