Recurrent Neural Networks

This is the engine-room post. The previous two established what an RNN is and why it mattered; here we replace intuition with explicit mechanics, tracing both the forward recurrence and the backward pass on a small enough example to follow by hand.

The payoff is that two later concepts — backpropagation through time and the vanishing-gradient problem — stop being vocabulary and become things you have actually watched happen. Doing the trace once is worth more than reading ten explanations.

The recurrence equation is the whole forward computation in one line: h_t = tanh(W_x · x_t + W_h · h_{t-1} + b). W_x decides how the new input influences the state, W_h decides how the previous memory persists, the bias shifts the result, and tanh squashes everything into the range [-1, 1] to keep the state bounded.

An optional output y_t = W_y · h_t reads a prediction off the state. That is the complete parameter inventory of a vanilla RNN: three matrices and a bias. Everything the model 'knows' lives in W_x, W_h, W_y, and b.

Listing the forward pass as ordered steps makes the loop concrete. You begin with h_0 set to zeros — an empty memory. At each step you read the current input and the previous hidden state, combine them through the recurrence to get the new state, optionally emit an output, and continue until the sequence ends.

The critical detail is that h_{t-1} is reused as an input to compute h_t. That single dependency is what threads information through time and what later forces backpropagation to flow backward through every step. Internalize the order now and the backward pass will make sense.

This forward-pass code mirrors the steps exactly and adds one production-relevant detail: the cache. As the loop runs, it stores the input, the resulting hidden state, and the pre-activation z at every step. These saved values are not optional bookkeeping — they are required by the backward pass.

This is the same forward-then-backward dependency you would have seen in the backpropagation material: you cannot compute gradients without the intermediate values from the forward pass. Storing them as you go is what makes BPTT possible, and it is also why long sequences cost memory.

The unrolling diagram is the conceptual bridge to training. Although the RNN is physically one cell looping on itself, you can lay each time step out as its own layer, producing a deep feedforward network where every layer shares the identical W_x and W_h.

This reframing is powerful because it means you do not need a new training algorithm — you can apply ordinary backpropagation to the unrolled graph. The 'depth' of this unrolled network equals the sequence length, which foreshadows why very long sequences cause the gradient problems explored two slides later.

Backpropagation through time (BPTT) is just backprop applied to the unrolled network, with one twist from weight sharing. Because the same weight matrix appears at every unrolled layer, that weight's true gradient is the SUM of the gradients computed at each individual time step.

This summing is easy to forget and important to get right: a single weight gets a learning signal from every position where it was used. Mechanically, you walk the time steps in reverse, accumulating each weight's contribution as you go, which is exactly what the next code slide implements.

This snippet makes BPTT explicit. Walking the cached states in reverse, at each step it combines the gradient arriving from this step's output with the gradient passed back from the future step (dh_next). It pushes that through the tanh derivative (1 - h squared), accumulates the contributions to W_x and W_h, and then computes the gradient to hand back to the previous step.

The line dh_next = Wh.T @ dz is the heart of it: the gradient is repeatedly multiplied by W_h as it travels backward through time. That repeated multiplication by the same matrix is the mechanism behind the vanishing and exploding gradient behavior shown next.

The bar chart visualizes vanishing gradients concretely. The gradient signal reaching recent time steps stays strong, but as you trace further back in time it shrinks — noticeably weaker at step 8 and nearly zero by step 20.

The cause is the repeated multiplication by W_h seen in the previous slide: factors smaller than one compound toward zero over many steps. The practical meaning is that a vanilla RNN's learning signal barely reaches distant past inputs, so it effectively cannot learn long-range dependencies. This chart is the visual proof of a limitation the day keeps returning to.

Here the vanishing/exploding problem is stated as the core failure mode. Because BPTT multiplies by W_h once per step, the magnitude of W_h's influence governs everything: values below one drive the gradient toward zero (vanishing, so distant context is forgotten), and values above one blow it up (exploding, so training destabilizes).

This is not a bug to be patched but an inherent property of the architecture, and it is precisely the problem that motivated LSTMs and GRUs. Their gating mechanisms create a more stable path for gradients to flow over long spans, which is why they dominate when long-range memory matters.

The final code slide shows that in practice the framework hides all this machinery. nn.RNN runs the forward recurrence, loss.backward() performs BPTT automatically through autograd, and you never write the manual loop from the earlier slides.

The one line you should still add yourself is clip_grad_norm_, which caps the gradient magnitude to prevent the exploding case from wrecking a training run. Seeing the hand-traced version first is what lets you trust and debug the framework version — you now know exactly what backward() is doing under the hood.

This recap pins down the mechanics: the recurrence is h_t = tanh(W_x x_t + W_h h_{t-1} + b); you cache states on the forward pass; unrolling turns the loop into one deep net with shared weights; BPTT sums each weight's gradient across all time steps; and the repeated W_h factor is what causes gradients to vanish or explode.

With the math traced by hand, you are ready to build the real thing. The next post assembles a working, trainable RNN in PyTorch.

The teaser points to the hands-on build. Having traced the forward and backward passes manually, the next post puts it together as a complete PyTorch model — data preparation, model definition, the training loop, and generation — so the mechanics become a working artifact you can run and modify.