Recurrent Neural Networks

This is the hands-on post, and it is intentionally code-heavy. The previous posts built understanding from concept through mechanics; here you assemble a complete, runnable character-level RNN in PyTorch so the ideas become a working artifact rather than abstractions.

The model learns to predict the next character in a sequence — the simplest task that exercises everything an RNN does. Each snippet is real code that runs in order, so you can copy the whole thing, train it, and then modify it to understand each piece by breaking it.

Setup defines a tiny dataset and the encoding that turns characters into the integer ids a neural network can consume. We build two lookup tables — stoi (string to index) and itos (index to string) — and convert the text into a tensor of ids.

This encode/decode boundary is something every sequence model needs: networks operate on numbers, not raw text, so you always need a clean mapping in both directions. Keeping the vocabulary small here means the model trains in seconds, which is ideal for experimentation.

The model class shows the canonical RNN stack: an embedding layer that maps each id to a dense vector, an nn.RNN that processes the sequence and maintains the hidden state, and a linear head that maps each step's hidden state to a score over the vocabulary.

The forward method threads an optional hidden state h through the RNN and returns it alongside the outputs. Returning h matters for generation, where you feed the model one character at a time and need to carry the memory forward between calls. This embed -> RNN -> linear pattern is the backbone of most sequence models you will write.

The training loop is where backpropagation through time actually happens — and notice how little code it takes. You form inputs and next-character targets by shifting the sequence by one, run a forward pass to get logits, compute cross-entropy loss, and call loss.backward(), which triggers autograd to run BPTT across the entire unrolled sequence.

The one RNN-specific addition is clip_grad_norm_ before opt.step(), guarding against the exploding gradients traced in the previous post. Everything you derived by hand earlier is compressed into these few lines; the framework handles the bookkeeping, but you now know exactly what it is doing.

The pipeline diagram names the five stages every training step moves through: encode characters into ids, run the forward pass of the RNN over all time steps, compute the next-character cross-entropy loss, run BPTT backward through time, and take an optimizer step.

Seeing the stages laid out helps when debugging — if training fails, you can ask which stage is at fault. A shape error lives in encode or forward; a NaN loss points to the backward/clip stage; a flat loss curve often means the step or learning rate is wrong. The diagram is a mental checklist for the loop above.

Generation demonstrates the model's learned behavior and a subtle mechanic: stateful, one-step-at-a-time inference. You seed with a character, then repeatedly feed the current character and the carried hidden state h back in, take the argmax of the logits as the next character, and append it.

The crucial part is threading h through every call — that is how the model remembers what it has generated so far. Using argmax here gives deterministic 'greedy' generation; swapping in sampling from the softmax would produce more varied, creative output. This loop is the inference-time mirror of the training loop.

These shape and convention gotchas are where most beginners lose hours. nn.RNN with batch_first=True expects (batch, time, features); out holds every step's output while h_n is only the final state; CrossEntropyLoss wants raw logits because it applies log-softmax internally; targets are the inputs shifted by one position; and detaching the hidden state between batches is how you truncate BPTT to a manageable length.

None of these throw obvious errors when you get them wrong — they silently produce a model that will not learn. Committing them to memory is the difference between an RNN that trains and one that mysteriously does not.

The LSTM upgrade shows off the modularity of PyTorch's RNN family: swapping nn.RNN for nn.LSTM is a one-line change, and the entire training loop keeps working. This matters because, as the mechanics post established, LSTMs handle long-range dependencies far better than vanilla RNNs.

The one wrinkle to remember is the return signature: an LSTM returns its state as a tuple (h_n, c_n), where c_n is the cell state that carries long-term memory. If you index the state assuming a single tensor, you will hit a shape error — a small price for a large capability upgrade.

This slide catalogs the runtime errors that actually bite in practice, several of which preview the dedicated mistakes post. Forgetting batch_first produces a silent dimension mismatch; passing softmax outputs into CrossEntropyLoss double-applies the softmax and weakens gradients; carrying the hidden state across batches without .detach() lets the autograd graph grow until you run out of memory; and skipping gradient clipping lets one exploding step destroy the model.

The pattern is that the worst RNN bugs are quiet — they do not crash, they just give you a model that does not learn. Knowing them in advance is the cheapest debugging you will ever do.

The data-flow diagram traces the tensor shapes through the model, which is the single most useful thing to hold in your head when wiring a sequence model. Integer ids of shape (B, T) become embeddings (B, T, H), pass through the RNN preserving (B, T, H), and the linear head maps them to logits of shape (B, T, V) — one score per vocabulary item per time step.

Most wiring bugs are shape bugs, so being able to recite this chain lets you catch a mismatch by reasoning rather than by trial and error. When something breaks, mentally walk these shapes and the offending layer usually reveals itself.

This recap consolidates the build: the embed -> RNN -> linear head stack is the standard skeleton; loss.backward() performs BPTT for you; gradient clipping is mandatory for RNNs; targets are inputs shifted by one; and upgrading from nn.RNN to nn.LSTM takes almost no changes.

With a working model in hand, the only thing left is to learn the traps — the quiet mistakes that break RNN training even when the code looks right. That is the focus of the final post.

The teaser points to the common-mistakes post. You now have a model that runs; the next step is hardening it by learning the five quiet failure modes — exploding gradients, missing long-term memory, hidden-state mishandling, shape errors, and softmax/padding traps — and the one-line fix for each.