Recurrent Neural Networks

This post is the on-ramp for the whole day. Before debating why recurrent networks mattered or tracing the math, you need a clean mental model of what an RNN actually is: a neural network with a loop, carrying a hidden state from one step of a sequence to the next.

The goal here is to lock in vocabulary and intuition so the later posts have somewhere to stand. Everything that follows — backprop through time, the PyTorch build, the common traps — is just consequences of the one idea introduced on this cover.

The defining feature of an RNN is memory. A feedforward network sees each input in isolation and forgets it the moment it produces an output. An RNN instead maintains a hidden state vector that summarizes everything it has consumed so far, and it feeds that state back into itself at the next step.

That feedback loop is the entire conceptual leap. The present prediction is shaped not just by the current input but by the accumulated history compressed into the hidden state. When people say an RNN 'has memory,' this loop is precisely what they mean.

Underneath the diagrams and the framework calls, every RNN reduces to a single update rule applied repeatedly: the new hidden state is some function of the previous hidden state and the current input. Memorizing that one sentence is worth more than memorizing any equation, because it tells you what to look for in any RNN variant.

LSTMs and GRUs are just fancier versions of this same recurrence — they change HOW h_t is computed from h_{t-1} and x_t, but not the fact that it is. Hold the rule fixed in your head and the variants become small deltas rather than new architectures.

These five terms are the working vocabulary you will see in every RNN discussion. A sequence is an ordered list of inputs; a time step is one position in it; the hidden state is the running memory; the recurrence is the dependency of the current state on the previous one; and weight sharing is the reuse of one parameter set across all positions.

Get these straight now and the rest of the day reads smoothly. Confuse 'hidden state' (the memory that changes each step) with 'weights' (the parameters that stay fixed across steps) and almost everything downstream becomes muddled, so it is worth pausing on that distinction in particular.

The cycle diagram captures the essence: an input arrives, it is combined with the previous hidden state, that produces an updated hidden state, and optionally an output is read off. Then the loop turns again for the next element.

The arrow that feeds the hidden state back to the 'combine' step is the recurrence — the one connection that separates an RNN from an ordinary network. If you remember nothing else visually, remember that backward-pointing arrow; it is the source of both the model's power (memory) and its problems (vanishing gradients), which later posts unpack.

Comparing against a feedforward network sharpens what is unique about recurrence. A feedforward net takes a fixed-size input, has no memory of previous inputs, treats each example independently, and is fully deterministic given its input. An RNN relaxes every one of those constraints.

The practical upshot: feedforward nets are right for tabular or image data where each example stands alone, while RNNs are right for sequences where order and history carry meaning. Choosing the wrong family for your data type is a foundational mistake that no amount of tuning will fix.

Weight sharing is the quietly brilliant part of the design. Rather than learning a distinct transformation for each position in the sequence, the RNN learns one transformation and applies it everywhere. This is the direct analogue of how a convolutional filter is shared across an image.

Two huge benefits follow. First, the parameter count is independent of sequence length, so the same model handles a 5-word or a 500-word input. Second, a pattern the network learns at one position generalizes for free to every other position, which is exactly the kind of inductive bias sequence data rewards.

This snippet is the conceptual definition turned into runnable code. rnn_step implements the recurrence directly: it combines the current input and previous hidden state through two weight matrices, adds a bias, and squashes with tanh. The loop then walks the sequence, threading the hidden state through.

Notice that h starts as zeros — the network begins each sequence with empty memory — and that Wx, Wh, and b never change inside the loop. Seeing the shared weights reused on every iteration makes the abstract idea of weight sharing concrete and undeniable.

Unrolling is the mental trick that makes RNNs trainable and understandable. Although the network is physically a single cell with a loop, you can lay out each time step side by side as if it were a separate layer. The result looks like a very deep feedforward network — one layer per time step — where every layer shares the same weights.

This unrolled view is not just a teaching aid; it is literally how training works. The next post shows that backpropagation through time is just ordinary backprop run over this unrolled graph, which is why the picture is worth internalizing early.

One mechanism, many task shapes. The same recurrent core supports one-to-many (a single image fanned out into a caption), many-to-one (a whole review collapsed into one sentiment label), and many-to-many (each input token mapped to an output token, as in translation or tagging).

What changes between these is only where you tap inputs and read outputs along the unrolled sequence — never the loop itself. Recognizing that these are configurations of one architecture, not different architectures, is what lets you reuse the same code across wildly different problems.

This recap consolidates the mental model into five durable takeaways: an RNN is a loop with a hidden state; that state depends on both the previous state and the current input; one weight set is reused at every step; the model is built for ordered, variable-length data; and its output can depend on the entire history.

If you can recite these five points, you are ready for the rest of the day. Each subsequent post assumes this foundation and builds on it rather than re-explaining it.

The teaser points forward to the 'why it matters' post. Having established what an RNN is, the natural next question is what this looped design actually bought the field — which problems in language, speech, and time series it suddenly made tractable. Day 46 in the series continues from there.