The Transformer, Explained

This closing post is a field guide to the mistakes that quietly break Transformers. The earlier posts gave you the skeleton, the motivation, the math, and a working block; this one inoculates you against the failures that do not announce themselves with a clean error message.

The theme is that the most dangerous Transformer bugs are silent. A missing mask, dropped positions, or an unscaled score rarely crashes — instead you get a loss curve that looks great while the model is secretly cheating, or a model that captures topic but mangles meaning. Each mistake here comes with a one-line fix, so the value is in being able to NAME the problem the instant you recognize its symptom.

Forgetting the causal mask is the most devastating Transformer bug, and it is purely silent. In an autoregressive model each position must attend only to itself and earlier positions. Without the mask, every token can see the entire sequence, including the future tokens it is supposed to predict — so the training loss plummets because the model is literally reading the answer.

The betrayal comes at inference: when you generate text token by token, those future positions do not exist yet, so the model that aced training produces garbage. The symptom is a suspiciously low training loss paired with broken generation. The fix is to apply a lower-triangular causal mask in every attention layer of the decoder.

This snippet shows the fix in its proper form. torch.tril builds a lower-triangular boolean matrix where position (i, j) is true only when j is at most i, encoding the rule that token i may attend to token j only if j is not in the future. Applying masked_fill with negative infinity to the inverse of that mask sets every future score to negative infinity before softmax.

The reason negative infinity works is that softmax exponentiates its inputs, and the exponential of negative infinity is zero — so future positions receive exactly zero attention weight, with no leakage. This is the same masked_fill trick from the build post; here it is the difference between a model that genuinely predicts the next token and one that cheats.

Dropping positional information is a conceptual mistake that produces a strange, hard-to-diagnose failure. Because self-attention is permutation-invariant, a model with no positional signal treats its input as an unordered bag of tokens — 'the cat sat' and 'sat the cat' are literally identical to it.

The symptom is subtle: the model learns topic and word co-occurrence reasonably well, so the loss is not catastrophic, but it consistently mangles syntax and word order because it has no way to represent sequence. The fix is to add positional information — sinusoidal encodings, learned position embeddings, or rotary encodings — to the token embeddings before they enter the first block, giving every token an awareness of where it sits.

This comparison makes the consequence of missing positions concrete. Without positional encoding the model sees a bag of tokens, so 'dog bites man' and 'man bites dog' are indistinguishable, syntax breaks down, and crucially the loss can look fine even as meaning fails — making the bug easy to miss. With positional encoding each token knows its slot, word order is preserved, syntax can be learned, and generation becomes coherent.

The broader lesson is that a healthy-looking loss curve is not proof of a correct model. This is a recurring theme across Transformer bugs: several of the worst failures leave the loss looking acceptable while quietly destroying a capability you care about, which is why understanding the mechanism matters more than watching the metric.

Ignoring the quadratic cost of attention is a resource mistake that ambushes teams at exactly the wrong moment. Attention builds an n-by-n score matrix comparing every token with every other, so memory and compute scale with the square of sequence length: doubling the context from 512 to 1024 tokens roughly quadruples the cost of the attention step.

The practical consequence is out-of-memory errors that appear only when you push to long context, often late in a project. The fix is to budget for it up front — cap your sequence length to what fits, and when you genuinely need long inputs, reach for an efficient variant like FlashAttention, which reduces memory overhead, or sliding-window and sparse attention, which avoid the full n-by-n matrix.

This bar chart visualizes the quadratic blow-up that catches people off guard. Going from 512 to 1024 tokens does not double attention memory — it quadruples it; going to 2048 multiplies it roughly sixteenfold relative to the 512 baseline. The bars grow far faster than the sequence length, which is the whole danger.

The takeaway is that long context is not a linear upgrade you can casually enable. A model that trains comfortably at 512 tokens may be completely infeasible at 4096 without architectural changes. Internalizing the shape of this curve is what lets you plan capacity correctly instead of discovering the limit through a crash, and it is why efficient-attention research is such an active area.

Mishandling padding masks is a batching mistake that silently corrupts results. To process sequences of different lengths in one batch, the short ones are padded with filler tokens to a common length. If you do not mask those pad positions, attention treats them as real content and spreads weight onto meaningless tokens, polluting the context vector of every genuine token in the sequence.

The fix is a padding mask that marks which positions are real and sets the pad positions to negative infinity before softmax, so they receive zero attention weight. Crucially, the padding mask is separate from the causal mask and serves a different purpose — one blocks the future, the other blocks filler — and in a decoder you need both, combined together.

This snippet shows how to combine the causal and padding masks correctly, which trips up many implementations. The causal mask is an (n, n) lower-triangular matrix; the padding mask starts as a (batch, n) boolean of which tokens are real and is reshaped to (batch, 1, 1, n) so it can broadcast across heads and query positions. A logical AND of the two yields a combined mask that is true only where a position is both non-future and non-padding.

Applying masked_fill with that combined mask sets every disallowed score to negative infinity in one step. The key insight is that the two masks have different shapes because they constrain different axes — the causal mask relates query and key positions, while the padding mask depends only on the key being a real token — and broadcasting is what lets them combine cleanly.

Skipping the square-root-of-d_k scaling is a from-scratch mistake that produces a baffling training failure. Without it, dot products in high dimensions grow large, softmax saturates toward a one-hot distribution, and in that saturated regime gradients are vanishingly small — so the model trains painfully slowly or diverges, with no error and no obvious cause.

The reassuring part is that every mature library scales automatically, so this trap only bites people implementing attention by hand. The fix is simply to divide the scores by the square root of d_k before softmax, exactly as the mechanics post derived. If your hand-rolled Transformer mysteriously refuses to learn, an unscaled score matrix is one of the first things to check.

This comparison sorts the failures into loud and silent, and the lesson is that the silent ones are far more dangerous. Loud bugs announce themselves immediately: a d_model that is not divisible by the head count, a mask whose shape will not broadcast over the scores, or an out-of-memory error on long sequences. You see them, fix them, and move on.

The silent bugs are the costly ones because they masquerade as success. A missing causal mask gives a training loss that looks too good to be true; missing positions break syntax while the loss stays acceptable; and missing scaling stalls learning with no message. Learning to suspect these silent failures whenever results are merely mediocre — rather than trusting the loss curve — is a hallmark of experience with Transformers.

The final checklist consolidates the five fixes into a pre-flight you can run before trusting any Transformer: if it is a decoder, apply the causal mask; always add positional information; budget for the n-squared cost of attention; mask padding tokens before softmax; and scale scores by the square root of d_k, preferring pre-norm when the network is deep.

Run through this list and you avoid the failures that cost most people their first weeks with Transformers. With the skeleton, the motivation, the math, a working build, and now the traps all covered, you have the full picture of the architecture that reshaped modern AI.

The teaser closes the day and opens the next topic. Having learned what a Transformer is, why it matters, how attention works, how to build one, and how it fails, the natural next step is how these models are actually trained at scale — the optimization, data, and engineering that turn an architecture into a capable model. The series continues there.