Self-Attention vs Cross-Attention

This closing post is a field guide to the mistakes specific to mixing self- and cross-attention. The earlier posts gave you the distinction, the task argument, the mechanics, and a working module; this one inoculates you against the failures that do not announce themselves with a clean error.

The theme is that these bugs are silent. Because self- and cross-attention share the exact same machinery, wiring them wrong rarely changes the shapes enough to crash — the model just learns the wrong alignment, or no alignment at all, while training proceeds and the loss curve looks plausible. Each mistake here comes with a one-line fix, so the value is in being able to NAME the problem the instant you see its symptom.

Feeding the keys and values from the wrong sequence is the signature cross-attention bug. Cross-attention must take K and V from the encoder and Q from the decoder. If you accidentally wire the decoder's own states into K and V, you have silently converted cross-attention back into self-attention — the decoder reads only itself and never touches the source.

What makes this so dangerous is that the shapes frequently still line up, since the decoder states are valid tensors of a compatible dimension, so no error fires. Training runs, the loss descends on the language-modeling component, and the model simply never learns to use the input. The result is a translator that produces fluent output unrelated to the source, and the fix is to be deliberate: Q from the decoder, K and V from the encoder, every time.

This snippet contrasts the wrong and right wiring so the fix is unmistakable. The wrong call passes the decoder state into all three of query, key, and value, which is self-attention masquerading as cross-attention — the source is never read. The right call passes the decoder state as the query but the encoder memory as both the key and the value, so the decoder actually attends to the source.

The difference is two arguments, but the effect is the difference between a model that conditions on its input and one that ignores it entirely. Because the wrong version produces no error and a believable loss, the only defense is to make the correct wiring a conscious habit and to verify, by printing the attention shape, that the score matrix is rectangular — proof that the keys really came from a differently-sized source.

Putting a causal mask on cross-attention is a mistake born of pattern-matching from decoder self-attention. Causal masking exists so a token cannot see the future it is predicting — but that logic applies only within the target sequence. On the source side there is no future to hide: the decoder is entitled to see the entire source at once when deciding what to generate.

Apply a causal mask to cross-attention and you cripple it. Early target positions would be blocked from attending to later source positions, so the model loses access to the back half of the input it is supposed to translate or summarize. The output degrades in a way that looks like a capacity or data problem but is really a misplaced mask. The rule is simple: causal masking belongs to decoder self-attention only, never to cross-attention.

This decision tree turns the masking question into a quick check you can run on any attention block. If the block is decoder self-attention, it needs a causal mask to hide the future. If it is cross-attention, it needs a padding mask only — never causal — because the source has no future to hide. If it is encoder self-attention, it likewise needs only a padding mask, since the encoder reads the whole source bidirectionally.

The broader lesson is to decide the mask from the block's role, not by copying whatever mask the previous block used. Causal masking is the exception, reserved for one specific place — the decoder's self-attention — and applying it anywhere else is almost always a bug. Running this three-way check on each attention sublayer catches the most common masking errors before they cost you a training run.

Forgetting the source padding mask corrupts cross-attention in any batched setting. To process source sentences of different lengths together, you pad the short ones up to a common length with dummy tokens. In cross-attention, the decoder will attend to those pad positions unless you mask them, blending meaningless filler into its conditioned output.

The fix mirrors the padding mask used elsewhere: before softmax, set the score columns corresponding to padded source positions to negative infinity, so their post-softmax weights are exactly zero and the decoder aligns only to real source tokens. Note that this is a padding mask over the source — distinct from the causal mask used in decoder self-attention — and it is the mask cross-attention actually needs. Omitting it lets the model waste attention on filler and can noticeably degrade alignment quality.

This snippet shows the source-padding fix in the cross-attention setting. Given a source mask that is true at real source positions and false at padding, you use masked_fill to set the scores at padded source positions to negative infinity before softmax. The indexing with None inserts broadcasting axes so the per-source-sequence mask lines up with the score tensor's shape, batch by heads by target-length by source-length, masking the source (key) dimension.

After softmax those padded source positions carry exactly zero weight, so the decoder never aligns to filler. The crucial detail is that the mask applies to the LAST axis — the source/key dimension — because that is the axis being normalized over. Getting that axis right is the only fiddly part, and a wrong shape here typically does raise an error, making this trap less silent than the wrong-source bug.

Assuming the query and key sequences are the same length is a self-attention habit that breaks cross-attention. In self-attention the query and key come from one sequence, so their lengths match and the score matrix is square — and a lot of code, especially hand-rolled masks, quietly bakes in that square assumption. Cross-attention scores are target-length by source-length: rectangular, and the two dimensions are generally unequal.

Carry the square assumption into cross-attention and things break in two ways. Code that constructs a square mask or indexes the diagonal will either error or silently misalign; logic that treats the score matrix as symmetric will be simply wrong. The fix is to treat the query and key lengths as independent everywhere, build masks of shape target-by-source for cross-attention, and never assume the attention matrix is square unless you know the call is self-attention.

This comparison contrasts the square self-attention case with the rectangular cross-attention case on the properties that trip people up. In self-attention the query and key lengths are equal, the scores are T-by-T, a causal mask is also T-by-T, and the diagonal is meaningful (a token attending to itself). In cross-attention the query and key lengths may differ, the scores are target-by-source, there is no causal mask, and the rows align target positions to source positions.

The table is a quick reference for spotting which assumptions are safe. Anything that relies on squareness — symmetric masks, diagonal indexing, equal-length loops — is valid only for self-attention. Cross-attention demands code that treats the two axes as independent. Keeping this contrast in mind prevents importing square-matrix assumptions into the rectangular case.

Treating cross-attention maps as proof of alignment is a conceptual mistake that has misled even careful analysis. A cross-attention heatmap looks exactly like a word-alignment table, and it is tempting to read it as the model declaring which source token produced which output token. But a high weight does not prove that source token caused the output: the value vectors, the residual connections that bypass attention, and the transformations in later layers all intervene between the weight and the final prediction.

The responsible stance is to treat the map as a hint about alignment, not as a faithful explanation. Published work has drawn confident conclusions from attention patterns that did not survive more rigorous causal probing. Cross-attention offers a uniquely legible window because it visibly connects two sequences, but mistaking that window for ground-truth alignment is a documented trap worth avoiding.

Believing self-attention can simply replace cross-attention is an architectural mistake worth understanding, because the substitution is tempting and not entirely wrong. You can concatenate the source and target into one sequence and use only self-attention; some modern architectures do exactly this. But it is not free, and pretending the trade-offs do not exist leads to poor designs.

Concatenation costs you three things. The source loses its protected, separately-computed encoding that a cross-attention setup can cache and reuse across decoding steps. The causal mask now entangles source and target in one triangular structure that you must construct carefully. And you pay quadratic attention cost over the combined length rather than over each sequence separately. Cross-attention keeps the source encoded once, cached, and cleanly conditioned — which is why encoder-decoder designs still use it where those properties matter.

The final checklist consolidates the fixes into a pre-flight you can run before trusting any model that mixes the two flavors: in cross-attention, take K and V from the source and Q from the target; never put a causal mask on cross-attention; always mask source padding before softmax; remember cross-attention scores are rectangular and never assume a square matrix; treat attention maps as hints about alignment rather than proof; and recognize that self-attention can mimic cross-attention only at a real cost in caching, masking, and compute.

Run through this list and you avoid the failures that cost most people their first weeks with encoder-decoder and multimodal models. With the distinction, the task fit, the mechanics, a working build, and now the traps all covered, you have the full picture of self- versus cross-attention.

The teaser closes the day and opens the next topic. Having mastered the two attention flavors — what separates them, why each matters, how they wire up, how to build them, and how they fail — the natural next step is to see how these attention sublayers stack with feed-forward layers, residual connections, and normalization to form the full Transformer block. The series continues there.