Self-Attention vs Cross-Attention

This is the engine-room post. The previous two established what the two flavors are and why each matters; here we replace intuition with the explicit wiring, tracing exactly how Q, K, and V are produced in each case and what shapes flow through the pipeline.

The payoff is that the two flavors stop blurring. By the end you will have seen that the four-step core is shared, that the projections differ only in their input source, that self scores are square while cross scores are rectangular, and that the masking rules diverge. Tracing the shapes once is worth more than re-reading the high-level distinction ten times.

Both flavors compute the identical formula: Attention of Q, K, V equals softmax of Q times K-transpose over the square root of d_k, all times V. The four operations — score, scale, softmax, blend — are byte-for-byte the same regardless of flavor. Nothing in this core distinguishes self from cross.

The distinction lives one step earlier, in how Q, K, and V are produced. This is the single most important thing to hold onto from the whole day: do not look for the difference inside the attention computation, because it is not there. Look at the inputs being projected into the query, the key, and the value. That is where, and the only where, self and cross part ways.

The projections are where the flavors diverge. In self-attention, a single input X is projected three ways — Q equals X times Wq, K equals X times Wk, V equals X times Wv — so all three derive from the same sequence. In cross-attention, the query is projected from the decoder sequence Y while the key and value are projected from the encoder sequence Z: Q from Y, but K and V from Z.

Note that each flavor still has its own learned weight matrices Wq, Wk, Wv; the difference is purely which tensor those matrices are applied to. This is worth stressing because beginners sometimes imagine cross-attention shares weights with self-attention or skips the projections. It does neither — it runs the same kind of learned projections, just on different input sequences.

This comparison places the two wirings side by side with their resulting score shapes. On the left, self-attention takes Q, K, and V all from input X, and because queries and keys come from the same T-length sequence, the score matrix is T by T — square. On the right, cross-attention takes Q from target Y and K, V from source Z, producing a score matrix of T_y by T_z — rectangular.

The score-shape column is the practical fingerprint of each flavor. A square score matrix means the queries and keys share an origin: self-attention. A rectangular one means they come from different sequences: cross-attention. When debugging, printing the shape of the score or weight tensor immediately tells you which flavor you are actually running, regardless of what you intended.

The square-versus-rectangular distinction has real meaning, not just bookkeeping. In self-attention, T queries score against T keys from the same sequence, so the T-by-T matrix relates every position to every other position within one sequence. In cross-attention, T_y target queries score against T_z source keys, so the T_y-by-T_z matrix is an alignment between two different sequences.

That rectangle is exactly the alignment map people visualize in translation: row i shows which source positions the i-th output token attended to. Its rectangular shape directly reflects that the target and source can have different lengths — a five-word output aligning to a seven-word input gives a five-by-seven map. Recognizing the rectangle as a cross-sequence alignment, rather than a within-sequence relation, is key to reading these maps correctly.

This pipeline diagram traces the tensor shapes through cross-attention specifically, because that is where shape confusion is most common. The query from the target has shape T_y by d. The transposed key from the source has shape d by T_z. Their product gives a score matrix of T_y by T_z — the alignment. Multiplying that by the value, shape T_z by d, yields an output of shape T_y by d, one vector per target position.

The key takeaway is that the output length matches the query length, T_y, not the source length. Cross-attention produces exactly one output vector per target position, each one a blend of source values weighted by how relevant that source position was. Walking these four shapes makes the rectangular score matrix feel inevitable rather than surprising.

This snippet is the entire distinction in code: one attention function, two call sites. The function scores q against k, scales, optionally masks, softmaxes, and blends v — knowing nothing about flavor. Calling it with x, x, x and a causal mask is decoder self-attention, producing T-by-T scores. Calling it with y, z, z and a padding mask is cross-attention, producing T_y-by-T_z scores.

Notice that the mask differs between the two calls, which previews the next slide: self-attention here uses a causal mask while cross-attention uses a padding mask. The function does not enforce this — it simply applies whatever mask you pass. That flexibility is exactly why the same code can serve both flavors and why getting the mask right at the call site is your responsibility, not the function's.

Masking is where the two flavors diverge beyond just their inputs, and confusing the rules is a classic bug. Decoder self-attention uses a causal mask so that a token cannot attend to positions after it — it must not see the future it is being trained to predict. Cross-attention, by contrast, is not causal: the decoder is allowed to look at the entire source sequence at once, because there is no notion of 'future' on the source side.

What cross-attention does need is a padding mask, to ignore pad tokens in the source. So the rule is: causal masking for decoder self-attention, padding masking for cross-attention, and both kinds of padding masking wherever batched sequences are involved. Applying a causal mask to cross-attention is a real mistake — it would block the decoder from the latter half of the source — and the mistakes post returns to it.

This snippet makes the two masks concrete and shows how different they are. The causal mask for decoder self-attention is a lower-triangular T-by-T matrix: position t may attend to positions zero through t, never beyond. The cross-attention mask is built from the source padding pattern instead — it marks which source positions are real tokens versus padding, then broadcasts across all T_y target positions to form a T_y-by-T_z mask.

The contrast is the point. The causal mask is square and triangular, encoding temporal order within one sequence. The cross mask is rectangular and column-based, encoding only which source positions are valid. They are not interchangeable, and a wrong mask here is a silent bug: the shapes may broadcast without error while the model attends to the wrong positions.

This flow diagram shows how a single decoder block stacks both flavors in sequence. The target enters, passes through masked self-attention where it reads itself with a causal mask, then through cross-attention where it reads the encoder output Z, then through a feed-forward sublayer with normalization, and out to predict the next token.

The ordering is fixed and meaningful: self-attention first so the target understands its own generated context, then cross-attention so it can pull in the relevant source information, then the feed-forward layer to process the combined representation. Seeing both flavors in one block makes concrete the earlier claim that encoder-decoder models use them together in distinct roles — this is literally where the two sit side by side.

This recap pins down the mechanics: the score-scale-softmax-blend core is shared; self-attention projects Q, K, V from one input while cross-attention takes Q from the target and K, V from the source; self scores are square T-by-T while cross scores are rectangular T_y-by-T_z; decoder self-attention is causal while cross-attention is not; and a decoder block runs self-attention, then cross-attention, then the feed-forward sublayer.

With the wiring and shapes traced by hand, you are ready to build the real thing. The next post assembles a single PyTorch module that serves both flavors and runs a forward pass that makes the square-versus-rectangular score shapes visible.

The teaser points to the hands-on build. Having traced the projections, the shapes, and the masking rules, the next post wires it all into a real PyTorch module — one attention layer that accepts separate query and key/value inputs, called one way for self-attention and another for cross-attention, with the differing score shapes printed out to prove the distinction.