Self-Attention vs Cross-Attention

This post is the on-ramp for the whole day. Before we argue why the choice matters, trace the shapes, build it, or list the traps, you need a clean mental model of the single distinction at the heart of every Transformer: self-attention versus cross-attention. The surprising truth is that they are the same mechanism with one knob turned.

The goal here is to lock in that distinction so it never blurs again. Everything later in the day — the task-fit argument, the shape trace, the PyTorch build, the silent bugs — is a consequence of where exactly the queries, keys, and values come from. Get the source of the arrows right and the rest of the day is detail.

The entire difference between the two flavors fits in one line: in self-attention, the queries, keys, and values are all derived from the same sequence; in cross-attention, the queries come from one sequence while the keys and values come from another. Everything downstream of that choice — scoring with a dot product, dividing by the square root of the key dimension, normalizing with softmax, blending the values — is byte-for-byte identical.

The thing to internalize is that there is no special 'cross-attention math' to learn. If you already understand scaled dot-product attention, you already understand both flavors. The only decision that distinguishes them happens one step earlier, when you decide which input tensors get projected into Q, into K, and into V.

Self-attention is a sequence looking at itself. Each token forms a query, scores that query against the keys of every token in the same sequence, and blends their values to produce a context-aware version of itself. This is how the pronoun 'it' learns to attend to the noun it refers to, how a verb attends to its subject, and how a token deep in a paragraph absorbs relevant context from far away.

The output of self-attention is the same sequence, re-represented so that every position now carries information about the positions it found relevant. Nothing leaves or enters the sequence; it simply becomes richer. This is why self-attention is the engine of understanding — it deepens the representation of whatever single sequence you feed it.

Cross-attention is one sequence looking at another. The decoder forms queries from the tokens it is generating and scores them against the keys of the encoder's output, then blends the encoder's values. The result is that each position in the target sequence pulls in exactly the parts of the source sequence it needs.

The canonical example is translation: as the decoder generates each French word, its query attends over the encoded English sentence, aligning the output word to the relevant source words. The same pattern powers image captioning, where words attend to image regions, and summarization, where summary tokens attend to the article. Cross-attention is the mechanism that lets information flow between two distinct sequences.

Stating where Q, K, and V come from in each flavor removes all ambiguity. In self-attention, all three are projected from a single input sequence X, so the query and key sequences are necessarily the same length. In cross-attention, the query is projected from the decoder sequence while the key and value are projected from the encoder sequence, so the query length and the key length can — and usually do — differ.

That length difference is not a detail; it is a tell. Whenever you see an attention block whose query and key/value inputs have different lengths, it is cross-attention. Whenever they are forced to be identical because they come from one tensor, it is self-attention. Tracking the source and the length of each input is the most reliable way to tell the two apart in real code.

This comparison sets the two wirings side by side to make the symmetry obvious. On the left, self-attention draws Q, K, and V all from the same input X, so the operation is 'X attends to X.' On the right, cross-attention draws Q from the decoder while K and V come from the encoder, so the operation is 'target attends to source.'

The deliberate visual point is that the boxes are the same machine — only the input cables differ. Seeing them aligned this way prevents the common misconception that cross-attention is a fundamentally different module. It is the identical attention block; you have simply plugged a different sequence into the key and value sockets.

This flow diagram shows cross-attention acting as a bridge between two sequences. The encoder processes the source and exposes its keys and values — 'what is available to read.' The decoder contributes the query — 'what I need right now.' The matching of query against keys, followed by blending the values, produces an output aligned to the relevant source positions.

Reading it as a bridge captures the essential role: cross-attention is the channel through which information crosses from one sequence to another. The encoder side answers 'here is what I contain,' the decoder side asks 'give me what's relevant,' and the blend is the answer. No other component in the architecture connects two separate sequences this directly.

This snippet is the whole argument in code: a single attention function, called two different ways. The function itself knows nothing about self or cross — it just scores Q against K, scales, softmaxes, and blends V. Calling it with the same tensor three times, attention(x, x, x), is self-attention. Calling it with the decoder state for Q and the encoder output for K and V, attention(dec, enc, enc), is cross-attention.

That one function serves both flavors is the clearest possible demonstration that the distinction is purely about inputs. There is no branch inside the function for 'self' versus 'cross.' The flavor is decided entirely by the call site, which is exactly why the rest of the day can treat them with shared machinery.

Why does the architecture bother with two flavors at all? Because they do genuinely different jobs. Self-attention builds rich internal representations by letting each token absorb context from its own sequence — it is the tool for understanding a single input deeply. Cross-attention bridges two different sequences so that information can flow from one to the other — it is the tool for conditioning one sequence on another.

A full encoder-decoder Transformer needs both, in fixed roles: self-attention in the encoder and decoder to understand each side, and cross-attention in the decoder to connect them. The next post develops this task-fit argument in detail, but the seed is here — each flavor exists because there is a job only it can do.

This recap consolidates the distinction into five durable takeaways: the math is shared and only the Q, K, V sources differ; self-attention has a sequence attend to itself; cross-attention has a target attend to a source; self-attention enriches a representation while cross-attention bridges two sequences; and a full encoder-decoder model uses both.

If you can recite these five points, you are ready for the rest of the day. Each subsequent post assumes this foundation — the task argument, the shape trace, the build, and the mistakes all build on this clean separation rather than re-explaining it.

The teaser points forward to the 'why it matters' post. Having established what each flavor is, the natural next question is what each one is actually for — which tasks demand self-attention, which demand cross-attention, why encoder-decoder models need both, and why decoder-only LLMs quietly dropped one of them. The next post takes up exactly that argument.