Self-Attention vs Cross-Attention

This post answers the 'so what.' Knowing that self- and cross-attention differ only in their inputs is only useful once you see why that one difference reshapes what an architecture can do. Pick the wrong flavor for a task and you don't get an error — you get a model that structurally cannot connect the things your task requires connected.

The through-line is task fit. Self-attention owns the job of understanding a single sequence; cross-attention owns the job of conditioning one sequence on another. From those two facts you can predict which blocks appear in an encoder-decoder model, why GPT-style models need only one flavor, and why multimodal systems reach for the other. This post makes that mapping explicit.

Self-attention is the workhorse for understanding. Whenever the goal is to build a deep representation of a single sequence — a sentence, a long document, or an image broken into patches — self-attention lets every element gather relevant context from the rest of the same sequence. Meaning that depends on surrounding context gets folded into each position's representation.

This is why encoders are stacks of self-attention, why BERT-style models are pure self-attention, and why a Vision Transformer applies self-attention over image patches. The common thread is that there is one sequence and the task is to understand it richly. No second sequence is involved, so no bridge is needed — self-attention alone does the work.

Cross-attention is the only attention that bridges two sequences, which makes it indispensable whenever the output must depend on a separate input. Translation conditions each target word on the source sentence; summarization conditions a summary on an article; image captioning conditions generated words on an image; retrieval-augmented generation conditions output on fetched documents.

The critical point is that self-attention cannot do this on its own. Self-attention only relates positions within one sequence. If your task requires information to flow from sequence A into the generation of sequence B, something has to connect them, and cross-attention is the standard mechanism that does. Remove it from a conditional generation model and the conditioning disappears.

This comparison maps concrete tasks to the flavor that owns them. On the self-attention side: encoding a sentence, reading a document, processing image patches in a ViT, and next-token prediction in a decoder-only model. On the cross-attention side: machine translation, summarization, image captioning, and retrieval-augmented generation.

The pattern that emerges is clean. Single-sequence understanding and within-stream generation are self-attention jobs. Any task whose name implies two distinct things being connected — translating between languages, grounding text in an image, conditioning on retrieved text — is a cross-attention job. Memorizing a few examples on each side gives you a fast heuristic for which flavor a new task will need.

Encoder-decoder Transformers need both flavors, each in a fixed role. The encoder uses self-attention to build a deep understanding of the source sequence. The decoder uses masked self-attention to understand what it has generated so far, and cross-attention to pull relevant information from the encoder's output as it generates each new token.

The cross-attention is what makes the decoder's output depend on the input. Remove it and the decoder is generating blind — it has no access to the source it is supposed to translate or summarize, so it can only produce fluent but unconditioned text. This is the clearest demonstration that the two flavors are complementary, not interchangeable: the model needs self-attention to understand and cross-attention to connect.

This stack diagram shows where each flavor lives in a full Transformer. The encoder block uses self-attention only. The decoder contains two attention sublayers in order: first masked self-attention over the target sequence generated so far, then cross-attention in which the target reads the encoder's output. The final output is therefore conditioned on the source.

Laying the layers out vertically makes the fixed roles concrete. Self-attention appears wherever a sequence must understand itself; cross-attention appears exactly once per decoder block, as the bridge to the encoder. When you read a Transformer diagram, this is the structure to look for — and the single cross-attention sublayer is the component that carries the conditioning.

Why do GPT-style decoder-only models use no cross-attention at all? Because they have no separate encoder to cross-attend to. There is only one sequence: the prompt followed by the generated tokens, all living in a single stream. Masked self-attention over that combined stream handles everything — reading the prompt and continuing it are both just attending to earlier positions in the same sequence.

Conditioning still happens, but it happens internally. The completion conditions on the prompt because the prompt tokens are earlier positions that the self-attention can reach. There is no second sequence and therefore nothing to bridge, so the cross-attention sublayer is simply absent. This is why the entire GPT family is built from a single attention flavor.

This snippet contrasts the two architectural patterns directly. In a decoder-only model, the prompt and the generated text are concatenated into one sequence, and a single masked self-attention call handles both — conditioning is internal to the stream. In an encoder-decoder model, the source is first encoded into a separate memory, and then cross-attention lets the target read that memory; conditioning crosses between two distinct sequences.

Seeing them side by side clarifies why the choice of architecture determines the choice of flavor. If your input and output naturally share one stream, self-attention alone suffices. If they are genuinely separate sequences that must be encoded independently, you need cross-attention to connect them. The flavor follows from the data layout.

Modern multimodal and retrieval systems lean heavily on cross-attention because it is the natural way to inject one source into another. A vision-language model lets text tokens attend to image features; a retrieval-augmented generator lets the decoder attend to fetched documents; an audio model lets text attend to acoustic features. In each case, cross-attention is the plumbing that lets a generator condition on an external source it did not produce.

The reason cross-attention is the default here is that the two sources are genuinely different sequences, often of different lengths and even different modalities, encoded separately. Cross-attention handles that asymmetry gracefully — the query side and the key/value side need not match in length or origin — which is exactly the property a multimodal bridge requires.

This bar chart dramatizes the cost of choosing the wrong flavor. When the flavor matches the task, fit is high. When a task that needs conditioning across two sequences is built with self-attention only, fit collapses — the model structurally cannot read the separate source. And a cross-attention block with no real source to read is worse still, attending to nothing useful.

The lesson the chart drives home is that this is not a tuning problem you can fix with more data or a bigger model. A missing bridge is an architectural deficiency: if the path for information to cross between sequences does not exist, no amount of training creates it. Matching the flavor to the task is a design decision you must get right up front.

This recap gathers the argument into five points: self-attention is for deep understanding of one sequence; cross-attention is for conditioning on a separate sequence; encoder-decoder models use both in fixed roles; decoder-only LLMs use self-attention only because they have no encoder; and multimodal models bridge modalities with cross-attention.

Together these let you predict an architecture from its task. With the motivation established and the task mapping clear, the next post opens the hood and traces the exact Q, K, V wiring and the tensor shapes that make each flavor work.

The teaser sets up the mechanics post. With the task-fit argument in hand, the next step is to make the wiring concrete: how the projections differ between the flavors, why self-attention scores are square while cross-attention scores are rectangular, and how the masking rules differ between decoder self-attention and cross-attention.