Positional Encodings

This post is the on-ramp for the whole day. Before we argue why position matters, trace the sinusoidal math, build it in code, or catalog the traps, you need a clean mental model of what positional encoding actually is — the extra information we add so a model can tell where each token sits in a sequence.

The goal here is to lock in the core intuition and vocabulary so the later posts have somewhere to stand. The formula, the PyTorch toolkit, and the common mistakes are all consequences of the single idea introduced on this cover: self-attention is order-blind, so order must be supplied deliberately.

Positional encoding is, at its simplest, a vector of the same size as a token embedding that says where in the sequence that token lives. You compute it once per position and combine it with the token's content vector so the model carries both 'what the token is' and 'where it sits' in a single representation.

The thing to hold onto is that this turns an unordered operation into an ordered one. Self-attention on its own treats the input as a set; adding position lets the very same attention math distinguish first from last, adjacent from distant. Everything later in the day is about how to compute that 'where' vector well.

Attention's order-blindness is the precise problem positional encoding solves. Self-attention scores every token against every other token using dot products, and nothing in that computation references a token's index. If you permute the inputs, you simply get the same outputs permuted the same way — the mechanism has no anchor to absolute or relative position.

This is why people call attention permutation-equivariant. It is a feature for sets, where order is meaningless, but a fatal flaw for sequences, where order is meaning. Because the attention math itself can never recover position, the information has to be injected before attention ever runs. That is the entire motivation for the rest of the day.

The 'same set, different meaning' comparison makes the stakes concrete with the classic example. 'Dog bites man' and 'man bites dog' are built from identical tokens; only the order differs, and that order flips the meaning entirely. To a model that sees only an unordered set of vectors, the two sentences are indistinguishable.

The right column shows the fix: tag each token with its index so the bag becomes a sequence. Once 'dog' is known to be at position zero and 'man' at position two, the model can recover the relationship between them. This single contrast is the cleanest argument for why positional encoding exists at all.

Where the encoding goes is as important as what it is. You compute a position vector with exactly the same dimension as the token embedding, then add the two together before the first attention layer. The resulting sum is a single vector per token that fuses content and position.

The reason this works is that downstream layers are powerful enough to disentangle the two signals from their sum, using the dimensions they already have. From the first layer onward, every attention and feed-forward operation can read order out of the combined representation — so you only need to inject position once, at the very bottom of the stack.

The injection-point diagram traces the path a token takes: from its integer ID, through an embedding lookup that produces the 'what' vector, then the addition of a positional 'where' vector, to a combined sum that finally enters attention now able to see order. Each stage does one job.

Visualizing it this way reinforces that positional encoding is a pre-processing step that happens before attention, not something baked into the attention computation (with RoPE as the notable exception, covered later). If you remember one picture for where positions live in a Transformer, this flow — embed, add position, attend — is the one to keep.

Absolute and relative position are the two big families, and the distinction runs through the whole field. Absolute encoding tags each slot with its index: position zero, one, two, and so on. It answers the question 'where am I?'. Relative encoding instead represents the distance between two tokens — three apart, five apart — and answers 'how far apart are we?'.

The reason this matters is that language often cares more about distance than absolute slot: a verb relating to its subject depends on their gap, not their exact indices. Modern models lean heavily toward relative schemes like RoPE and ALiBi precisely because relative position generalizes better across lengths, a theme the next post develops.

Why add the position vector rather than concatenate it? Concatenation would lengthen every token vector, consuming extra dimensions and compute purely to carry position. Addition keeps the dimension fixed and trusts the network to learn to separate content from position within the space it already has.

In practice addition works well, and it is the choice the original Transformer made — a pragmatic, inexpensive decision rather than a deep theoretical necessity. There are dimensions to spare in a high-dimensional embedding, and the model readily learns to allocate some of that capacity to positional signal. Knowing this is a design choice, not a law, helps when you later meet schemes like RoPE that abandon addition entirely.

This snippet makes attention's order-blindness undeniable. We take three token vectors, shuffle them, and compute the attention-style score matrix for both the original and the shuffled set. The sorted score values come out identical, demonstrating that permuting the input only permutes the scores — no information about order survives in the raw attention computation.

The lesson is that this is a property of the math, not a quirk of small examples. Because the dot-product scoring has no reference to position, any order-sensitive task is impossible until you inject position. Seeing it fail in four lines of code is more convincing than any prose argument, and it sets up exactly why the rest of the day matters.

This recap consolidates the mental model into five durable takeaways: attention alone is permutation-blind; positional encoding injects the missing 'where'; it is added to the token embeddings at the same dimension; absolute encoding tags an index while relative encoding tags a distance; and without any of this, sequence order is simply lost.

If you can recite these five points, you are ready for the rest of the day. Each subsequent post assumes this foundation — the why, the sinusoidal math, the code, and the mistakes all build on the picture of injecting position into an order-blind attention mechanism rather than re-explaining it.

The teaser points forward to the 'why it matters' post. Having established what positional encoding is, the natural next question is what it actually buys — why word order is meaning rather than decoration, what a model loses without positions, and why the hard problem of generalizing to longer sequences drove the invention of RoPE, ALiBi and the rest. The next post takes up exactly that argument.