Positional Encodings

This is the engine-room post. The previous two established what positional encoding is and why order matters; here we replace intuition with the original sinusoidal formula, tracing exactly how each position vector is built and why the design choices are deliberate rather than arbitrary.

The payoff is that the wall of sines and cosines stops being mysterious. By the end you will have read the formula term by term, understood why many frequencies are used instead of one, seen the linear-offset property that quietly enables relative reasoning, and met the rotary embeddings that most modern LLMs now prefer. Tracing it once is worth more than re-reading the formula ten times.

The sinusoidal formula assigns each position a vector whose entries are sinusoids of geometrically increasing wavelength. For a position pos and dimension index i, even dimensions take the sine and odd dimensions take the cosine of pos divided by 10000 raised to the power 2i over d, where d is the model dimension. The base of 10000 sets how slowly the longest wavelengths vary.

The key structural fact is that different dimensions oscillate at different rates: low dimensions have short wavelengths and change quickly with position, high dimensions have very long wavelengths and change slowly. This spread of frequencies, packed into one vector, is what lets a fixed-size encoding distinguish a large range of positions, and it is the foundation for everything else in the post.

Breaking the formula into its named parts makes it easy to hold in your head. pos is which position in the sequence you are encoding. i indexes which dimension of the output vector you are filling. d is the total embedding dimension, fixed for the model. The term 10000 raised to 2i over d sets the wavelength for that dimension, growing as i grows. And the sine-on-even, cosine-on-odd convention pairs each frequency into a sine and a cosine.

Laying it out this way clarifies that there are really only two moving inputs — position and dimension — feeding a deterministic function. Nothing is learned; the entire table is fixed by the formula. Knowing each symbol's role makes the from-scratch code two slides later read as a direct transcription rather than something new to decode.

Why use many frequencies instead of a single sine wave? A single sinusoid repeats every wavelength, so two positions a full period apart would receive identical encodings — a collision that destroys the very information you are trying to encode. Stacking many frequencies, fast and slow together, makes each position's combined pattern unique across a huge range.

The clock analogy captures it: an hour hand alone cannot tell 1:00 from 1:30, but hour, minute, and second hands together pin an exact instant. In the encoding, slow dimensions act like the hour hand giving coarse position, while fast dimensions act like the second hand giving fine resolution. The combination yields a unique fingerprint for every position the model will ever see.

This bar chart visualizes the spread of oscillation frequencies across dimensions, which is the heart of the design. Early dimensions oscillate quickly — short wavelengths that change with every step of position — while later dimensions oscillate slowly, with wavelengths so long they barely move across the whole sequence.

Reading the chart, you can see why the encoding works as a multi-resolution code: fast dimensions resolve fine, local position differences, and slow dimensions carry coarse, global position. The chart is schematic rather than exact, but the monotonic decrease in frequency across dimensions is precisely what the 10000-to-the-2i-over-d term produces, and it is what gives every position a distinct combined signature.

The linear-offset property is the quiet bit of mathematical elegance that makes sinusoidal encoding more than a lookup table. Because of the angle-addition identities for sine and cosine, the encoding at position pos+k can be written as a fixed linear transformation of the encoding at pos — and crucially, that transformation depends only on the offset k, not on pos itself. The same rotation maps every position to its k-step neighbor.

The practical consequence is that a model can learn to reason about relative distance using nothing but these absolute codes: to attend 'three tokens back,' it can apply a fixed linear map regardless of where it currently is. This is how absolute sinusoidal encoding smuggles in relative-position capability for free, and it foreshadows why later schemes like RoPE make relative position even more explicit.

This from-scratch implementation makes the formula concrete and runnable. It builds a column of positions and a row of dimension indices, forms the angle as position divided by 10000 raised to the appropriate power, then writes sine into the even dimensions and cosine into the odd ones using a where-based selection. The result is a table of shape sequence-length by model-dimension.

The code is a near-literal transcription of the formula, which is the point: once you have traced the math, the implementation holds no surprises. Printing the shape confirms you get one vector per position at the model's dimension. Running this and plotting a single row reveals the characteristic banded wave pattern, turning the abstract formula into something you can actually see.

This comparison sets fixed sinusoidal encoding against learned positional embeddings, the two classic absolute schemes. The sinusoidal version has no parameters, is defined by formula for any position no matter how large, and therefore extrapolates beyond the training length by construction — it was the original Transformer's choice. Learned embeddings instead store one trainable vector per position, which can fit the data more flexibly but are capped at a maximum length and have no defined behavior beyond it; this is what BERT and GPT-2 used.

The trade-off is flexibility versus reach. Learned embeddings can adapt to quirks of the training distribution, but they cannot represent a position they never saw. Sinusoidal embeddings are rigid but unbounded. This tension — fit versus extrapolation — is exactly what later schemes like RoPE try to resolve.

Rotary Position Embedding, RoPE, abandons the add-a-vector approach entirely. Instead of summing a position vector into the embedding, it rotates the query and key vectors by an angle proportional to their position before the attention dot product. Because rotating both vectors and then taking their dot product yields a result that depends only on the difference of their angles, the attention score between two tokens ends up depending on their relative distance rather than their absolute positions.

This is a clever reframing: relative position falls directly out of the attention computation, with no separate relative-position table. RoPE tends to extrapolate to longer sequences better than learned absolute embeddings and integrates naturally with attention, which is why it has become the default in most modern large language models. The code post implements a minimal version so the rotation is no longer abstract.

This flow diagram lines up the major positional schemes by where and how they inject order. Sinusoidal and learned embeddings both add a vector to the input embedding, differing only in whether that vector is computed or looked up. RoPE rotates the query and key vectors inside attention. ALiBi takes yet another route, adding a distance-based penalty directly to the attention scores so that far-apart tokens are attended to less.

Seeing them side by side clarifies that 'positional encoding' is a family of strategies, not one technique. They differ in whether they touch the input or the attention, whether they encode absolute or relative position, and how well they extrapolate. Knowing the landscape helps you read any model's config and immediately understand how it handles order.

This recap pins down the mechanics: sine goes on even dimensions and cosine on odd; wavelengths span a geometric range from short to very long; using many frequencies gives every position a unique fingerprint; the offset between two positions is a fixed linear transform, which yields relative reasoning from absolute codes; and RoPE rotates the query and key vectors to encode relative position directly in attention.

With the math traced by hand, you are ready to build the real thing. The next post implements sinusoidal encoding, a learned variant, and a minimal RoPE in PyTorch, then verifies the relative-distance property on actual tensors so the formula becomes a runnable artifact.

The teaser points to the hands-on build. Having traced the sinusoidal formula, understood the multi-frequency design and the linear-offset property, and met RoPE and ALiBi, the next post wires it all into real PyTorch: a sinusoidal table, an embedding layer that adds it, a learned alternative, a minimal rotary implementation, and a check that the relative property actually holds.