Positional Encodings

This is the hands-on post, and it is intentionally code-heavy. The previous posts built understanding from concept through mechanics; here you assemble a working positional-encoding toolkit, wire it into an embedding layer, and verify on real tensors that the encodings behave as the theory promised.

Each snippet is real, runnable code that fits together: the sinusoidal table, the addition into token embeddings, a learned alternative, a minimal RoPE, a forward pass with shape checks, and a verification of the relative-distance property. The most valuable thing you can do is run it, print the shapes, and plot a row to watch the wave — that experimentation is how the formula from the previous post turns into intuition you own.

This is the sinusoidal encoding as a clean function and a near-direct transcription of the formula. It allocates a zero table of shape sequence-length by d, builds a position column, and computes the per-dimension frequency divisor using the exponential-of-log form, which is the numerically stable way to write 10000 raised to a negative power. It then writes sine into the even-indexed columns and cosine into the odd ones.

The slicing with 0::2 and 1::2 is the practical realization of 'sine on even dimensions, cosine on odd.' Writing the divisor as exp of a scaled negative log avoids overflow that a naive power could cause for large dimensions. The function returns a table you can add to any batch of embeddings, and it is defined for any sequence length you ask for.

This module shows the correct way to add the encoding to token embeddings. It owns a token embedding layer and registers the precomputed positional table as a buffer rather than a parameter, so it moves with the model to GPU but is never updated by gradient descent. In the forward pass it embeds the token IDs and adds the positional table sliced to the actual sequence length.

Two details matter. Registering the table as a buffer is what keeps it from being treated as a learnable weight while still being saved and device-managed correctly. Slicing the table to x.size(1) ensures the addition lines up with however many tokens the current batch has, broadcasting cleanly over the batch dimension. This little module is exactly what sits at the bottom of a real Transformer encoder.

This is the learned alternative, which trades the fixed formula for trainable position vectors. It wraps an embedding layer indexed by position rather than by token. In the forward pass it builds an index range up to the current sequence length, looks up the corresponding learned vectors, and adds them to the input, broadcasting over the batch.

The contrast with the sinusoidal version is instructive: here the position vectors are parameters the optimizer tunes, so they can fit the data more flexibly, but they exist only up to max_len. Note the device handling — building the index on the same device as x avoids a subtle cross-device error. This is the scheme BERT and GPT-2 used, and seeing it next to the sinusoidal version makes the fixed-versus-learned trade-off tangible.

This minimal RoPE implementation makes the 'rotate, don't add' idea concrete. It computes a per-position, per-dimension-pair angle theta, then takes the cosine and sine of those angles. It splits the input into even and odd dimensions, treats each pair as a 2D point, and rotates it by the angle — the standard rotation formula with the cosine and sine terms — before flattening the pairs back into a full vector.

The crucial structural requirement is that d be even, because RoPE operates on pairs of dimensions. After this rotation, the dot product between a rotated query and a rotated key depends only on their relative offset, which is the whole point of RoPE. This is a stripped-down version for understanding; production implementations add caching and broadcasting, but the rotation core is exactly this.

This is the moment of truth: instantiate the embedding module, feed it a real batch of token IDs, and check the output shape. With a vocabulary of 100 and dimension 32, an input of shape (2, 10) — two sequences of ten tokens — should produce an output of shape (2, 10, 32), since the positional addition preserves the embedding shape exactly. Printing the standalone sinusoidal table confirms it is (10, 32), one vector per position.

Verifying these shapes is the fastest sanity check that the wiring is correct. If the output is not (2, 10, 32), the addition or slicing is wrong. Reading the printed shapes is how you confirm the positional encoding integrates cleanly with token embeddings before you ever start training a model on top of it.

This snippet verifies the property that justifies the whole sinusoidal design: distance in encoding space tracks the gap between positions. It measures the norm of the difference between encodings five positions apart at two different locations in the sequence — positions 5 and 10, then 25 and 30 — and the two distances come out close, because both pairs share a gap of five.

This is the empirical fingerprint of the linear-offset property discussed in the mechanics post: the relationship between two encodings depends mainly on how far apart they are, not where they sit absolutely. Confirming it on real tensors is reassuring and also a useful debugging habit — if same-gap pairs gave wildly different distances, your encoding would be broken. It is a concrete way to check that positions behave as relative as well as absolute signals.

The data-flow diagram traces tensor shapes through the embedding stage, which is the part most people wire incorrectly. Token IDs of shape (batch, time) go through the token embedding to become (batch, time, d), the positional table of shape (time, d) is added by broadcasting, and the result of shape (batch, time, d) flows into the encoder now carrying order.

Being able to recite this chain lets you catch a mismatch by reasoning rather than trial and error. The one broadcast step — adding a (time, d) table to a (batch, time, d) tensor — is where shape bugs hide, so seeing it laid out as a flow makes the correct alignment obvious. When something throws, mentally walking these shapes usually reveals whether the slice or the broadcast is the culprit.

These shapes-and-gotchas are where practitioners lose time with positional encodings. The table is (sequence-length, d) and broadcasts over the batch, so you never tile it manually. It should be registered as a buffer, not a parameter, so it is not accidentally trained. You must slice it to the actual sequence length, taking pe up to T, so it lines up with the batch. RoPE requires an even dimension because it rotates pairs. And learned positions have a hard cap at max_len that you must respect.

None of these are conceptual misunderstandings of positional encoding; each is a concrete implementation trap. Forgetting the buffer registration makes your table a stray parameter; getting the slice wrong shifts every position; running RoPE on an odd dimension fails or silently drops a dimension. Internalizing this list saves a great deal of debugging, and several of these reappear as the silent bugs in the final post.

This comparison frames the choice between writing positional encodings yourself and using a library implementation. The from-scratch version exposes every term, is ideal for learning, and is trivial to visualize and inspect — at the cost of more code you could get subtly wrong. The library version, such as the RoPE baked into Hugging Face models, is tested, fast, and reduces boilerplate, which is what you want in production.

The sensible workflow uses both: implement it once by hand to understand and to be able to debug, then rely on the library for any real training run. Understanding the manual version is precisely what lets you trust the library and diagnose it when a long-context run starts misbehaving, so the two are complementary rather than competing approaches.

This recap consolidates the build: the sinusoidal function returns a (time, d) table; you add it to token embeddings sliced to the sequence length; the learned variant is just an embedding over positions; RoPE rotates pairs of dimensions by a position-dependent angle; and you should verify the relative-distance property to confirm your encoding behaves.

With a working positional toolkit in hand — sinusoidal, learned, and rotary, all verified on real tensors — the only thing left is to learn the traps. The final post catalogs the quiet mistakes that make positional encoding produce wrong results without ever raising an error.

The teaser points to the common-mistakes post. You now have positional encodings that run and whose shapes and properties check out; the next step is hardening your instincts against the silent failures — forgetting to add the encoding at all, off-by-one slicing, exceeding a learned scheme's max length, double-counting position, and expecting free extrapolation — each with its clean fix.