Positional Encodings

This closing post is a field guide to the mistakes that quietly break positional encoding. The earlier posts gave you the intuition, the math, and a working toolkit; this one inoculates you against the failures that do not announce themselves with a clean error message.

The theme is that positional bugs fail silently. A forgotten addition, a one-off slice, or an over-optimistic assumption about length rarely crashes — instead it hands you a model that trains and scores fine in the lab, then degrades on longer inputs or subtly mangles meaning. Each mistake here comes with a one-line fix, so the value is in being able to name the problem the moment you see its symptom.

Forgetting to add the encoding at all is the most common and most embarrassing positional bug. You write a careful sinusoidal table or register a learned embedding, and then the single line that adds it to the token embeddings never makes it into the forward pass. Because the shapes still line up perfectly without the addition, nothing complains.

The model then trains as a bag of words: it learns co-occurrence but never order, plateaus on anything sequence-sensitive, and tempts you to blame the architecture, the data, or the learning rate. The symptom is mediocre performance specifically on order-dependent tasks. The fix is trivial once you suspect it — add the positional vector to the embedding — but you have to think to check it, because there is no error pointing you there.

This snippet contrasts the wrong and right versions so the fix is unmistakable. In the wrong version the positional slice is computed (or worse, commented out) but never added, so the token embedding flows downstream with no positional information. In the right version the positional table, sliced to the sequence length, is added to the token embedding in the same line.

The difference is a single plus operation, but its effect is the entire difference between a sequence model and a bag of words. Because the bug produces no error and only mediocre order-task performance, the habit worth building is to write the addition in the same line as the embedding lookup, so the two are never separated and one cannot be silently dropped.

An off-by-one or length-mismatch error smears the positional signal in a way that is hard to spot. Your table is sized to the maximum length, but a given batch is shorter; if you add the full table you get a shape clash, and if you fix that by slicing with the wrong start or end index, every token ends up paired with the wrong position by one slot. A one-position shift is subtle: training still proceeds, just with consistently misaligned positions.

The result is a quiet accuracy tax rather than a crash. The model learns around the shift to some degree but never gets the clean positional signal it should. The fix is to slice the table from zero to exactly the sequence length and to assert that the sliced shape matches, catching the mismatch deterministically instead of letting it degrade results invisibly.

This snippet shows the correct slicing discipline. You read the actual sequence length T from the tensor, slice the positional table from the start up to T, and add it — explicitly not the full untrimmed table and not an off-by-one window like positions one through T+1. The assertion that the sliced table's shape matches the trailing dimensions of x turns a silent misalignment into a loud, immediate failure.

The broader habit is to make positional alignment checkable rather than assumed. Because a one-slot shift produces no error on its own, adding a cheap assertion is the simplest way to guarantee the positions you add are the positions you intend. It costs nothing at runtime once verified and saves you from a class of bugs that otherwise only show up as a vague accuracy gap.

Exceeding a learned encoding's maximum length is a bug that switches between loud and silent depending on your luck. Learned positional embeddings define exactly max_len vectors, one per trainable position. Feed a sequence longer than max_len and you try to index past the embedding table: if you are lucky it throws an index error, but depending on how indices are produced you can also get a silent clamp or wraparound that maps real positions onto the wrong vectors.

Either way, any position beyond the training maximum is fundamentally undefined for a learned scheme. This is exactly the weakness that sinusoidal and RoPE encodings avoid, since both are defined by formula for any position. The practical lesson: if you chose learned positions, you must enforce the length cap explicitly and plan a retraining or extension strategy before you ever feed longer inputs.

This comparison sorts the schemes by how they handle inputs longer than training. Learned embeddings have a hard cap at max_len, are out of range past it, offer no extrapolation, and require retraining to extend. Sinusoidal and RoPE encodings are defined for any position, degrade gracefully rather than crashing, offer some extrapolation, and in RoPE's case can be scaled further with the right tricks.

The takeaway is that the encoding you pick is also a decision about your maximum usable context. If long context matters, a learned absolute scheme is a dead end without retraining, while formula-based schemes at least give you a fighting chance. This connects directly back to the length-extrapolation problem raised in the why-it-matters post — it is the same issue, now seen from the implementation side.

Double-counting position is a subtler structural mistake. Positional information should be injected once, at the input, because the network propagates it upward through every layer. Some implementations mistakenly re-add the positional encoding inside each block, or stack a learned absolute scheme on top of a model that already uses RoPE, which doubles or confuses the positional signal and can hurt performance.

The rule is to inject position at the right place exactly once. RoPE is the deliberate exception: it operates inside attention by rotating queries and keys, so a RoPE model should not also add a positional vector at the input. The fix is to be clear about which scheme you are using and to confirm position enters the model through exactly one mechanism, not two stacked accidentally.

This decision tree turns the inject-once question into a quick check. If you are using RoPE, position is applied inside attention by rotating the query and key vectors, and you should not add a positional vector at the input — doing both double-counts. If you are not using RoPE, the right pattern is to add the positional encoding exactly once at the input; if you have not, you either forgot it or added it twice, and both are bugs.

The broader lesson is to treat the point of injection as a deliberate design decision rather than a default. Because both omission and duplication pass the shape checks silently, the only reliable defense is to consciously verify, for each model, that position enters through one and only one mechanism appropriate to the scheme you chose.

Expecting free length extrapolation is a wishful assumption rather than a coding error, but it bites just as hard. Training a model up to a couple thousand tokens and then running it at sixteen thousand, expecting coherent output, ignores that most positional schemes degrade past their training length. Even RoPE, the most extrapolation-friendly common choice, generally needs explicit help — position interpolation or NTK-aware scaling — to stretch far beyond what it saw in training.

The responsible stance is to measure behavior at the length you will actually deploy, not just at training length, and to extend context deliberately using known techniques rather than hoping it generalizes for free. This is the same length-extrapolation problem from earlier in the day, and treating it as solved by default is one of the easiest ways to ship a long-context model that quietly falls apart in production.

This comparison sorts positional failures into loud and silent and makes the counterintuitive point that the silent ones are far more dangerous. Loud bugs — indexing past a learned max_len, a dimension mismatch between the encoding and the embedding, running RoPE on an odd dimension — announce themselves with an exception, so you fix them and move on.

The silent bugs are the costly ones: forgetting the addition turns the model into a bag of words with no error, an off-by-one slice smears the positional signal while training proceeds, and bad extrapolation rots long-context output that looked fine at training length. None of these throw; they just hand you a model that appears to work but is subtly broken. Learning to suspect these silent failures when results are merely mediocre — or degrade only at longer lengths — is exactly what separates experienced practitioners from beginners.

The final checklist consolidates the fixes into a pre-flight you can run before trusting any positional code: actually add the encoding to the embeddings; slice the table from zero to the real sequence length; respect a learned scheme's max_len on long inputs; inject position once, in the place appropriate to your scheme; never assume free length extrapolation; and always measure at the length you will actually deploy.

Run through this list and you avoid the failures that cost most people their first weeks with positional encodings. With the intuition, the math, a working build, and now the traps all covered, you have the full picture of how Transformers represent order.

The teaser closes the day and opens the next topic. Having mastered positional encoding in isolation — what it is, why it matters, how it computes, how to build it, and how it fails — the natural next step is to see how attention, positions, residual connections, normalization, and feed-forward layers snap together into the full Transformer block, the unit the entire modern architecture is built from. The series continues there.