Temperature, Top-p, Top-k

This is the mechanism post, and its purpose is to replace hand-wavy intuition with the exact operations. Temperature, top-p, and top-k are not aesthetic choices; each is a precise transformation applied to a vector of numbers in a specific order. Once you can see those operations, the behavior of every setting becomes predictable rather than mysterious.

The deliberate structure here is to start from logits and softmax, then layer each dial on top in the order it actually executes. By the end you should be able to trace a single token's journey from raw score to final selection.

Everything starts with logits. The model's final linear layer outputs one logit per vocabulary token — a raw, real-valued score that can be negative or large, with no inherent normalization. Softmax converts these into a valid probability distribution using p_i = exp(z_i) / Σ exp(z_j), which guarantees the values are positive and sum to one.

The key property to internalize is that softmax is sensitive to the gaps between logits, not their absolute values. Adding a constant to every logit changes nothing, but stretching or shrinking the gaps changes everything — which is precisely the lever temperature pulls.

Temperature applies the transformation p_i = softmax(z_i / T). Dividing by T before softmax rescales the gaps between logits. When T is below 1, the gaps grow, so after exponentiation the leading token dominates even more — the distribution sharpens and behavior tends toward deterministic. When T is above 1, the gaps shrink, mass leaks into the tail, and the distribution flattens toward randomness.

At exactly T=1 the division is a no-op and you sample from the model's native distribution. This is why T=1 is the conceptual 'neutral' setting, and values below or above it bias toward sharper or flatter respectively.

The bar chart isolates the effect of temperature on a single quantity: the probability assigned to the top token, given the same underlying logits. At T=0.5 the top token's share climbs steeply because the gaps were stretched. At T=1.0 it sits at the model's native value. At T=1.5 it drops as mass redistributes toward less likely tokens.

The takeaway is that temperature does not change the ranking of tokens — the most likely token stays most likely — it only changes how dominant that leader is. Sharpening makes the leader nearly inevitable; flattening gives the rest a fighting chance.

The limiting behavior of temperature clarifies the whole range. As T approaches 0, the softmax collapses entirely onto the single highest logit: every other probability goes to zero and you have pure greedy decoding — fully deterministic. As T grows toward infinity, all gaps vanish and the distribution approaches uniform, where every token is equally likely and the output is essentially noise.

In practice usable temperatures sit roughly between 0 and 2. Beyond about 1.5–2 the tail tokens get enough mass that grammar and coherence break down quickly, so the theoretical 'infinite' end is never actually useful.

Top-k is a sort-and-clip operation. After you have probabilities, you sort tokens from most to least likely, keep the top k, set every other probability to zero, renormalize the survivors so they sum to one again, and sample. The defining trait is that k is a hard, context-blind count.

This is both its strength and its weakness. It guarantees you never sample outside the k best tokens, which is predictable, but it's indifferent to the model's confidence. If the 50th and 51st tokens are nearly tied in probability, top-k=50 still cuts the 51st arbitrarily, and on a very confident step it keeps dozens of near-worthless tokens in the pool.

Top-p replaces the fixed count with a cumulative-mass cutoff, the 'nucleus.' You sort tokens high to low, walk down the list accumulating their probabilities, and stop the moment the running sum reaches p. The set of tokens collected so far is the nucleus; everything after the cutoff is discarded, the nucleus is renormalized, and you sample from it.

Because the stopping point is defined by mass rather than count, the nucleus size is adaptive. A confident step where one token holds 0.95 of the mass yields a tiny nucleus; an uncertain step with probability spread thin yields a large one. This responsiveness to confidence is exactly what makes top-p a popular default.

The flow diagram lays out the five concrete steps of nucleus sampling: sort by probability descending, take a cumulative sum, cut at the threshold p, renormalize the survivors so they sum to one, and sample a single token. Seeing it as discrete steps demystifies what 'nucleus sampling' actually computes.

The step that trips people up is renormalization. After truncation the surviving probabilities no longer sum to one, so you must rescale them before sampling — otherwise the draw is invalid. Both top-k and top-p share this renormalize-then-sample tail; only the truncation rule differs.

This comparison pins down where in the pipeline each dial intervenes, which is the source of most confusion. Temperature acts on the logits, before softmax, and reshapes the entire curve while leaving every token theoretically reachable — it never deletes anything. Top-k and top-p act after probabilities exist and physically remove tokens from the candidate set by truncation.

Understanding this 'before vs after' split explains why the dials compose cleanly: temperature decides how sharp the curve is, then the truncation filters decide how much of that curve you're even allowed to draw from. They are complementary, not redundant.

The from-scratch implementation makes the abstract operations concrete and verifiable. Temperature divides the logits first. The argsort gives the ranking; if top_k is set, the candidate list is clipped to the first k indices. Softmax (computed in a numerically stable way by subtracting the max) turns survivors into probabilities. If top_p is below 1, a cumulative sum plus searchsorted finds exactly how many tokens are needed to reach the mass, the set is clipped there, and probabilities are renormalized before np.random.choice draws one.

Reading this twenty-line function end to end is the fastest way to be certain you understand the dials — there is no hidden magic, just sort, scale, clip, normalize, and draw.

This slide nails down the order, which matters because the operations are not commutative in effect. Temperature scales the logits first, setting the overall sharpness. Top-k then clips to a fixed count. Top-p clips to a probability mass — and when both are set, top-k typically runs before top-p, so top-p operates on the already-clipped set. Finally you renormalize and sample exactly one token.

Getting the order wrong, or assuming the filters happen before temperature, leads to incorrect mental predictions about output. The fixed sequence — scale, count-clip, mass-clip, normalize, sample — is the backbone of every sampler you'll meet.

The closing slide hands off to the hands-on post. Now that the math is explicit, the natural next step is to watch it run: turn each dial on a real model and observe the same prompt's output shift. Theory becomes belief once you see temperature visibly loosen the text and top-p visibly tighten it.

The sequencing is intentional — math first so the code isn't magic, code next so the math isn't abstract. Together they give you both the why and the proof.