Temperature, Top-p, Top-k

This post sets the foundation for everything that follows, so the goal is a clean mental model rather than depth. The single most important idea is that a language model does not 'choose' a word in any human sense. At every step it produces a number for every token it knows, and a separate, swappable piece of machinery turns those numbers into one actual token. Temperature, top-p, and top-k are knobs on that machinery, not on the model itself.

Keeping the model and the sampler mentally separate is what unlocks the rest of the topic. The weights are frozen; the dials only change how you read the model's opinion.

When people first hear 'the model predicts the next word,' they picture a single answer popping out. What actually happens is that the final layer emits a vector of logits — one raw, unbounded score per vocabulary token, often 50,000 or more of them. Softmax exponentiates and normalizes those scores into a probability distribution that sums to 1.

Sampling is the act of drawing one token from that distribution. Greedy decoding always takes the most probable token; everything else introduces controlled randomness. The three dials in this post all operate on this distribution, which is why understanding 'logits then softmax' first makes the rest trivial.

Temperature is the dial that reshapes the entire probability curve. Mechanically it divides the logits by a number T before softmax runs. Dividing by a value below 1 stretches the gaps between logits, so after softmax the top token gets even more mass — the distribution becomes 'peaky' and the model behaves more deterministically. Dividing by a value above 1 squeezes the gaps, spreading mass toward unlikely tokens and making output more varied and surprising.

The crucial nuance is that temperature never removes any token from consideration; it only changes how likely each one is. Even at low temperature a rare token can, in principle, be sampled — it's just very improbable.

Top-k is the simplest of the truncation dials. After the probabilities exist, you sort tokens from most to least likely, keep the top k of them, throw the rest away, and renormalize the survivors before sampling. The number k is a fixed count that does not care about context: top-k=50 keeps fifty candidates whether the model is supremely confident or genuinely torn.

The edge case worth remembering is k=1, which is identical to greedy decoding — only the single best token survives, so there's nothing to randomize. Larger k values widen the pool of acceptable tokens and let more variety through.

Top-p, also called nucleus sampling, replaces the fixed count with a fixed probability budget. You sort tokens by probability, then keep adding them to your candidate set until their cumulative probability reaches p — say 0.9 — and discard everything beyond that. The size of the surviving set is not fixed; it expands and contracts with the model's confidence.

This adaptivity is the whole point. When the model is sure, two or three tokens already cover 90% of the mass, so the choice stays tight. When the model is uncertain, dozens of tokens may be needed to reach p, so the pool grows and variety increases naturally. That responsiveness is exactly what top-k lacks.

This diagram fixes the order of operations in your head, which is where most confusion lives. The flow is always the same: the model produces logits, temperature reshapes their sharpness, then top-k and/or top-p trim the candidate set, and finally one token is sampled from whatever survives.

Seeing it as a pipeline makes clear that these dials compose rather than compete. Temperature acts first on the raw scores; the truncation filters act afterward on the resulting probabilities. You can use any subset of them, and the next posts will show exactly how they interact when stacked.

The side-by-side comparison isolates the single difference people most often blur: top-k commits to a count, top-p commits to a mass. Both are ways of cutting off the long tail of unlikely tokens, but they answer different questions. Top-k asks 'how many candidates?' and top-p asks 'how much probability?'

The practical consequence is that top-p adapts to the situation while top-k does not. On a confident step, top-k=50 may keep 49 essentially worthless tokens in the pool, whereas top-p=0.9 keeps only the handful that genuinely matter. That adaptivity is why top-p is the more common default for general text.

This slide is the cheat sheet: three dials, three distinct effects. Temperature governs overall randomness by reshaping the curve. Top-k caps the maximum number of candidates by raw count. Top-p caps candidates by cumulative probability. They are orthogonal controls that can be combined.

The second half of the slide guards against the most damaging misconception in the whole topic: none of these dials add knowledge or ability. They can only narrow or reweight choices the model already has. A higher temperature can't make a model 'more creative' in the sense of knowing more — it just lets it pick lower-ranked tokens it already considered.

The code shows that in practice you usually set these as plain numeric arguments on a single API call, which can make them feel trivial — and that triviality is exactly why they get set carelessly. One line controls whether your output is rock-steady or wildly variable.

The inline comment about top_k is an important honesty note: not every hosted API exposes all three dials. OpenAI's chat API, for instance, exposes temperature and top_p but not top_k, while many local runtimes (llama.cpp, vLLM, Hugging Face) expose all three. Knowing which knobs your platform actually offers prevents you from tuning a parameter that's silently ignored.

This slide kills four misconceptions at once because they recur constantly. First, these are not a 'creativity slider' that injects new ideas — they reweight existing options. Second, they are not three names for one thing; each does something mechanically distinct. Third, nothing here touches the weights, so it is not training or fine-tuning. Fourth, temperature and top-p are not interchangeable; they operate at different stages and compose.

Naming the wrong assumptions explicitly is more useful than just stating the right ones, because learners arrive carrying these exact beliefs and need them dislodged before the correct model can stick.

The summary compresses the post into four lines you could recite from memory. The model produces probabilities and the sampler turns them into a single token. Temperature reshapes the probability curve's sharpness. Top-k keeps the best k tokens by count. Top-p keeps just enough tokens to cover probability mass p, with the count adapting to confidence.

If those four sentences are solid, the rest of this series — the math, the code, and the mistakes — slots in cleanly on top of them. This is the scaffold; the depth comes next.

The closing slide points forward to the stakes. Now that the three dials are clear as concepts, the natural next question is why they matter enough to obsess over. The next post argues that these settings sit directly between a capable model and a reliable product feature.

That framing matters because it reframes sampling from an academic curiosity into a product decision. The same model can power a flawless extractor or a flaky one depending entirely on numbers you might otherwise leave at their defaults.