Temperature, Top-p, Top-k

This final post is a field guide to the ways sampling goes wrong, because in practice these three numbers fail quietly and the failures masquerade as model problems. A flaky JSON endpoint, an incoherent brainstorm, an eval that won't reproduce — the root cause is frequently a misused dial, not a broken model. Naming the failure modes is what lets you recognize and fix them fast.

The mistakes here are not exotic edge cases; they are the default ways teams misuse temperature, top-p, and top-k. Each slide pairs the trap with the concrete fix.

The most expensive mistake is high temperature on structured output. When a downstream parser expects strict JSON, SQL, or function-call arguments, any stray token is a hard failure — a missing quote, an extra comma, an invented field, or a chatty preamble. At temperature 1.0 the sampler reaches into the tail often enough that these errors show up under real traffic even if your demo looked fine.

The fix is unambiguous: temperature 0, or very near it, whenever code consumes the output. Reserve any creativity for human-facing prose. This single change resolves a large share of 'the model keeps returning invalid JSON' incidents.

The second trap is stacking temperature and top-p without understanding that they compound. A developer raises temperature for more variety, then also widens top-p to 'open it up more,' and the combined effect pushes output past coherent into garbled. Because both dials increase the reach into unlikely tokens, applying both at the extreme multiplies the randomness.

The discipline is to pick one primary dial and hold the other moderate. A reliable recipe is a moderate temperature with top-p around 0.9 — variety with a guardrail. Cranking both simultaneously is how 'creative' turns into 'incoherent.'

Third is the genuine conceptual mix-up between top-k and top-p. They feel interchangeable but answer different questions: top-k fixes the number of candidates, top-p fixes the probability mass. The consequence shows up on confident steps — top-k=50 still keeps 49 near-zero tokens in the running, while top-p=0.9 keeps only the two or three that actually matter.

Reaching for the wrong one produces subtly wrong behavior that's hard to diagnose because both 'kind of work.' The fix is to be explicit in your own head and in code comments about whether you mean 'a count of candidates' or 'a fraction of the mass.'

This comparison makes the top-k versus top-p distinction concrete on a single generation step. With top-k=50 you always carry exactly fifty tokens forward regardless of confidence, so when the model is sure you're dragging a long tail of worthless options along — the count is fixed and the mass varies. With top-p=0.9 you might keep as few as two tokens when the model is confident and many more when it's uncertain — the mass is fixed and the count varies.

The adaptivity of top-p is exactly why it tends to be the better default for general text: it tightens automatically when the model is sure and loosens when it isn't, which top-k cannot do.

The fourth trap is inheriting defaults from an unrelated context. A blog post's temperature=0.8 was almost certainly tuned for casual chat, not for your invoice-extraction job — yet that same number gets copy-pasted across wildly different tasks. The result is one global value powering both a poem generator and a data parser, serving neither well.

The fix is to derive settings from the task's actual requirements: does code read the output, and do you want variety or consistency? Those two questions determine the values. Settings should come from the job, not from a screenshot of someone else's working code.

The fifth trap is ignoring reproducibility, which turns evaluation into noise. Any temperature above zero makes generation non-deterministic, so a test that passed yesterday can fail today with no code change, and teams misread that sampling noise as a real regression. Engineering hours evaporate chasing phantom bugs.

The fix is to pin temperature to 0 and set a fixed seed for any test, benchmark, or eval. Determinism in evaluation is non-negotiable if you want your pass/fail signal to mean anything. Save the randomness for production paths where variety is the actual goal.

This slide catches a subtler misconception: that top_p=0 means 'maximally strict' or 'most deterministic.' In reality the behavior at that boundary is implementation-dependent — some libraries keep only the single top token (effectively greedy), others behave in undefined or surprising ways at the exact edge. Relying on extreme top-p as a determinism proxy is fragile.

The correct path to determinism is temperature=0 or explicit greedy decoding (do_sample=False), which are well-defined everywhere. Don't try to back into deterministic behavior through an edge case of a truncation filter; use the dial designed for it.

The decision tree compresses the whole post into one practical flowchart. First question: will code parse the output? If yes, the answer is temperature=0 with greedy decoding — correctness over everything. If no, the next question is whether you need lots of variety: if so, push toward T≈1.0 with top_p≈0.95; if not, settle into the balanced default of T≈0.7 with top_p≈0.9.

This tree is deliberately simple because most real decisions are this simple. The vast majority of tasks fall cleanly into one of these three buckets, and just asking 'does code read this?' first prevents the most damaging mistakes outright.

The code turns the decision tree into enforceable defaults. A small dictionary maps task modes to vetted parameter sets: 'json' pins temperature to 0 with a seed for deterministic, parseable output; 'chat' uses the balanced 0.7/0.9; 'ideas' opens up to 1.0/0.95 for variety. The wrapper function selects a profile by name so no one is hand-typing magic numbers at the call site.

The closing comment encodes the single most important rule: anything parsed by code always uses 'json' mode and never a high temperature. Centralizing settings like this makes the right choice the easy, default choice and the wrong choice a visible deviation in review.

The checklist is the post — and arguably the whole series — distilled into rules you can paste into a code review guideline. If output is parsed, temperature is 0, full stop. Don't push temperature and top-p to extremes together. Be explicit about whether you mean top-k (a count) or top-p (a mass). Pin temperature 0 and a seed in tests. Tune per task and never copy settings blindly.

These five lines won't make you a sampling researcher, but they will prevent the overwhelming majority of real-world sampling failures. That's the right tradeoff for a practitioner shipping features.

The closing slide bridges to the next day while wrapping the topic. You now understand sampling as a deliberate, tunable layer between a capable model and a dependable feature — concept, stakes, math, code, and the failure modes that bite teams in production.

The series continues building the NLP and LLMs toolkit, and how a model chooses its next word is foundational to everything downstream — from decoding strategies to how generation interacts with prompting and retrieval. Carrying these three dials forward as a mental model will make the next topics click faster.