Temperature, Top-p, Top-k

This post deliberately steps back from mechanism to make the case for why anyone should care. It's easy to treat temperature, top-p, and top-k as obscure tuning knobs for researchers, but in production they are among the highest-leverage settings you control. The same model behind your feature can feel reliable or chaotic depending on three numbers.

The through-line of the post is that picking these values is a product decision tied to the specific job, not a global default you inherit once and forget. Each task has a right answer, and the default rarely is it.

The core tradeoff is determinism versus diversity. On one end you want the same correct answer every single time — extraction, classification, code. On the other end you want variety and surprise — brainstorming, copywriting, fiction. Sampling parameters are how you place a task on that spectrum.

The danger of defaults is that they silently pick a spot for you, and it's usually a generic middle that suits neither extreme. A hosted API defaulting to temperature around 0.7–1.0 is fine for chat but quietly wrong for a parser, and the failure only shows up under real traffic.

This comparison grounds the abstract tradeoff in concrete jobs. On the determinism side sit tasks where output is consumed by code or graded for correctness: JSON payloads, function-call arguments, SQL statements, classification labels, arithmetic, and field extraction. Any deviation here is a bug, not a flourish.

On the diversity side sit tasks where sameness is the failure: brainstorming, marketing variants, story and dialogue, idea generation. Here a deterministic model that returns the identical 'safe' answer every run is useless. Sorting your task into the correct column is the first practical step in choosing settings.

Structured output is the most common place teams get burned, so it deserves its own slide. At high temperature the sampler occasionally reaches into the tail and grabs a low-probability token. In free-form prose that's harmless variety; in a JSON object it might be a stray comma, an unquoted key, an invented field, or a chatty comment that derails the parser.

Low temperature concentrates probability mass on the single well-formed continuation the model already prefers. That's why temperature=0 is the right default for anything a downstream parser will read — you want the model's most confident, most syntactically valid path, every time.

The flip side is that creative tasks need room to breathe. If you set temperature near zero for a brainstorm, the model returns its single most probable answer on every run — safe, obvious, and identical. Diversity requires deliberately letting plausible-but-not-top tokens through, which means raising temperature.

The stable recipe for creative work is moderate-to-high temperature paired with a top-p around 0.9, which keeps variety high while still cutting off the genuinely incoherent tail. The point is that 'good settings' are entirely relative to the task; there is no universally correct value.

The bar chart makes the tradeoff tangible by showing how output variability rises with temperature. Near zero, repeated runs are nearly identical — high reliability, low variety. Around 0.7, outputs are noticeably varied while staying coherent, which is why that region is a popular default for chat.

Push toward 1.5 and variability spikes, but so does the risk of incoherence and broken structure. The chart is a reminder that you're buying variety with reliability and vice versa; there is no free lunch, only a position on the curve you choose on purpose.

Reproducibility is the cost most teams forget until it bites. Any temperature above zero makes generation non-deterministic: the same prompt can produce different outputs on different runs. That turns an eval suite flaky, makes a reported bug impossible to reproduce, and sends engineers chasing 'intermittent' failures that are really just sampling noise.

The fix for anything you test or benchmark is to pin temperature to 0 and set a fixed seed where the platform supports it. Reserve randomness for production paths where variety is actually wanted, and keep your evaluation deterministic so you can trust your signal.

Cost and latency hide inside these settings in a way that's easy to miss. High randomness on a structured task raises the probability of malformed output, and malformed output usually triggers a retry — sometimes an automatic one in your wrapper code. Every retry is another billed API call and another network round-trip.

So tightening sampling on structured tasks isn't merely about correctness; it directly reduces token spend and tail latency by eliminating do-overs. Teams that switch their extraction endpoints to temperature=0 frequently see both error rates and costs drop at the same time.

The code demonstrates the practical takeaway: maintain a small map of named profiles, one per task type, instead of scattering magic numbers through the codebase. An 'extract' profile pins temperature to 0 for deterministic parsing; a 'chat' profile uses balanced 0.7/0.9; an 'ideate' profile opens up to 1.1/0.95 for variety.

Centralizing settings this way makes the per-task philosophy enforceable and reviewable. When someone adds a new endpoint, they pick a profile deliberately rather than copying whatever temperature happened to be nearby, which is exactly the discipline this post argues for.

These rules distill the stakes into action. Default to temperature 0 for anything parsed by code, because correctness beats variety there every time. Raise temperature only where variety is genuinely the product goal. Pin temperature 0 plus a seed in your eval and test suites so your signal is trustworthy.

The final rule — treat these as per-task config, not globals — is the post's thesis in one line. A single global temperature can't be right for both your invoice parser and your tagline generator, and pretending it can is the root of most sampling pain.

The bottom-line slide reframes the whole topic: these dials sit in the gap between a capable model and a reliable product. You can have a genuinely excellent model and still ship a flaky feature purely because the sampling is wrong for the job. The model's quality and the feature's reliability are separable, and sampling is a big part of what separates them.

The encouraging corollary is that a large fraction of 'the AI is unpredictable' complaints evaporate once these values are chosen deliberately per task. It's one of the cheapest, highest-leverage reliability fixes available.

The closing slide points to the mechanism. Having argued why the dials matter, the next post opens the hood to show the actual math — how dividing logits by T reshapes the curve, and how top-k and top-p truncate the candidate set with precise operations.

Understanding the stakes first makes the math land harder: once you know that a wrong temperature can break a parser or flake an eval, you're motivated to understand exactly what the number does to the distribution. Motivation before mechanism is the deliberate teaching order here.