Temperature, Top-p, Top-k

This is the hands-on post, and its job is to convert the previous post's math into things you can run and feel. Every snippet uses gpt2, a small open model that loads on a laptop, so there's no excuse not to actually execute them. We isolate one dial at a time, then reimplement the sampler by hand so the API arguments stop being opaque.

The deliberate teaching arc is baseline first, then one variable at a time. If you change temperature, top-k, and top-p all at once you can't attribute the effect to any of them — so we hold the others fixed and move one knob per experiment.

The first snippet establishes a deterministic baseline with greedy decoding. The critical detail is do_sample=False: when sampling is off, Hugging Face's generate ignores temperature, top_k, and top_p entirely and simply takes the highest-probability token at every step. Run it twice and you get byte-for-byte identical output.

This baseline matters for two reasons. It's the reference point against which every sampled variation is compared, and it's a common gotcha — beginners set temperature but forget do_sample=True, then wonder why nothing changes. Greedy is also the correct choice whenever you need full determinism.

Here we turn exactly one dial: temperature, with do_sample=True and the other filters disabled (top_k=0, top_p=1.0) so nothing else interferes. Looping over 0.3, 0.7, and 1.3 lets you watch the same prompt produce increasingly varied continuations. At 0.3 the text is tight and predictable; at 1.3 it wanders and takes risks.

The manual_seed call before the loop is what makes the comparison fair — it fixes the random draw so the only thing changing between runs is T itself. Slicing out[0][ids.shape[1]:] prints just the newly generated tokens, not the echoed prompt, which makes the differences easier to read.

Now we add the truncation filters on top of a fixed temperature. top_k=50 first restricts the pool to the fifty most likely tokens; top_p=0.92 then keeps only enough of those to cover 92% of the remaining mass. The inline comment captures the ordering that the previous post established: top_k runs before top_p, so top-p operates on the already-clipped set.

This is the realistic, production-style configuration — a moderate temperature paired with both filters acting as guardrails against the incoherent tail. Seeing all three set together, with a comment on their interaction, ties the whole topic into one concrete call.

This is the heart of the post: the sampler reimplemented in PyTorch so nothing is hidden. Logits are divided by T for temperature. For top-k, torch.topk finds the k-th largest value and everything below it is set to negative infinity, so softmax sends those tokens to zero. After softmax, top-p sorts the probabilities, uses a cumulative sum to mask tokens beyond the mass threshold, scatters the kept values back to their original positions, and multinomial draws one token from the renormalized result.

The mask expression cumsum(s) - s > top_p is the subtle part: subtracting the token's own probability ensures the token that crosses the threshold is itself kept, matching standard nucleus behavior. Tracing this function line by line is the surest way to truly own the concept.

The pipeline diagram mirrors the hand-written sampler step for step, so the code and the picture reinforce each other. Logits enter, get divided by T, the top-k mask sends rejects to negative infinity, softmax produces probabilities, the top-p mask zeros the tail, and multinomial draws a single token.

Mapping each code line to a visual stage closes the loop between the math post and this one. When you next pass these arguments to an API, you can picture exactly which stage each one controls and in what order they fire.

Reproducibility gets its own demonstration because it's where sampling surprises people in production. The run function seeds the RNG, then generates with sampling on. Calling run(42) twice yields identical text — same seed, same params, same output — while run(42) versus run(7) diverges. This proves that the randomness is controllable, not magical.

The practical lesson is that 'non-deterministic' doesn't mean 'uncontrollable.' For tests and reproducible bug reports you pin the seed and the parameters; for production variety you let the seed float. The same machinery serves both needs depending on whether you fix the seed.

The final inspection snippet pulls back the curtain on the distribution itself. We run a forward pass, grab the logits at the last position, apply temperature via softmax, and print the top five tokens with their probabilities. Suddenly the abstract 'distribution over the vocabulary' is a concrete, readable table.

This is also a debugging superpower. When output looks wrong, printing the top candidates and their probabilities tells you whether the model genuinely favored a bad token or whether your sampling settings let an unlikely one through. Seeing the numbers turns guesswork into diagnosis.

These gotchas are the ones you just witnessed in the snippets, collected so they stick. do_sample must be True or temperature, top_k, and top_p are silently ignored. Setting top_k=0 and top_p=1.0 disables those filters so you can isolate temperature. top_k is applied before top_p when both are present. And setting a seed is what makes any sampled run reproducible.

These four account for the overwhelming majority of 'why didn't my parameter do anything' confusion. Internalizing them saves hours of bewildered debugging the first time you wire sampling into real code.

The hosted-API snippet closes the gap between local experimentation and production. The same conceptual dials — temperature and top_p — map directly onto the OpenAI chat call, and a seed argument provides best-effort reproducibility. The 'best-effort' caveat is honest: hosted providers can't always guarantee determinism across infrastructure changes the way a local model with a fixed seed can.

Note what's absent: there's no top_k here, because OpenAI's chat API doesn't expose it. That reinforces the earlier point that you should know which knobs your specific platform actually offers before you try to tune them.

The closing slide hands off to the mistakes post. Having seen the dials work correctly, the natural next step is to study how they fail — stacking them carelessly, applying high temperature to structured output, and trusting inherited defaults. Seeing the correct behavior first makes the failure modes legible.

The progression across the five posts is intentional: concept, stakes, math, working code, and finally the field guide to misuse. By this point you can run the dials; next you learn how not to break them.