Language Models 101

This is the hands-on post, and its purpose is to collapse the gap between understanding how a language model works and running one yourself. Everything from post 3 — the forward pass, logits, softmax, sampling, the generation loop — appears here as a few lines of real Python using Hugging Face transformers and a small GPT-2 that runs on any laptop.

The blocks are ordered to run top to bottom, but each is self-contained enough to read on its own. By the end you will have inspected a real next-token distribution, compared decoding strategies, controlled creativity with temperature, generated full text, and even scored how 'expected' a sentence is.

This block installs the libraries and loads GPT-2, a small open model perfect for experimentation. AutoModelForCausalLM loads a decoder language model — exactly the next-token predictor described all day — and the matching tokenizer converts text to and from token IDs. Calling .eval() puts the model in inference mode, disabling training-only behaviors like dropout.

Printing the vocabulary size, 50257, makes one thing concrete: every prediction this model makes is a distribution over these 50257 possible tokens. That number is the width of the softmax output you will inspect in the next block. GPT-2 is old and small, but it runs the identical pipeline as frontier models, which makes it an honest teaching tool.

This block exposes the heart of a language model: the next-token distribution. Encoding 'The capital of France is' and reading logits[0, -1] gives the raw scores for the token that should come next — the [0, -1] selects the first (only) sequence and its last position. Softmax turns those scores into probabilities, and topk surfaces the five most likely tokens with their probabilities.

Seeing ' Paris' rank near the top is the satisfying payoff: the model genuinely placed high probability on the correct continuation, learned purely from text. Inspecting this distribution is the single best way to build intuition — try your own prompts and watch which tokens the model considers likely versus unlikely.

This block contrasts the two fundamental decoding strategies on the same distribution. Greedy decoding takes argmax — always the single most likely token — so it is deterministic: the same input always gives the same output. Sampling instead draws from the distribution with multinomial, so a token with probability 0.2 is chosen roughly twenty percent of the time, producing varied output across runs.

This is the core tradeoff you will tune constantly. Greedy is safe and repeatable but can be bland and prone to loops; sampling is diverse and creative but less predictable. Running both on the same logits and seeing the deterministic versus varied results makes the distinction tangible before you reach the temperature knob.

This block introduces temperature, the most important sampling knob. Dividing the logits by a temperature before softmax reshapes the distribution. A low temperature like 0.3 sharpens it, concentrating mass on the top tokens for safe, repetitive output. A high temperature like 1.5 flattens it, giving unlikely tokens more chance and producing wilder, more creative — and more error-prone — text.

Running the same logits at three temperatures and watching the outputs shift from conservative to adventurous makes the parameter intuitive. The practical rule that emerges: lower temperature for factual or structured tasks where you want reliability, higher temperature for brainstorming or creative writing where variety helps.

This block uses the high-level .generate() method, which wraps the entire autoregressive loop from post 3 into one call. It runs the forward pass, samples a token, appends it, and repeats up to max_new_tokens, here generating thirty tokens of continuation. The do_sample, temperature, and top_k arguments control the decoding behavior explored in the previous blocks.

top_k=50 restricts sampling to the fifty most likely tokens at each step, a common trick to keep output coherent while still allowing variety. Setting pad_token_id avoids a noisy warning. This is how you actually generate text in practice — the manual loops earlier were to build understanding; .generate() is what you reach for in real code.

This comparison summarizes the decoding knobs as a single mental model. The left column — low temperature or greedy — gives deterministic, safe, coherent output that can sometimes loop or repeat, and is well suited to factual answers. The right column — high temperature or sampling — gives random, creative, diverse output that can occasionally go off the rails, and suits brainstorming.

Having just run both extremes yourself, this table is no longer abstract; each row maps to behavior you observed in your own console. The takeaway is that decoding settings are not an afterthought — they shape the output as much as the prompt, and choosing them deliberately is part of using a language model well.

This block flips the model around: instead of generating, it scores how 'expected' a given sentence is. Passing the input as both ids and labels makes the model compute the average cross-entropy loss — how surprised it was by each actual token. Exponentiating that loss gives perplexity, a classic language-model metric where lower means the text was more predictable to the model.

This reveals that a language model is fundamentally an evaluator of likelihood, not only a generator. Perplexity is used to compare models, detect out-of-distribution text, and measure fit. Seeing a fluent English sentence get low perplexity, versus gibberish getting high perplexity, connects directly back to post 1's definition: the model assigns probability to text.

Even with working code, a handful of mistakes trip up almost every beginner. Forgetting torch.no_grad() at inference wastes memory building a gradient graph you never use. Reading logits[0] instead of logits[0, -1] grabs scores for the wrong position. Setting temperature to exactly zero divides by zero — use greedy decoding instead for determinism. Skipping pad_token_id produces noisy warnings. And expecting GPT-2 to know recent events ignores that its knowledge is frozen at training time.

Each of these either errors out cryptically or silently gives wrong results, which is what makes them worth memorizing. They map directly onto the conceptual points from earlier posts — position matters, temperature reshapes the distribution, and the model only knows what it was trained on.

This checklist turns the post into practical discipline. Always use .eval() and torch.no_grad() for inference to get correct, efficient behavior. Tune temperature and top_k to match your task rather than accepting defaults. Use greedy decoding for factual answers and sampling for creative ones. Always set max_new_tokens so generation terminates. And always pair a model with its own tokenizer so the token IDs line up.

These five habits catch the overwhelming majority of problems beginners hit when running language models, and they connect every conceptual lesson from the day to real, reliable code.

Post 4 gave you a working model to experiment with; post 5 gives you the mental corrections to use it wisely. Being able to generate text is not the same as understanding what the output is and is not — and the most common, costly errors come from misreading what a language model fundamentally does.

The final post is the field guide to the misconceptions: that the model knows facts, that hallucinations are bugs, that bigger is always smarter, that it remembers you, and that its output is fixed. Each is traced back to the one true fact — it is a next-token predictor — and turned into practical advice.