Language Models 101

This cover promises to replace the mystery of 'how do LLMs work' with a concrete pipeline. The reassuring truth is that there is no single magic step — there is a sequence of well-defined stages, each doing ordinary math. Tokenize the text, embed the tokens, run them through transformer layers that mix context, and produce a probability distribution over the next token.

The post walks the full forward pass, explains what attention computes in plain terms, shows how logits become probabilities, sketches how training adjusts weights, and closes with the generation loop. By the end the pipeline should feel mechanical rather than mystical.

This pipeline diagram is the spine of the whole post: the four stages every decoder language model runs to predict the next token. Tokenize turns text into integer IDs. Embed turns each ID into a learned vector. The transformer stack mixes those vectors so each token's representation reflects its context. The final stage maps the result to next-token probabilities.

Keep this four-stage flow in mind as the map for everything that follows. Each subsequent slide zooms into one stage. The power of the model comes not from any single stage being clever, but from stacking many transformer layers in the middle so the contextual representation becomes deep and rich.

The first two stages convert text into numbers the network can operate on. Tokenization splits the input into tokens and maps each to an integer ID using the model's vocabulary. The embedding layer then looks up a learned vector for each ID — a dense list of numbers that encodes something about that token's meaning and usage.

Because a plain bag of vectors has no notion of order, position information is added at this stage too, so the model can distinguish 'dog bites man' from 'man bites dog'. After embedding, your sentence is a grid of numbers, one row per token, and every later stage is matrix math on that grid. This is the bridge from human-readable text to machine-operable representation.

Stage three is where the real work happens: the transformer layers, built on attention, let every token gather information from every other token. If the context is 'The cat sat because it was tired', the representation for 'it' can attend to 'cat' and pull in the information needed to resolve the reference. Each token's vector is updated to reflect the relevant context around it.

Stacking many such layers — dozens in large models — builds an increasingly abstract, context-aware representation. Early layers might capture local syntax; later layers capture longer-range meaning and relationships. This repeated context-mixing is what separates a transformer from the fixed, tiny window of an n-gram model.

This flow diagram unpacks attention without the matrix algebra. Each token produces a query ('what information am I looking for?'), and every token also offers a key ('what do I provide?') and a value ('here is my content'). The query is scored against all the keys to decide how much to attend to each token, and the values are blended according to those scores.

The intuition to keep is that attention is a soft, learned lookup: each position dynamically decides which other positions are relevant and pulls a weighted mixture of their content into itself. Doing this in parallel across all positions, in every layer, is how the transformer efficiently routes information across the whole sequence.

Stage four converts the model's internal representation into an actual prediction. The final layer takes the vector for the last position and maps it to one raw score, called a logit, for every entry in the vocabulary. A vocabulary of fifty thousand tokens yields fifty thousand logits.

Those raw scores are not yet probabilities — they can be any real numbers. The softmax function exponentiates and normalizes them so they are all positive and sum to one, producing a valid probability distribution over the next token. That distribution is the model's answer to 'what comes next?', and it is what generation samples from.

This snippet makes the logits-to-probabilities step concrete and runnable. Three raw scores go in; softmax exponentiates each and divides by the total so the outputs are positive and sum to exactly one. The largest logit becomes the highest probability, but smaller logits still get nonzero mass.

Seeing the numbers — [2.0, 1.0, 0.1] becoming roughly [0.66, 0.24, 0.10] — demystifies the final stage entirely. This is the exact operation a real model performs, just over tens of thousands of vocabulary entries instead of three. Verifying that the probabilities sum to one confirms you are looking at a genuine distribution, which is what makes sampling well-defined.

Training is where the billions of weights get their values. At every position in the training text, the model predicts a distribution over the next token, and a loss function — cross-entropy — measures how much probability it placed on the actual correct token. Low probability on the right token means high loss.

Backpropagation then computes how each weight contributed to that error and nudges every weight slightly to make the correct token more likely next time. Repeat this across trillions of tokens and the weights gradually come to encode the statistical structure of language: grammar, facts, and style all fall out of relentlessly minimizing next-token prediction error.

This cycle diagram captures autoregressive generation, the loop that turns a one-step predictor into a text generator. Run the forward pass to get next-token probabilities, sample a token from them, append that token to the context, and repeat — continuing until a stop token is produced or a length limit is hit.

The important subtlety is that each new token becomes part of the context for the next prediction, so the model conditions on its own output as it goes. This is why generation is sequential and why long generations can drift: a single off token gets fed back in and influences everything after it. The loop is simple, but it is the source of both the model's fluency and its tendency to wander.

This snippet ties the whole pipeline together with a real model. Loading GPT-2 and its tokenizer, encoding a prompt, and reading the last position's logits gives you the genuine next-token scores. Taking the argmax — the highest-scoring token — is greedy decoding, the simplest way to pick the next token, and decoding it shows the model's single most likely continuation.

Running this is the bridge from theory to practice: every stage from the diagrams is present in these few lines. The model call performs tokenize, embed, transformer, and the logit projection; you then turn logits into a choice. Post 4 expands this into full generation with sampling and temperature.

This recap distills the mechanism into five recallable lines. The forward pass is tokenize, then embed, then transformer layers, then predict. Attention is what lets tokens share context with each other. Softmax converts the final raw logits into a probability distribution. Training means predicting, measuring the loss against the true token, and updating the weights. And generation is just looping the forward pass, feeding each new token back in.

Those five lines are the complete operating model of how a language model works. Everything more advanced — bigger context windows, better attention variants, alignment tuning — is refinement layered on this skeleton.

Post 3 explained the machinery; post 4 makes you run it. Reading about the forward pass is one thing; watching a real model assign 'Paris' high probability after 'The capital of France is', then changing temperature and seeing the output get wilder, is what makes it stick.

The next post loads GPT-2 in a few lines, inspects the next-token distribution, compares greedy decoding with sampling, and generates full text — turning every concept from this post into code you can poke at.