Language Models 101

This cover reframes the central surprise of the field: an objective that sounds almost trivial — guess the next word — turned out to be the engine of the entire LLM revolution. The post's job is to explain why, so the boom stops feeling like magic and starts feeling like a consequence you could have predicted.

The argument has four pillars: next-token prediction implicitly demands real knowledge, the objective is self-supervised so data is effectively unlimited, scaling that setup produces predictable gains, and beyond certain scales new abilities emerge. By the end you should be able to look at the LLM boom and see the mechanism behind it.

The crux of why next-token prediction is powerful is that the task is shallow but the knowledge to perform it is deep. To correctly complete 'The capital of France is ___' the model must encode a geographic fact. To complete a half-written function it must encode programming semantics. To finish a logical argument it must track the argument's structure.

The objective never says 'learn geography' or 'learn to code'. It only ever says 'predict the next token'. But because the text the model trains on encodes all of human knowledge expressed in language, getting good at prediction forces the model to internalize that knowledge as a side effect. The simplicity of the objective and the richness of what it requires are the whole story.

Self-supervision is what makes the whole approach scale, and it is worth understanding precisely. In supervised learning you need humans to label data — expensive and slow. In self-supervised learning the label comes from the data itself: hide the next token, ask the model to predict it, and the hidden token is the correct answer, for free.

This is why models can train on trillions of tokens. Every sentence ever written online is simultaneously an input (the context) and a label (the next token). No annotation team could ever hand-label data at that scale. Self-supervision unlocked the gigantic, cheap training sets that, combined with scale, produced modern language models.

This cycle captures the self-supervised training loop in four repeating steps. Take a chunk of raw text. Mask or hide the next token the model is supposed to predict. Have the model predict it. Compare the prediction to the real token and update the weights to reduce the error.

The elegance is that the loop needs no human judgment at any step — the corpus supplies both the question and the answer. Run this billions of times over a massive corpus and the weights gradually come to encode the statistical structure of language. Keep this loop in mind as the mechanism that converts raw internet text into a capable model.

For a long time, scaling up language models gave only modest, diminishing returns, and many researchers assumed the approach would plateau. Around 2018 to 2020 the picture changed dramatically: when data, parameters, and compute were scaled together, capability improved smoothly and predictably rather than flattening.

These relationships were formalized as 'scaling laws' — performance improving as a power law in model size, dataset size, and compute. The practical impact was enormous: it meant that spending more, in the right proportions, reliably bought more capability. That predictability is what justified the massive investments that produced today's frontier models.

This bar chart visualizes the scaling story at a glance. Small models are weak and generic. Medium models become fluent. Large models start to learn from examples given in the prompt (few-shot learning). Very large models exhibit emergent skills that smaller ones essentially lack. The rising bars represent capability climbing with scale.

The lesson is not that scale is the only thing that matters — post 5 pushes back on 'bigger is always smarter' — but that, holding method fixed, scale was the dominant lever that moved the field from narrow curiosities to general-purpose systems. Understanding this curve is key to understanding why the industry poured resources into ever-larger models.

Emergent abilities are the most striking part of the scaling story. Certain capabilities — multi-step arithmetic, following instructions, learning a task from a few in-context examples — are nearly absent in small models and then appear relatively sharply once a model crosses some scale threshold. They were never trained for explicitly.

This emergence is what made the field feel like it was discovering rather than engineering. The same single objective, next-token prediction, scaled far enough, produced behaviors that look qualitatively new. There is healthy debate about how sharp these transitions really are versus artifacts of measurement, but the practical reality stands: large models can do things smaller ones simply cannot, without any change to the training objective.

This snippet illustrates the payoff of a general next-token predictor: one model handles many tasks with no retraining, just different prompts. Framing translation, sentiment classification, and information extraction as text-completion problems lets the same weights solve all three. The model is not 'configured' for each task; the prompt simply steers which continuation it produces.

This is the practical face of generality. In the old paradigm each of these would be a separate trained system. With a capable language model, the task lives in the prompt rather than in the weights, which is why prompting became a discipline of its own and why a single model can power thousands of distinct applications.

This comparison crystallizes the before-and-after of the LLM era. Before, NLP meant one narrow model per task, each needing its own labeled dataset, brittle outside its training distribution, and requiring retraining to adapt. After, a single general model trained on raw text handles many tasks, generalizes flexibly, and is adapted by prompting rather than retraining.

Reading the two columns together makes the magnitude of the shift obvious. It is not that models got a bit better at one thing; the entire workflow inverted from building bespoke systems to steering one general system. That inversion is the practical meaning of 'why it matters'.

This recap reduces the argument to five lines. Next-token prediction implicitly demands real knowledge, so mastering it produces capability. Self-supervision makes training data effectively unlimited. Scaling laws made capability gains predictable with investment. New abilities emerge past scale thresholds. And the end result is one general model replacing hundreds of narrow ones.

Those five points are the complete answer to why a humble objective became the foundation of modern AI. Hold them and the field's trajectory stops looking like a series of miracles and starts looking like a coherent consequence.

With the why established, the natural next question is mechanical: how does a model actually turn a context string into a next-token probability? Post 3 opens the hood on the forward pass — tokenize, embed, contextualize with attention, and predict — and shows how training tunes the weights and how generation loops.

Understanding the machinery is what lets you reason about cost, context limits, and failure modes concretely, and it sets up the hands-on code in post 4.