Language Models 101

This cover sets the foundation for the entire 'Language Models 101' day by cutting through the hype around the term. People hear 'language model' and imagine something that thinks, knows, or understands. The honest, useful definition is far narrower: it is a function that takes some text and assigns a probability to what comes next.

The goal of this first post is purely conceptual. Before you learn why next-token prediction became the foundation of AI (post 2) or how a transformer actually computes it (post 3), you need a clean mental model of what a language model is and where it came from. Everything in the following posts hangs off this single idea.

A language model, formally, assigns a probability to a sequence of tokens. The usual way it does this is by factoring that joint probability into a product of next-token predictions: the probability of each token given all the tokens before it. So the central operation is always 'given this context, how likely is each possible next token?'

The word 'learns' is doing real work here. The model is not given grammar rules or a dictionary. It is shown enormous amounts of text and adjusts its internal parameters until its predicted distributions match what actually appears in the data. Grammar, facts, and even style emerge as side effects of getting good at that prediction.

This framing — probability distribution over text — is the through-line of the whole field. n-gram models, GPT, and BERT-style models are all different machinery for estimating the same kind of distribution.

Next-token prediction is the concrete task that makes the abstract definition usable. Show the model a context like 'The cat sat on the' and it produces a score for every token in its vocabulary: 'mat' might get high probability, 'floor' a bit less, 'banana' almost none. These scores form a probability distribution over the entire vocabulary.

Generation is then just a loop. Pick a token from that distribution, append it to the context, and ask the model to predict again. Do this repeatedly and coherent text streams out, one token at a time. This autoregressive loop is the engine behind every chatbot and code assistant you have used, no matter how sophisticated the wrapper around it looks.

This flow diagram traces a single prediction step so the abstract 'distribution over tokens' becomes concrete. The context string enters, the model scores every possible next token, the result is a probability distribution, and one token is selected and appended before the cycle repeats.

The key insight to carry away is that the model does not plan a whole sentence at once. It commits to exactly one token, then re-evaluates the entire context including its own new token. Fluent paragraphs are built up from thousands of these tiny, local decisions, which is both the source of the model's flexibility and the reason it can drift off track.

Long before neural networks, language models were built by counting. An n-gram model estimates the probability of the next word using only the previous n-1 words, read straight off frequency counts in a corpus. A bigram model conditions on one previous word; a trigram on two. The whole 'model' is a giant table of counts.

This matters because n-grams are the direct conceptual ancestor of modern LLMs. They share the exact same objective — predict the next token given context — and differ only in how they estimate the distribution. n-grams use raw counts and a fixed, tiny context window; transformers use learned weights and a huge context. Seeing the lineage demystifies what today's models are fundamentally doing.

This trace walks a tiny corpus through bigram estimation so the counting idea is undeniable. The corpus contains 'I like tea' and 'I like coffee'. To estimate what follows 'I like', you count the continuations: 'tea' appears once and 'coffee' once. Normalizing gives each a probability of 0.5.

Following it line by line shows there is no magic — the model is literally a frequency table. The obvious weakness is also visible: with only a one or two word window, the model has no idea about anything earlier in the sentence, and it has never seen most word combinations. Those limitations are exactly what neural language models, with their long context and learned generalization, were built to overcome.

This snippet builds a working bigram model in eight lines so the concept stops being theoretical. It walks through the text in overlapping pairs, and for each word records a counter of which words followed it. Printing the counter for 'like' shows 'tea' and 'coffee' each once; for 'i' it shows 'like' twice and 'love' once.

Running this yourself is the fastest way to feel what a language model is at its core: a structure that, given a word, tells you what tends to come next. To turn counts into probabilities you would simply normalize each counter so its values sum to one. Everything fancier in the field is a more powerful way of estimating these same conditional distributions.

Why is a probability distribution over text so valuable? Because an enormous range of tasks can be reframed as 'which continuation is most probable?'. Generation is sampling from the distribution. Autocomplete is showing the top continuation. Spelling correction is finding the most probable intended text. Ranking translations or transcriptions is scoring candidates by likelihood.

This is the deep reason one capability scales into so many products. You do not need a separate algorithm for each application; you need one good estimator of P(next token | context) and a way to ask it the right question. That unification is what makes language modeling such a powerful and general tool, and it sets up post 2's argument about why the simple objective became so dominant.

It is worth nailing down what a language model is not, because each misconception leads to real mistakes. It is not a database of facts: it stores statistical patterns, not verified records, so it can be fluent and wrong at the same time. It is not a reasoning engine with beliefs or intentions; it has no model of truth, only of likelihood.

The practical consequence is that the model optimizes for plausible-sounding text, not correct text. When those two things line up — common, well-attested facts — it looks brilliant. When they diverge — rare facts, novel reasoning, arithmetic — it can produce confident nonsense. Holding 'models language, not the world' in your head is the single most useful corrective, and post 5 turns it into a full field guide.

This recap compresses the concept into five recallable lines. A language model is a probability distribution over text. Its core skill is predicting the next token given context. You generate text by sampling from that prediction, token by token. n-gram counting models were the original version of this idea. And crucially, the model captures likelihood, not truth.

If you remember only these five points, you have the scaffold to absorb why the objective is so powerful, how a transformer computes it, and the failure modes — the subjects of the next four posts.

This post established the concept; the next moves to consequences. It is easy to dismiss 'predict the next word' as a toy task, which makes the question in post 2 sharp: why did this simple objective become the foundation of modern AI rather than a footnote?

The answer is that predicting the next token well secretly requires learning grammar, facts, and reasoning, and that scaling this self-supervised objective produced capabilities nobody explicitly programmed. Post 2 makes that case concrete with self-supervision, scaling laws, and emergent abilities.