BPE & WordPiece

This cover frames the central puzzle of modern language models: they operate on a fixed, finite vocabulary of integer IDs, yet they must handle an effectively infinite space of words, names, typos, code, and emoji. The resolution is that the model never stores whole words at all. It stores fragments, and rebuilds anything from them.

The goal of this first post is purely conceptual. Before you learn why tokenization controls your bill (post 2) or how the merge algorithm runs (post 3), you need a clean mental model of what subword tokenization is and where it sits in the stack. Everything else hangs off that picture.

A subword tokenizer is a small, trained component that runs before the neural network ever sees your text. Its job is narrow: turn a string into a list of integer IDs, and turn IDs back into a string. To do that it carries a learned vocabulary — a table mapping each known chunk of text to an ID.

The word 'learned' is important. The vocabulary is not hand-written by linguists; it is derived statistically from a large corpus during a one-time training step. BPE and WordPiece are two recipes for building that vocabulary. Once built, the tokenizer is frozen and shipped alongside the model.

The payoff of the subword approach is that rare or unseen words never become a dead-end. A word the tokenizer has never encountered is simply broken into smaller pieces it does know, all the way down to single characters or bytes if necessary.

To see why subwords win, look at the two extremes. A word-level vocabulary assigns one ID per whole word. That sounds natural but it scales terribly: you need separate entries for 'run', 'runs', 'running', 'runner', every proper noun, every typo, and every new coinage. The vocabulary balloons into the millions and you still meet words at inference time that simply are not in it.

The opposite extreme is character-level: a tiny vocabulary of letters and symbols, with no unknown-word problem at all. But now 'tokenization' is twelve tokens instead of one, sequences become very long, and each token carries almost no meaning, which makes the model's job harder and slower.

Subword tokenization is the engineered compromise between these poles, and that tradeoff is the whole reason the technique exists.

This comparison lays the two extremes side by side so the middle ground is obvious. On the left, word-level tokenization gives short sequences but a gigantic vocabulary riddled with unknowns and no notion that 'running' and 'runner' share a root. On the right, character-level gives a tiny vocabulary and zero unknowns but punishingly long sequences and weak per-token meaning.

Neither column is acceptable for a production language model. Reading the two lists together is the fastest way to internalize why an intermediate granularity — pieces larger than characters but smaller than words — is the design every modern tokenizer converges on.

Subword tokenization picks a target vocabulary size, typically somewhere between 30,000 and 50,000 entries, and fills it with the most useful chunks learned from the corpus. Frequent whole words earn their own slot, so 'the', 'is', and 'token' stay intact and cheap. Less frequent words are represented as a sequence of smaller pieces.

The result hits all three goals at once: the vocabulary is small enough to be practical, sequences stay reasonably short because common words are single tokens, and there are no out-of-vocabulary failures because anything unknown decomposes into known parts. This balance is exactly why the technique underpins GPT, BERT, Llama, and essentially every current model.

This flow traces a single rare word through the tokenizer to make the abstract idea concrete. 'tokenization' is not common enough to have its own slot, so the tokenizer splits it using its learned vocabulary into 'token' and 'ization', two pieces it does know. Those pieces are then mapped to their integer IDs, which is what actually flows into the model.

The key insight to carry away is that the model never receives the letters t-o-k-e-n-i-z-a-t-i-o-n. It receives a short list of integers. The string is a human convenience; the integers are the reality the network operates on.

BPE and WordPiece are siblings: both build a subword vocabulary by starting small and greedily merging pieces, and both are used in production at scale. The differences are in the merge criterion and in surface conventions. BPE chooses which pieces to merge purely by frequency — glue together whatever adjacent pair appears most often. It powers GPT, Llama, and RoBERTa, and in its GPT byte-level form uses no special continuation marker.

WordPiece, used by BERT and its descendants, instead merges the pair that most increases the likelihood of the training data, a slightly smarter criterion that favors informative merges. It also marks word-internal pieces with a '##' prefix so the tokenizer can tell which pieces begin a word. Post 3 unpacks both criteria in detail; here the goal is just to see them as two flavors of the same idea.

This snippet makes the concept tangible in three lines. Loading BERT's tokenizer and running it on a normal sentence reveals the machinery directly: 'tokenization' becomes 'token' + '##ization', and 'unbelievable' shatters into 'un' + '##bel' + '##iev' + '##able'. The '##' prefix is WordPiece's marker for a piece that continues a word rather than starting one.

Running this yourself is the fastest way to make subword tokenization stop being abstract. Try your own name, a technical term, or a typo and watch how the tokenizer always finds a way to represent it from known pieces.

It is worth nailing down what these algorithms are not, because each misconception leads to a real bug. First, the tokenizer is not the model — it is a deterministic preprocessing step that runs before any neural computation. Second, the splits are not linguistic. 'token' + 'ization' happens to align with a morpheme boundary, but that is a coincidence of frequency statistics, not grammar; plenty of splits cut words in linguistically nonsensical places.

Third, and most practically, tokenizers are not interchangeable. A model learns embeddings tied to the exact ID-to-piece mapping it was trained with. Swap in a different tokenizer and every ID points to the wrong embedding, so the model reads noise while still producing fluent-looking output. This single fact is the root of the most common tokenization disaster, covered in post 5.

This recap compresses the whole concept into five lines you can recall on demand. Subword tokenization lives between the word and character extremes. It works by learning a vocabulary of frequent chunks from a corpus. Rare words are split into those chunks rather than discarded, so there are no unknown tokens. BPE chooses merges by raw frequency; WordPiece chooses them by data likelihood.

If you remember only these five points, you have the mental scaffold to absorb the cost implications, the merge algorithm, and the failure modes in the posts that follow.

This post established the concept; the next moves to consequences. It is tempting to treat tokenization as a boring preprocessing detail, but it quietly governs three things you care about deeply: how much you pay per request, how much text fits in the context window, and how well the model handles code, math, and non-English languages.

Post 2 makes those stakes concrete, with the per-token billing model, the 'language tax', and why subwords make models robust to typos and brand-new words.