Tokenization Explained

This is the opening post of a five-part series on tokenization, and its job is to install one clean idea before any algorithm or pricing math arrives: the model never reads your words, it reads tokens. The headline is deliberately visceral — the model cuts your text into pieces, and those pieces are not the words you typed.

Tokenization is the genuine first step of every language model, sitting before embeddings, attention, and generation. Framing it as the foundation everything else stands on sets up the rest of the series, where we'll show how this single step decides cost, context limits, and even which tasks a model can do at all.

The definition is intentionally precise: a token is the smallest unit of text the model processes, and crucially it's usually a sub-word fragment rather than a whole word. The second half of the definition — mapping each token to an integer ID — is the part beginners miss. The neural network does linear algebra on numbers; it has no concept of letters or strings.

The phrase 'the model never sees letters; it sees a list of numbers' is the load-bearing intuition for the entire series. It directly explains the strawberry-counting failure in the next post and the truncation bugs in the last one. Once a reader internalizes that text becomes integers before anything else happens, a lot of model behavior stops looking mysterious.

This slide answers the obvious first question: if we have words, why not just use them as tokens? The answer is a three-way tradeoff. A word-level vocabulary would need millions of entries to cover every name, typo, and neologism, and it would still fail on anything genuinely new by emitting an unknown-token placeholder.

Characters sit at the opposite extreme — a tiny vocabulary, but sequences become very long and each unit carries almost no meaning, making the model's job harder. Sub-words are the engineered compromise: a fixed, manageable vocabulary that can still represent any input by composing it from fragments. This tradeoff is the reason modern tokenization exists at all, and it motivates the BPE algorithm covered later.

The comparison table makes the word-versus-subword tradeoff concrete side by side. The left column shows why pure word-level tokenization fell out of favor: an enormous vocabulary that still breaks the moment it meets a word it never saw during training. The right column shows the modern default — a fixed vocabulary in the tens of thousands that gracefully composes unseen words from known pieces.

The example 'unhappiness' splitting into 'un' + 'happi' + 'ness' is the clearest demonstration. The model has never needed to memorize 'unhappiness' as a unit; it can build the meaning from familiar fragments it has seen thousands of times. This is exactly the property that lets a fixed vocabulary handle an effectively infinite language.

This slide walks the actual mechanics of turning a string into model input in three moves: normalize, split, look up. Normalization handles lowercasing and Unicode cleanup so that visually identical text maps consistently. Splitting applies the tokenizer's learned rules. Lookup replaces each token string with its integer ID from the vocabulary.

The key takeaway is the final sentence: that list of integer IDs is the real input to the network. Everything a reader thinks of as 'sending text to a model' is actually sending a list of numbers. This reframing pays off across the series, because cost, context limits, and many quality issues are all properties of that number list, not of the original text.

The pipeline diagram visualizes the string-to-numbers journey with a concrete example: the word 'tokenizing!' becomes fragments, which become IDs. Showing the exclamation mark as its own piece quietly makes an important point — punctuation is tokenized too, and isn't glued to the word it follows.

The diagram is intentionally simple because its purpose is to cement the mental flow, not to be technically exhaustive. The IDs shown are illustrative. What matters is that the reader can trace a single string left to right and watch it transform into the numeric form the model actually consumes.

The code slide grounds the whole concept in something runnable on the first post, so 'tokens' stop being abstract. Loading GPT-2's tokenizer and calling tokenize on 'tokenizing is fun' returns the actual fragments, including the 'Ġ' marker that GPT-style byte-level tokenizers use to denote a leading space.

Seeing 'tokenizing' split into 'token' + 'izing' is the payoff of the sub-word discussion two slides earlier — the abstraction is now observable in three lines of code. The second call, encode, shows the integer IDs, reinforcing the definition's claim that text becomes numbers. The 'Ġ' detail also foreshadows that spaces are part of tokens, a subtlety that trips people up later.

This slide defines the vocabulary — the model's fixed, frozen dictionary mapping every known token to an ID. The numbers matter for intuition: GPT-2 has roughly 50,000 entries while many modern models exceed 100,000. The size is a design choice balancing coverage against the cost of a larger embedding table and output layer.

The critical, often-missed point is that the vocabulary is learned once during tokenizer training and then frozen. After that, anything you ever type must be expressed using only those pieces. This is why a brand-new word or an unusual symbol gets shattered into many small tokens — the vocabulary can't grow to accommodate it, so it falls back to smaller fragments it already knows.

Setting boundaries is as valuable as the definition itself, so this slide names three things tokenization is not. It is not simply splitting on spaces — punctuation, emoji, and sub-word pieces are all handled distinctly. It is not grammar-aware; the tokenizer has no notion of nouns or verbs, only statistical fragments. And one token is emphatically not one word.

That last point — token count is not word count — is the thread that runs through the entire series. A single emoji can consume several tokens while a common English word is exactly one. Misunderstanding this is the root of the cost-estimation and context-limit mistakes covered in the final post, which is why it's worth establishing the boundary this early.

The recap distills the post into five lines a reader could repeat from memory: a token is the model's smallest text unit, modern models use sub-words, every token maps to an integer ID, the vocabulary is fixed and finite, and token count is not word count.

These five points are the scaffold for the rest of the series. The why-it-matters post builds on 'billed per token' and 'token count ≠ word count'; the how-it-works post explains how that fixed vocabulary gets built; the code post makes all five observable; and the mistakes post is essentially a list of what happens when you forget them.

The closing card points forward to the Why It Matters post, which converts this conceptual foundation into stakes a reader feels: money, context limits, and quality. After establishing what a token is, the natural next question is why anyone should care.

The teaser is framed around cost and context deliberately, because those are the two things every LLM developer hits within their first week. Promising to explain the bill and the context window makes the next post feel immediately useful rather than academic.