Tokenization Explained

This third post is the mechanical core of the series, and the cover poses the precise question the algorithm answers: why does 'tokenizing' split into 'token' + 'izing' while 'the' stays whole? The answer is neither magic nor grammar — it's a greedy, frequency-driven merge algorithm called Byte-Pair Encoding, learned entirely from data.

The post's promise is that once you see the algorithm, every weird split becomes explicable. BPE, WordPiece, and Unigram are all variations on the same bottom-up idea, and demystifying the training loop turns the tokenizer from a black box into a simple, inspectable procedure.

Starting from bytes is the foundation that makes modern tokenizers robust, so it leads the post. Because any character on Earth can be represented as one to four bytes, a byte-level tokenizer has no concept of a truly unknown input — emoji, foreign scripts, and obscure symbols all decompose into bytes it can always represent.

This is why byte-level BPE never needs to emit an unknown-token placeholder, a real limitation of older word-level systems. In the worst case it falls back to encoding raw bytes, which is inefficient but never fails. This robustness is the quiet reason GPT-style models handle arbitrary text gracefully, and it connects back to the language-tax discussion: unusual scripts are representable, just expensively.

This slide describes BPE training in plain terms: scan a huge corpus, count every adjacent symbol pair, merge the single most frequent pair into a new token, then recount and repeat thousands of times. The merges accumulate into the vocabulary from the bottom up.

The intuition to carry away is that frequency drives everything. Sequences that appear constantly — 'th', 'ing', 'the' — get merged early and earn their own tokens, which is exactly why common words are single tokens. Rare sequences never accumulate enough frequency to merge, so unusual words stay fragmented. This directly explains the splitting patterns the reader has been seeing since the first post.

The cycle diagram captures BPE training as the simple loop it actually is: count pairs across the corpus, find the top pair, merge it and add it to the vocabulary, then repeat until the vocabulary reaches its target size. Presenting it as a cycle emphasizes that the same four steps just run over and over.

The value of seeing it as a loop is demystification. There's no deep learning in tokenizer training, no gradients, no neural network — just counting and merging. A reader who grasps that the 'intelligence' of a tokenizer is really accumulated frequency statistics will never again treat the splits as arbitrary or mysterious.

The by-hand code slide strips BPE down to its essence so the reader can watch the first merge happen. It takes a tiny vocabulary of split characters, counts adjacent pairs with a Counter, and identifies the most frequent pair as the one to merge. The comment makes explicit that this then repeats: merge, rescan, merge again.

This is deliberately a teaching implementation, not production code — real BPE tracks word frequencies and end-of-word markers more carefully. But running even this simplified version makes the algorithm tangible. Seeing 'l' and 'o' identified as the top pair to merge connects the abstract description to a concrete, executable step the reader can modify and explore.

Encoding is the inference-time counterpart to training, and this slide draws the distinction clearly: training learns the merge rules, encoding merely replays them. Given new text, the tokenizer breaks it into base characters and then applies the learned merges in the exact order they were created, greedily combining pairs until no more apply.

The important property is determinism. The same input always produces the same tokens, every time, which is what makes token counts reliable and reproducible. This reliability is precisely what the previous post depended on when it talked about counting tokens to budget a call — the count is stable because encoding is a fixed replay of learned rules.

This code slide moves from the toy implementation to a real, production vocabulary via tiktoken, so the reader sees authentic behavior. Encoding 'lower lowest' and then decoding each ID individually reveals which words survive as single tokens and which fragment.

The pedagogical point is the contrast: a very common word like 'lower' is likely a single token, while 'lowest' may split, because frequency during training decided their fates. Decoding token by token is also a genuinely useful debugging technique — when a model behaves strangely on some input, inspecting the actual tokens often reveals an unexpected split at the root of the problem.

The comparison slide situates BPE among its siblings so the reader isn't confused by the alphabet soup of tokenizer names. BPE, used by GPT models, greedily merges the most frequent pairs bottom-up with byte-level fallback. WordPiece, used by BERT, is similar but chooses merges by which one most improves the training corpus likelihood rather than raw frequency.

Unigram takes the opposite direction: it starts from a large candidate vocabulary and prunes tokens that contribute least, keeping a probabilistically optimal subset. SentencePiece is the toolkit that implements these in a language-agnostic way, treating the input as a raw stream including spaces. The reader doesn't need to memorize the math — just to recognize these as variations on one bottom-up-or-prune theme.

Special tokens are the final piece of the mechanism, and they often surprise newcomers because they aren't in the text yet occupy real vocabulary IDs. The tokenizer injects them to encode structure the model relies on: BERT's [CLS] and [SEP] mark the start and boundaries of sequences, GPT's <|endoftext|> signals where generation should stop, and padding and mask tokens support batching and training.

Understanding that these are reserved, real entries in the vocabulary matters practically. They count against your token budget, they appear when you decode raw IDs, and forgetting them is a concrete mistake the final post addresses. They're the bridge between the abstract algorithm and the messy reality of feeding text to a specific model.

The trace diagram visualizes where special tokens sit relative to your actual content: a [CLS] marker opens the sequence, your real tokens 'what is BPE ?' sit in the middle, and a [SEP] marker closes it. Coloring the special tokens differently from the input makes the injection obvious at a glance.

The diagram reinforces that these tokens wrap around your text rather than being part of it, which is exactly the mental picture needed to avoid the off-by-a-few-tokens errors covered later. Seeing the structure laid out linearly makes the otherwise invisible scaffolding of a model's input concrete.

The recap compresses the mechanism into five memorable beats: start from bytes so there's no unknown token, merge the most frequent pair and repeat, build the vocabulary bottom-up, replay merges greedily at encoding time, and use special tokens to mark structure.

These five points give the reader a durable model of how any BPE-family tokenizer works. The framing is intentionally algorithm-first so that when they later encounter WordPiece or a new model's tokenizer, they can place it as a variation rather than learning it from scratch.

The CTA hands off to the code-heavy fourth post, promising a runnable, paste-and-go tour. Having understood the algorithm conceptually, the reader is ready to manipulate real tokenizers — counting, inspecting, decoding, and even training one.

This follows the natural pedagogical arc of the series: definition, motivation, mechanism, then hands-on practice. The next post is where the reader stops reading about tokenization and starts running it themselves.