BPE & WordPiece

This is the hands-on post, and its purpose is to collapse the distance between understanding the algorithm and running it. Everything described in post 3 — training, merging, encoding — happens here in a few lines of real Python, using the two libraries you will actually reach for in practice: Hugging Face 'tokenizers' for training your own, and 'tiktoken' for the exact GPT tokenizer.

The blocks are ordered to run top to bottom, but each is self-contained enough to read on its own. The aim is that by the end you have trained a tokenizer, inspected its pieces, compared BPE with WordPiece, and counted GPT-4 tokens — the full practical loop.

This block sets up a deliberately tiny, repetitive corpus so the training is fast and the resulting merges are easy to interpret. Repeating a small set of words like 'low', 'lower', 'lowest', 'newer', 'newest' hundreds of times makes the frequency statistics obvious — you will be able to predict which pairs merge first.

The extra sentence with 'tokenization', 'unbelievable', and 'reusable' seeds some longer words so you can watch them fragment. In a real project you would point the trainer at gigabytes of text, but a toy corpus is the right way to first see the mechanics clearly.

Here you train an actual BPE tokenizer. The model is BPE with an unknown-token slot; the pre-tokenizer splits on whitespace so the trainer sees words; and the BpeTrainer runs the count-merge-repeat loop until it hits the requested vocabulary size of 60. The deliberately tiny vocab size forces interesting splits on the longer words.

The whole training process is a handful of lines because the library encapsulates the loop from post 3. Printing the vocabulary size at the end confirms training finished. Swap in a larger corpus and a bigger vocab_size and this same code trains a production-grade tokenizer.

This block exercises the trained tokenizer in both directions. encode() takes a string and returns an object exposing both the human-readable .tokens (the subword pieces) and the .ids (the integers the model would actually consume). Inspecting .tokens is the single best way to build intuition about how your tokenizer behaves on real input.

decode() reverses the process, turning IDs back into text. Verifying that decode(encode(x)) reproduces your original text is a cheap, essential sanity check — if a round-trip mangles your text, something is misconfigured before you have even involved a model.

Now you train WordPiece on the identical corpus and vocab size, changing only the model and trainer classes. Running the same word, 'unbelievable', through it lets you compare the two algorithms head to head on equal footing. The most visible difference is the '##' prefix on continuation pieces — WordPiece marks which fragments continue a word, while BPE here does not.

This direct comparison is the payoff of the post. The splits may differ in where boundaries fall because WordPiece optimizes likelihood rather than raw frequency, exactly as post 3 described. Seeing two tokenizers disagree on the same word makes the algorithmic difference tangible.

This block introduces tiktoken, which gives you the genuine tokenizer used by OpenAI's GPT models rather than one you trained. Encoding a sentence returns the real token IDs and count that would be billed. The loop that decodes each ID individually is the trick for seeing exactly how GPT carves up your text, piece by piece.

This is the tool you will use most often in day-to-day LLM work — not to train tokenizers, but to count and inspect tokens for cost estimation and debugging. It reveals surprises like numbers and code fragmenting in ways plain English does not.

This comparison summarizes what running both tokenizers reveals. BPE output, especially the byte-level GPT variant, carries no '##' markers and encodes spacing into the tokens themselves; it chooses merges by frequency and is the style behind GPT and Llama. WordPiece output shows '##' on every continuation piece, chooses merges by likelihood, encodes via greedy longest-match, and is the BERT-family style.

Having just produced both outputs yourself, this table is no longer abstract — each row corresponds to something you can point to in your own console output, which is the best way to lock the distinction into memory.

Tokenizers are meant to be persisted and shipped, and this block shows how. Saving to a single JSON file captures the entire vocabulary and merge list, and reloading it reproduces the exact same tokenization. This portability is why a model checkpoint always travels with its tokenizer file.

The practical rule this enables: whenever you distribute or deploy a model, the tokenizer JSON must go with it, versioned together. A model paired with the wrong tokenizer version is the bug that post 5 opens with, and saving them as a unit is the cleanest way to prevent it.

Even with clean code, the conceptual mistakes here are the expensive ones. Pairing a model with the wrong tokenizer corrupts every embedding lookup. Forgetting special tokens like [CLS] and [SEP] breaks models that depend on them as structural anchors. Assuming one token equals one word wrecks cost and context budgeting. Ignoring the leading-space convention in byte-level BPE produces token sequences the model never trained on. And retraining a vocabulary on too little data yields a tokenizer that fragments real-world text badly.

Each of these runs without raising an error, which is exactly what makes them dangerous — and why post 5 is dedicated entirely to them.

This checklist turns the post into deployment discipline. Always load the model's own tokenizer rather than constructing one separately. Count tokens with tiktoken before you rely on any cost or context estimate. Save the tokenizer and model together so they can never drift apart. And test that decoding round-trips your actual text before trusting the pipeline.

These four habits catch the overwhelming majority of tokenization problems before they reach production.

Post 4 gave you the tools; post 5 gives you the scar tissue. Knowing how to train and run a tokenizer is not the same as knowing the silent ways it goes wrong in real systems — mismatched tokenizers, leading-space traps, the token-equals-word fallacy, digit fragmentation behind bad arithmetic, and the non-English language tax.

The final post is the field guide to those failure modes, why each happens, and the concrete fix, so you can avoid debugging them in production at 2am.