BPE & WordPiece

This closing post is the field guide to tokenization failures, and its framing is deliberate: the worst tokenization bugs are silent. There is no stack trace, no crash, no error log. The code runs cleanly, the output reads plausibly, and yet the model is quietly degraded, the bill quietly inflated, or the numbers quietly wrong.

Because nothing fails loudly, these issues survive into production and waste enormous time to diagnose. The five mistakes here cover the ones that bite real systems most often, each paired with why it happens and how to prevent it.

The most damaging and most common tokenization mistake is pairing a model with a tokenizer it was not trained on. A model's learned embeddings are indexed by token ID, and that ID-to-piece mapping is specific to one tokenizer. Feed IDs from a different tokenizer and every lookup retrieves the wrong embedding — the model is effectively reading a scrambled language.

What makes this insidious is that it does not crash. The model still produces grammatical, confident-sounding output; it is just subtly or severely wrong, with no signal that anything broke. Teams can ship this and only notice weeks later when quality metrics inexplicably lag.

The fix for the mismatch problem is a discipline, not a clever trick: always load the tokenizer and the model weights from the same source identifier. Using AutoTokenizer.from_pretrained(name) and AutoModel.from_pretrained(name) with an identical 'name' guarantees the ID mapping the model expects matches the IDs the tokenizer produces.

The rule to internalize is to never mix a tokenizer from one checkpoint with weights from another, even if they are the same architecture. Saving and loading them as a unit, as shown in post 4, is the structural way to make this mistake impossible.

The leading-space trap is specific to byte-level BPE tokenizers like GPT's, and it surprises almost everyone the first time. In these tokenizers, the space before a word is folded into the token, so ' dog' (with a leading space) and 'dog' (without) are two completely different token IDs. The model treats them as different inputs.

This bites when you build text by hand — concatenating strings, stripping whitespace, or splicing fragments — because you can easily produce token sequences the model rarely or never saw during training. The result is subtly worse generation that is maddening to trace back to its cause.

This trace makes the leading-space issue impossible to ignore. Encoding 'dog' yields one ID; encoding ' dog' with a single leading space yields a different ID entirely. Same three letters, different token, different embedding, different behavior.

The takeaway in the final line is the fix: let the tokenizer handle spacing. Pass it natural, well-formed text and avoid hand-assembling token sequences or aggressively stripping whitespace. When you respect the tokenizer's own conventions, the spacing just works; when you fight them, you create phantom bugs.

The 'one token equals one word' assumption is comforting and wrong, and it routinely breaks cost and context budgets. English averages roughly 0.75 words per token, so even clean prose is denser in tokens than the word count suggests. Code, JSON, numbers, and non-English text are far denser still, sometimes several tokens per word.

Budgets built on the word assumption fail in two directions: prompts overflow the context window unexpectedly, and cost projections come in well under the real bill. The only safe practice is to measure actual token counts for representative inputs rather than estimating from word counts.

This chart shows just how far the word-equals-token assumption drifts across content types. Plain English is the friendly case at roughly 0.75 words per token. JSON drops lower because braces, quotes, and keys fragment. Raw numbers are worse, since digit strings split into pieces. And a script like Thai can fall to a fraction of a word per token, meaning text costs several times more than its word count implies.

Reading the bars as 'lower means costlier' gives an at-a-glance sense of which inputs will blow your budget. Any system handling code, structured data, or non-English text needs real measurement, not estimation.

Digit and spelling failures trace directly back to tokenization, which is a satisfying 'aha' for anyone puzzled by LLM arithmetic mistakes. Tokenizers split numbers in inconsistent, content-dependent ways — '18452' might become '18' and '452' — so the model never sees a clean, place-value representation of the number. Reliable arithmetic on fragments it cannot cleanly perceive is genuinely hard.

The same root cause explains why models stumble on tasks like counting letters in a word or reversing a string: the characters are bundled inside subword tokens, so the model literally cannot see them individually. These are not reasoning failures so much as perception failures imposed by tokenization.

This snippet proves the digit-fragmentation point with the real GPT-4 tokenizer. Encoding a sentence containing '18452' and decoding each token individually reveals the number breaking into pieces like '18' and '452' — the model never receives the full number as a single unit.

Running this is the fastest way to viscerally understand why you should not trust raw LLM arithmetic, and why production systems route real calculations to a tool or code interpreter instead. Once you see the number shatter in your own console, the model's math quirks stop being mysterious.

The language tax returns here as a deployment mistake rather than an abstract fairness point. Building a product on an English-centric tokenizer and serving it to non-English users means their text consumes two to four times more tokens for the same meaning. That translates directly into higher bills for those users, less of their content fitting in the context window, and slower responses.

The fix is to measure token counts in your actual target languages before you set pricing, design context limits, or promise performance. A limit that feels generous in English can be punishingly tight in another script, and discovering that after launch is an expensive lesson.

This final list is the consolidated fix sheet for every mistake in the post. Always pair a tokenizer with its own model. Let the tokenizer own spacing rather than hand-editing token sequences. Budget in real, measured tokens instead of word counts. Never trust raw LLM arithmetic — route real math to a tool. And test token counts in every language you support before committing to limits or pricing.

Five rules, each mapping to one silent failure mode. Adopt them and you sidestep the tokenization bugs that most often surface in production.

This wraps the five-post arc on BPE and WordPiece: the concept, the stakes, the algorithm, the hands-on code, and now the failure modes. Tokenization is the foundation layer — it turns text into the integer IDs everything else operates on.

Day 58 moves one step up the stack to embeddings: how those token IDs become dense vectors that capture meaning, and why words used in similar contexts end up near each other in vector space. Tokens get the model something to read; embeddings give it something to understand.