BPE & WordPiece

This cover reframes tokenization from an academic footnote into a business concern. The tokenizer is the layer almost nobody on a product team thinks about, yet it sets the denominator for every cost and capacity number in an LLM application. Understanding it is the difference between a prompt that costs a cent and one that costs a dime for the same information.

The post walks through four concrete stakes — cost, vocabulary-size tradeoffs, cross-language fairness, and robustness — so that by the end you can look at a feature and predict how the tokenizer will help or hurt it.

The single most important fact for anyone using an LLM commercially: you are billed per token, and the context window is denominated in tokens, not words or characters. The tokenizer is the function that converts your text into that token count, which means it directly determines spend, the share of the window your prompt consumes, and part of your latency.

This is why two prompts that look the same length to a human can cost very differently. A prompt dense with code, numbers, or non-English text produces more tokens than a prompt of plain English at the same character count. If you optimize anything about an LLM system, optimizing token count is usually the highest-leverage lever you have.

This chart visualizes the core surprise: identical character counts produce wildly different token counts depending on content. Plain English is the cheapest, hovering around 0.75 words per token. Code costs more because symbols, indentation, and identifiers fragment. Languages in non-Latin scripts like Hindi cost more still, and a mess of emoji and typos can fall back to raw bytes, ballooning the count.

The practical lesson is to never estimate cost or context usage from character or word length alone. The only reliable number is the actual token count produced by the specific tokenizer your model uses, which is why post 4 shows you how to measure it.

Before subwords, word-level systems had a hard ceiling: any word not in the fixed vocabulary collapsed into a single [UNK] token, destroying information. Names, slang, technical jargon, and newly coined terms all hit this wall, and the model simply lost them.

Subword tokenization eliminates this failure mode by construction. The worst case for an unknown word is that it gets spelled out from smaller known pieces, or in byte-level schemes from raw bytes. The model always receives a meaningful, lossless representation. This robustness is a quiet but enormous upgrade — it is why modern models gracefully handle text full of novel terms instead of degrading.

Choosing the vocabulary size is a genuine engineering tradeoff with no free lunch, and this comparison lays out both sides. A larger vocabulary packs more text into each token, so sequences are shorter and a given prompt costs fewer tokens. But it requires a bigger embedding table (more parameters), and the rarest pieces appear so seldom in training that their embeddings stay poorly learned.

A smaller vocabulary inverts everything: fewer parameters and every piece is seen often enough to train well, but text fragments into more tokens, lengthening sequences and raising cost. Real models pick a point in the middle — commonly 32k for BERT-era models, up to 100k+ for recent LLMs — balancing these forces against their training data and target languages.

The 'language tax' is one of the most underappreciated fairness issues in deployed AI. Because most tokenizers are trained on corpora dominated by English, English maps to very few tokens per word. Languages written in other scripts — Hindi, Thai, Arabic, and many others — frequently fragment into several tokens for a single word, sometimes down to individual characters or bytes.

The downstream effects are concrete and unfair. A speaker of one of these languages pays two to four times more for the same request, fits far less of their document in the context window, and gets slower responses. When you build a multilingual product, measuring the token cost per target language is not optional — it directly shapes pricing, limits, and user experience.

This mindmap organizes the entire 'why it matters' story into four branches you can hold in your head. Cost covers per-token billing and the language tax. Context covers how the window is measured in tokens and how prompts get truncated when they overflow. Quality covers how digit and code fragmentation degrade math and programming performance. Robustness covers graceful handling of typos and brand-new words.

Used as a checklist, these four branches let you predict where tokenization will help or hurt any feature you design, before you write a line of code.

Robustness to messy input is a direct, almost free benefit of decomposing words into pieces. A typo like 'teh' or a never-before-seen product name does not break the tokenizer; it simply produces a sequence of subword or byte pieces the model can still reason about. The system degrades gracefully instead of failing outright.

Byte-level BPE, used by the GPT family, takes this to its logical end. Because it operates on the 256 possible byte values rather than on characters, it can represent literally any input — every emoji, every rare script, every binary smudge — with no possibility of an unknown token. The cost is that exotic input fragments into many tokens, but it is never lost.

This snippet turns the cost discussion into something you can act on immediately. The tiktoken library exposes the exact tokenizer used by OpenAI models, so you can count tokens precisely before sending a request. Here a nine-word English sentence is exactly nine tokens — a tidy case — while a long rare word like 'antidisestablishmentarianism' fragments into several pieces.

Making token counting a habit pays off constantly: it lets you predict cost, verify a prompt fits the context window, and catch the cases where seemingly short text is secretly expensive. Post 4 builds this into a full workflow.

These takeaways convert the post into operating rules. Budget in tokens, never words, because the conversion ratio swings hard with content and language. Expect non-English text to cost more and plan pricing and limits accordingly. Treat prompt length as a cost and latency lever — shorter is cheaper and faster. Trust that subwords have eliminated the [UNK] failure mode. And always measure real token counts before you ship rather than estimating.

Internalizing these five rules is what separates someone who is surprised by their LLM bill from someone who predicted it.

With the stakes clear, the natural next question is mechanical: how does the tokenizer actually decide which pieces go in the vocabulary? Post 3 opens the hood on BPE's training loop — the deceptively simple count-merge-repeat process that started life as a 1994 compression algorithm — and shows precisely how WordPiece changes the merge rule.

Understanding the algorithm is what lets you reason about why a particular word splits the way it does, and sets up the hands-on training in post 4.