Tokenization Explained

This second post pivots from 'what' to 'why', and the cover frames tokenization not as plumbing but as the unit of account for the entire LLM economy. The three opening claims — you're billed per token, your context window is measured in tokens, your model's blind spots are token artifacts — are each unpacked in the slides that follow.

The goal is to move the reader from seeing tokenization as a low-level detail to recognizing it as the lens through which cost, capability, and fairness all become predictable. By the end, a string of seemingly unrelated mysteries should resolve into a single underlying cause.

The first and most concrete reason tokenization matters is money: every major LLM API meters usage in tokens, not words or characters. The rule of thumb worth memorizing — roughly 0.75 words per token, or about 4 characters per token for English — lets a reader sanity-check any bill or budget.

The slide deliberately points out that both input and output are billed. Beginners often budget only for their prompt and forget the completion costs tokens too, sometimes far more if the model is verbose. Internalizing 'a 1,000-word prompt is ~1,300 tokens before any reply' turns abstract pricing pages into numbers a developer can actually plan around.

The bar chart drives home that not all text tokenizes equally, which is the single most useful practical insight in the post. English prose is the efficient baseline. Code tokenizes worse because symbols, brackets, and identifiers fragment heavily. Non-English text is the worst offender, often 2-3x the tokens. JSON sits high too, thanks to its braces, quotes, and repeated keys.

The values are illustrative of relative intensity rather than precise measurements, and the axis label says as much. The takeaway is the ordering, not the exact heights: the same visible amount of content can cost wildly different amounts depending on what kind of content it is. This directly sets up the cost-estimation mistake in the final post.

Context windows are the second place tokenization quietly governs behavior. The advertised size — 8k, 128k, or a million — is a token budget, not a word count, and everything competes for it: the system prompt, the full chat history, any retrieved documents, and the model's own answer all draw from the same pool.

The practical consequence is that when the budget is exceeded, content gets dropped, usually the oldest messages, which is why long conversations 'forget' their beginning. Knowing token math is therefore the same as knowing how much actually fits, and it's the difference between designing a prompt that reliably works and one that silently loses information.

The strawberry problem is the most memorable illustration that many 'dumb' model errors are really tokenization artifacts. When you ask how many r's are in 'strawberry', you assume the model can see the letters — but it can't. The word may be a single token or split into a couple of chunks, and counting characters requires reasoning about something the model never directly observes.

This reframing is genuinely useful because it changes how a reader debugs. Instead of concluding 'the model is bad at counting' or 'the model is dumb', they learn to ask whether the task requires character-level access that tokenization has hidden. Many spelling, rhyming, and counting failures share this exact root cause.

The flow diagram lays the strawberry problem out as a chain so the cause is unmistakable: you see the letters s-t-r-a-w-b-e-r-r-y, but the model sees something like the two tokens 'straw' and 'berry'. When asked to count the r's, it has no per-letter access, so it guesses — and often guesses wrong.

The diagram's value is showing that the failure is structural, not a matter of the model being insufficiently smart. No amount of additional reasoning fully fixes a problem where the relevant information was discarded at the tokenization step. This is the kind of insight that separates someone who merely uses LLMs from someone who understands their failure modes.

The language tax surfaces an uncomfortable consequence of how tokenizers are trained: predominantly on English text. Because BPE merges the most frequent sequences, English words and fragments earn efficient single tokens, while scripts that were underrepresented in training — Hindi, Thai, Burmese, and many others — get broken into many small pieces.

The practical fallout is real and threefold: speakers of those languages pay more per equivalent message, get slower responses, and have less effective context window. Framing this as 'tokenization encodes a real inequity' is honest rather than alarmist — it's a measurable, structural bias, and naming it prepares the reader to budget and test for it in the final post.

The code slide makes the language tax measurable rather than abstract. Encoding the same idea in English and in Hindi with the same tokenizer shows the English version at a handful of tokens and the Hindi version at four times as many or more. Running it yourself is far more convincing than being told.

Using tiktoken's cl100k_base here ties the demonstration to a real, widely used encoding, so the numbers reflect what an actual GPT-4-family call would cost. The Hindi string is written with explicit Unicode escapes to keep the JSON portable and unambiguous. The lesson lands hard: identical meaning, multiples of the cost, purely because of the tokenizer's training distribution.

Retrieval-augmented generation is where all of this compounds into real operational cost, which is why it gets its own slide. Every retrieved chunk is injected into the prompt and billed as input tokens on every single call — so chunk size, overlap between chunks, and the number of results retrieved are not just quality knobs, they're direct cost-and-latency levers.

Calling tokenization 'the hidden accountant behind every RAG system' captures the point: teams obsess over retrieval quality and forget that each design choice has a token price paid repeatedly at scale. A reader who connects RAG configuration to token budgets will design cheaper, faster systems without sacrificing much quality.

The recap compresses the post into five lines of stakes: you're billed per token not per word, context windows are token budgets, letter-level tasks fail because models see tokens, non-English text costs 2-3x more, and RAG cost is fundamentally a token-budget problem.

Each line maps to a slide and, more importantly, to a real decision a developer makes. The recap is designed so that someone who only remembers these five sentences still walks away able to estimate a bill, plan a context window, debug a counting failure, anticipate multilingual cost, and reason about RAG economics.

The CTA hands off to the How It Works post, promising to open the hood on BPE — the algorithm that actually builds the vocabulary every previous claim depends on. Having established why tokenization matters, the reader is primed to ask how the splits are decided in the first place.

Framing the next post around 'building a vocabulary from raw text' signals that it will be mechanical and concrete, the satisfying payoff for anyone who's been wondering why 'tokenizing' splits one way and 'the' another.