Context Windows & KV Cache

This post sets up the two objects that govern everything about how large language models handle input: the context window and the KV cache. The headline pairs them deliberately, because they are two sides of the same coin. The window is the conceptual limit on what the model can see; the cache is the implementation detail that makes generating tokens within that window fast.

Before we judge whether bigger windows matter or crack open the attention math, the goal of post one is simply to make these two terms concrete and unmysterious. Almost every confusing behavior you'll meet later — forgotten instructions, surprise costs, slow generation — traces back to one of these two ideas.

The crucial word in this definition is 'single pass.' A language model is fundamentally stateless: it takes a sequence of tokens in, runs one forward pass, and produces a probability distribution over the next token. The context window is the maximum length of that input sequence. There is no hidden memory bank the model consults between requests; if a piece of text is not inside the window for this specific call, it has zero influence on the output.

This reframes 'memory' in a useful way. What feels like a chatbot remembering your name is just your name being re-included in the token sequence on every turn. The window is the model's entire universe for the duration of one request, and the size of that universe is fixed.

Tokens are the unit that actually matters, and they are neither words nor characters. A tokenizer breaks text into sub-word fragments chosen to be statistically efficient over the training corpus. Common words are often a single token, while rare words, code, and non-English text fragment into many. The string 'unbelievable' might become three tokens; a chunk of JSON might be far denser than its character count suggests.

The practical rules of thumb — roughly 0.75 words per token, or about 4 characters per token for English — are fine for back-of-envelope estimates but dangerous for hard limits. When you need to know whether something fits, you measure with the model's own tokenizer. A 128K-token window sounds enormous until you realize a dense technical document plus a long chat history can consume it surprisingly fast.

This slide makes a point people consistently miss: the context window is a single shared budget, and every category of content draws from the same pool. The system prompt that defines the assistant's behavior, the tool and function schemas you expose, the documents retrieved for a RAG query, the entire back-and-forth of the conversation, and the tokens the model is actively generating all live in the same finite space.

That shared-budget framing has immediate consequences. A verbose system prompt isn't free — it permanently reduces how much conversation or document content you can fit. Adding more retrieved chunks crowds out history. Designing an LLM feature is largely an exercise in allocating this budget on purpose rather than letting it fill up by accident.

The KV cache is the performance machinery behind generation, and it's worth understanding even at a conceptual level. Inside the attention mechanism, every token is projected into three vectors: a Query, a Key, and a Value. The Keys and Values of all the tokens processed so far are exactly what a new token needs to attend to the past. The KV cache simply stores those Key and Value vectors so they don't have to be recomputed for every new token.

The important nuance is what the cache is not. It is not extra semantic memory and it does not extend the window. It's a speed optimization that lives only for the duration of a single request. When the request ends, the cache is discarded. Post three will open this up in full; here it's enough to know the cache exists to make decoding fast.

The stack diagram shows the literal layering of a typical request. At the bottom sits the system prompt — fixed instructions that anchor the model's behavior. Above it sits the bulk of the content: conversation history and any retrieved documents. Then the current user message, and finally the model's own generated tokens, which grow as it writes.

Visualizing it as a stack with a hard top edge makes the failure mode obvious. When the stack tries to grow past the window, something has to give — usually the oldest content at the bottom of the visible region gets dropped, or the request is rejected. Generation eating into the same budget is why setting a large max-output value can itself push you over the edge.

The hard ceiling on context isn't an arbitrary product decision so much as a consequence of how self-attention scales. Standard attention compares every token against every other token, which means compute and memory grow with the square of the sequence length. Doubling the context roughly quadruples the attention cost. That quadratic wall is why windows are capped at training time rather than left open-ended.

There's a second, subtler point: the advertised maximum is a ceiling, not a sweet spot. Models extended to very long contexts — through positional-encoding tricks or fine-tuning — frequently show degraded reasoning and recall well before the stated limit. Treat the maximum as the edge of a cliff, not a target to fill.

This comparison untangles two things people routinely conflate: what the model learned during training and what it can see right now. Training data is the billions of tokens compressed into the model's weights during pretraining. It's frozen, vast, and implicit — the model 'knows' it but can't quote it precisely. The context window is the live text you supply at request time, explicit and exact.

The analogy that sticks: weights are long-term memory, the window is short-term working memory. The model cannot move anything from the window into the weights — it does not learn from your prompts. Whatever you put in the window influences only the current request and then evaporates. Persistent memory across sessions is something you engineer on top, not a property of the model.

This snippet makes the abstract concrete by counting tokens directly. Using tiktoken with the cl100k_base encoding — the same family used by several widely deployed models — we encode a short sentence and see it become seven integer token IDs. The point isn't the exact number; it's the habit of measuring rather than guessing.

Every serious LLM application has a token-counting step somewhere, because it's the only reliable way to know whether content fits the window and to estimate cost. The integers printed are the actual indices into the model's vocabulary; the model never sees your text as letters, only as sequences of these IDs. Internalizing that tokens are the true currency removes a lot of confusion later.

Naming what is not in the window is as important as defining what is. Anything from a separate, earlier API call is gone unless you explicitly resend it. A file the model processed in a previous request has left no trace. Knowledge outside its training data that you didn't paste in simply does not exist for the model. And tokens that scrolled off the front edge of a long conversation are no longer influencing anything.

This list is the antidote to magical thinking about LLMs. When a model 'forgets' or 'ignores' something, the first question is almost always: was that content actually inside the window for this call? More often than not, the answer is no, and the fix is in your code, not the model.

The summary compresses the whole post into four lines you can recall under pressure. The window is working memory, capped and measured in tokens. The model is stateless, so every call starts from a blank slate and only knows what you include. The KV cache is a speed trick for fast generation, not a form of memory or a way to extend context. And exceeding the limit means the oldest tokens get dropped or the request fails.

Hold these four facts and most LLM behavior stops being surprising. They're the foundation the next four posts build on, from why the window shapes products to how the cache works in code.

That wraps the conceptual foundation. You now have a clean mental model: the context window as the model's finite, per-request working memory measured in tokens, and the KV cache as the optimization that keeps generation fast inside that window.

The next post shifts from definitions to stakes — why this single constraint quietly determines whether an AI feature is reliable, how it drives cost and latency, and why content buried in the middle of a long context gets ignored.