Context Windows & KV Cache

Post three is the engine room. Having defined the window and argued why it matters, we now open up the actual mechanism: how attention processes tokens and why the KV cache makes generating long sequences practical. This is the most technical post of the five, and it's where the speed and memory behavior of LLMs stops being mysterious.

The core insight to hold onto: generation is autoregressive — one token at a time — and the work done for past tokens never needs redoing. The KV cache is simply the realization of that insight in code, and it's the reason the GPU memory bill for long context is what it is.

Attention is the heart of the transformer, and it reduces to three vectors per token plus one weighted sum. Each token is projected into a Query (what this token is looking for), a Key (what this token offers to others), and a Value (the content this token carries). To compute attention for a token, you take its Query, score it against every Key via a dot product, normalize those scores with softmax into weights, and use the weights to take a weighted average of all the Values.

That weighted average is the output that flows up to the next layer. Stacking many such layers, each refining the representation, is what lets the model build rich contextual understanding. The Query/Key/Value framing is worth memorizing because the entire KV cache story hangs on which of these three can be reused.

The flow diagram traces a single token's path through one attention operation. The token embedding is projected into Q, K, and V. The Query is multiplied against the transpose of the Keys to produce raw attention scores. Softmax converts those scores into a probability-like set of weights that sum to one. Finally, those weights multiply the Values and sum, producing the attention output.

Seeing it as a linear pipeline clarifies where the reuse opportunity lives. The K and V projections happen per token and depend only on that token. The expensive recompute people fear comes from redoing K and V for the entire history at every step — which is precisely what the cache eliminates.

Generation splits into two phases with very different performance profiles, and conflating them causes a lot of confusion. Prefill processes the entire prompt in one parallel pass — all tokens at once — computing and storing their Keys and Values. Because it's a big parallel matrix operation, prefill is compute-bound: it's limited by raw arithmetic throughput.

Decode is the opposite. Tokens are produced one at a time, and each step appends exactly one new Key/Value pair to the cache. The arithmetic per step is small, but the model must read the entire growing cache to compute attention, so decode is memory-bandwidth-bound. This is why generating long outputs feels slow even on fast hardware: you're bottlenecked on shuttling the cache through memory, not on math.

This compare diagram puts the two phases side by side. Prefill reads the full prompt at once, runs as a parallel compute-heavy operation, and its job is to build the cache. Decode runs one token per step, is sequential and memory-heavy, and its job is to append to the cache. Understanding this split explains real-world latency: 'time to first token' is dominated by prefill, while 'tokens per second' afterward is dominated by decode.

It also explains optimization priorities. Speeding up prefill is about compute and parallelism; speeding up decode is about reducing how much memory each step has to touch, which is exactly what cache-shrinking techniques target.

This is the deep reason the cache works: causality. In a decoder-only language model, attention is masked so each token can only attend to itself and the tokens before it — never the future. As a consequence, the Key and Value vectors of an earlier token depend solely on that token and its predecessors, and absolutely nothing that comes later can change them.

That immutability is what makes caching not just an optimization but a correct one. Once you've computed token five's K and V, they are frozen for the rest of the request. You can store them and reuse them for every subsequent generation step with zero loss of accuracy. If attention were bidirectional, as in an encoder, this guarantee would vanish and the trick wouldn't apply.

Without the cache, every new token would require recomputing the Keys and Values for the entire sequence so far. Generating the nth token would cost work proportional to n, and generating a full sequence would therefore cost on the order of n-squared total — quadratic and brutal for long outputs. The KV cache flattens this: each token's K and V are computed exactly once, then stored.

With the cache, producing a new token only requires computing one new Query, one new K/V pair, and attending against the stored cache — linear work per step. The trade is memory for compute: you avoid the recomputation but must hold every past K and V in GPU memory. For long sequences that trade is overwhelmingly worth it on speed, which is why caching is on by default everywhere.

Here's where the cache becomes the dominant cost. Its size follows a clean formula: two (one each for K and V), times the number of layers, times the number of attention heads, times the dimension per head, times the sequence length, times the bytes per element. Every factor multiplies, so the cache grows linearly with sequence length but is also scaled up by the model's depth and width.

The concrete number is sobering: a 7-billion-parameter model with 32 layers, in 16-bit precision, spends over 2 GB of cache for just 8K tokens. Stretch the context to 128K and the cache can exceed the size of the model weights themselves. This is the true bottleneck for long-context serving — not compute, but the memory the cache demands, especially when serving many users at once.

This snippet shows the cache's payoff in the structure of a generation loop. Notice that inside the loop, only the single new token is fed to the model, while the cache (passed as past_kv) carries all the context from before. The model returns the next-token logits and an updated cache, you pick a token, and you repeat. The full history is never re-fed.

This is the conceptual shape of what every inference engine does under the hood. The first call (prefill) processes the prompt and seeds the cache; every subsequent call (decode) feeds one token and grows the cache by one position. Post four will run this for real in Hugging Face transformers, but the logic here is the whole idea in eight lines.

This function turns the cache-size formula into a calculator so the memory numbers stop being hand-wavy. Plug in layers, heads, head dimension, and sequence length, and it returns the byte count, with the factor of two accounting for storing both Keys and Values. The example — 32 layers, 32 heads, 128-dim heads, 8,192 tokens at 2 bytes each — yields about 2.15 GB.

Running this for your target model and context length is genuinely useful capacity planning. It tells you how much GPU memory each concurrent request's cache will consume, which directly determines how many users you can serve on a given card. When you see why 128K context is expensive to host, this formula is the reason.

These are the real techniques used in production to keep the cache from eating all your memory. Grouped-query and multi-query attention (GQA/MQA) share a small number of Key/Value heads across many Query heads, slashing the cache by a large factor with minimal quality loss — which is why most modern models use them. Quantizing the cache to 8-bit or 4-bit cuts its footprint further by reducing bytes per element.

Sliding-window attention keeps only the most recent tokens' K/V, bounding the cache regardless of total length. And PagedAttention, popularized by vLLM, manages cache memory in fixed pages like an operating system manages RAM, eliminating the waste from over-reserving contiguous blocks. Together these are why serving long context at scale is feasible at all.

That's the engine. You now understand attention's Query/Key/Value decomposition, the prefill-versus-decode split, why causality makes Keys and Values safe to cache, how the cache size explodes with length, and the tricks that tame it.

Next post makes all of this tangible: real Hugging Face code that counts tokens, generates with the cache, benchmarks cache-on against cache-off, and inspects the actual cache tensors so the formulas turn into something you can see and time yourself.