Context Windows & KV Cache

Post five collects the failure modes. After the concept, the stakes, the mechanism, and the code, this is the practical payoff: the specific, repeatable mistakes that turn context windows and the KV cache from a feature into a bug source. Every one of these looks like the model misbehaving, but each is really a gap in how the system was built — which means each has a concrete fix on your side of the line.

Learning these from a carousel is dramatically cheaper than learning them from a production incident, a surprise bill, or a model that mysteriously 'forgot' what a user told it thirty seconds ago.

The first and most expensive mistake is treating the window as effectively infinite because the demo never tested its limits. Demos use a few short messages; real users paste entire documents, long error logs, and sprawling conversations. When the total token count crosses the window, behavior depends on the platform — some silently drop the oldest content, others reject the request — but neither is what you want happening unexpectedly in production.

The fix has two parts: count tokens before every send so you know where you stand, and have an explicit overflow strategy — trim oldest turns, summarize history, or chunk the input. The mistake isn't using a lot of context; it's not planning for what happens when there's too much.

Mistake two is budgeting context by words or characters instead of tokens. The rough four-characters-per-token heuristic is fine for casual estimates but fails exactly where it matters: on dense content. Code, JSON, structured data, emoji, and non-English text all tokenize far more densely than plain English prose, so a document you estimated at 2,000 'words' can easily be 3,000 or more tokens.

That underestimate is precisely how you end up overflowing a window you thought had room. The discipline is simple: when a limit actually matters, measure with the model's real tokenizer rather than a guess. Heuristics are for rough planning; the tokenizer is for decisions.

This fix shows the discipline in action. The fits function tokenizes the system prompt, history, and user message, sums their real token counts, and checks against the limit — but with one crucial detail: it subtracts a reserve for the model's response. Because output tokens draw from the same window budget as input, ignoring the reserve is how you fit the prompt perfectly and then run out of room mid-generation.

This pattern — measure all components, reserve for output, compare to limit — belongs in every LLM application's request-building path. It replaces hopeful guessing with a deterministic check, and it's a handful of lines. The reserve value should match your maximum expected output length plus a small safety margin.

Mistake three is expecting the model to remember across separate calls. The API is stateless: each request is processed in isolation, and the model knows only what's in that request's window. If your code doesn't resend the prior conversation, the model genuinely has no access to it — 'it forgot my name' almost always decodes to 'we didn't include the name in this call.'

The correction is a mental model shift: persistent memory is something you build, not something the model provides. Whether you resend full history, summarize it, or store and retrieve facts from a database, the memory lives in your system. The model is a stateless function from window to next token, and treating it as anything more invites bugs.

The decision tree gives you a triage procedure for the most common symptom: a wrong or forgetful answer. First ask whether all the needed text was actually inside the window for that call. If not, you have an overflow or a stateless-call problem — the fix is to resend or trim so the content is present. If the text was in the window, ask whether the key fact sat near the middle. If so, you're likely hitting lost-in-the-middle, and the fix is to move it to the edges.

Only when the content was present and well-positioned and the answer is still wrong should you conclude it's a genuine model limitation. This ordering matters because it directs you to the cheap, fixable causes first, instead of blaming the model and prematurely reaching for a bigger or different one.

Mistake four is burying the most important content in the middle of a long prompt, directly into the lost-in-the-middle dead zone established in post two. When the critical instruction, constraint, or fact is sandwiched in the center of a large context, the model is measurably more likely to overlook it, even though it's right there in the input.

The fix exploits the recall curve: lead with the task and key constraints, or restate them at the very end where attention is strong, ideally both for anything truly critical. Don't trust the model to excavate the center of a long context. This is a free reliability gain — pure prompt ordering, no model change, no extra tokens of consequence.

Mistake five shifts to the systems side, for anyone self-hosting models. The KV cache grows with both the number of concurrent users and the length of each one's context, and it lives in scarce GPU memory. Left unbounded, it leads to out-of-memory crashes under load or quietly collapses throughput as the server can fit fewer simultaneous requests. This is the production face of post three's memory formula.

The fixes are the cache-management techniques: cap the maximum context length you'll accept, use models with grouped-query attention to shrink per-token cache, quantize the cache to fewer bits, and serve with a system like vLLM that uses PagedAttention to pack cache memory without waste. Capacity planning for an LLM service is largely KV-cache planning.

This trim function is the practical answer to history overflow. It walks the conversation from newest to oldest, keeping messages while they fit within a token budget and stopping once the next message would exceed it, then restores chronological order. Keeping the most recent turns is the right default because recency usually carries the most relevant context, and it sits at the high-recall end of the window.

This is a deliberately simple strategy; production systems often layer summarization on top — compressing dropped older turns into a short recap rather than discarding them outright. But even this basic version prevents the most common overflow failure, and it makes the cost-control point from post two concrete: bounded history means bounded, predictable per-request cost.

The summary table pairs each mistake with its fix for quick recall. Treating the window as infinite is fixed by counting tokens and planning overflow. Counting words is fixed by using the actual tokenizer. Trusting automatic memory is fixed by resending or building real memory. And burying a fact in the middle is fixed by leading or ending with it.

The through-line across all four is that these are engineering problems with engineering solutions, not model deficiencies. Internalizing the mistake-to-fix mapping means that when something goes wrong in production, you reach for the right diagnosis quickly instead of flailing or blaming the model.

This final checklist is the operational distillation of the entire day. Token-count every request before sending. Reserve tokens for the output so generation doesn't run out of room. Resend or summarize history each turn because the model is stateless. Place critical content at the edges to dodge lost-in-the-middle. And cap and compress the KV cache when self-hosting to keep serving stable under load.

Five practices, each cheap to adopt, each preventing a class of failure that routinely surprises teams. Pin this list to your LLM application's request path and you've eliminated most context-related incidents before they happen.

That completes Day 65. You can now reason about context windows and the KV cache the way an engineer does: the window as finite, token-measured, position-sensitive working memory, and the cache as the causality-enabled speed mechanism with a real, plannable memory cost.

From concept to stakes to mechanism to code to mistakes, you've seen the full picture. This is the foundation that makes everything about prompt design, RAG, agents, and self-hosted serving make sense — because all of them are, at bottom, the art of managing what fits in the window and what it costs to process.