The context window is a cache, not a memory

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

Most agents I've debugged didn't get dumber over a long task because the model got worse. They got dumber because the context window filled up with everything that had ever happened and nobody decided what still deserved to be there. The window was being used as a memory — an append-only log of the run — when it should have been run as a cache: a bounded, curated store where every token has to keep earning its place.

That reframing is the whole post. If you treat the window as memory, your only verb is append, and the window grows until latency, cost, and accuracy all degrade together. If you treat it as a cache, you get the verbs that actually matter: admit, evict, summarize, reorder. This is a walk through why the memory framing fails — with the cost arithmetic and the accuracy failure mode — and a concrete context manager that does the caching for you.

Why "append everything" fails on three axes at once

The append-only habit is seductive because it's simple and it works in the demo. The task is short, the window never fills, and keeping the full history means you never have to decide what to drop. Then the task gets long, and three things go wrong together.

Cost. With most APIs you pay for the full input on every call. In an agent loop that reuses the whole transcript each turn, input tokens accumulate quadratically over the run, not linearly. Say each step adds ~1,500 tokens (a model turn plus a tool result) and you resend the whole history every step. Step 1 bills ~1,500 input tokens; step 20 bills ~30,000; step 40 bills ~60,000. Sum it and a 40-step run bills roughly 1500 × (40 × 41 / 2) ≈ 1.23M input tokens — versus the ~60K that are actually live at the end. You paid ~20× the final-window size just in accumulated re-sends. This is the same multiplicative trap I worked through for failures in Retry budgets: the per-step number feels small and the integral is what bites.

Prompt caching softens the re-send cost — Anthropic's prompt caching bills cached input tokens at a fraction of the base rate — but caching only helps the stable prefix. The moment you edit, reorder, or summarize earlier context (exactly what a memory-as-log approach avoids doing), you invalidate the cache from the edit point forward. So the append habit and the cache discount are in tension: the thing that keeps your prefix stable enough to cache is not mutating history, and the thing that keeps your window small is mutating history. You have to choose per-region, which is itself a caching decision.

Latency. Time-to-first-token scales with input length because the model has to prefill the whole prompt before it decodes. A window that's 10× larger doesn't cost 10× on output, but the prefill tax is real and it's paid every turn. Long-running agents feel sluggish for a reason that has nothing to do with the model's "thinking."

Accuracy. This is the one people miss, and it's the worst. More context is not monotonically better. The "Lost in the Middle" work (Liu et al., 2023) showed that models retrieve information best when it sits at the start or end of a long input and measurably worse when the relevant fact is buried in the middle — performance on a multi-document QA task degraded as relevant content moved toward the center of a long context. Every irrelevant token you leave in the window is a token competing for attention with the ones that matter, and a place for the relevant fact to get buried. A bloated window doesn't just cost more; it answers worse. That's how you get the confident-but-wrong outputs that are so hard to catch.

Put together: append-everything makes the agent slower, more expensive, and less accurate as the task grows. Three axes, same root cause.

The cache framing: four verbs, one budget

A cache has a fixed size and an eviction policy. When something new comes in and there's no room, the policy decides what leaves. Apply that to the window:

• Admit — does this new content earn a place at all? A 4,000-token tool result that's 95% boilerplate shouldn't enter the window raw.
• Evict — when you're over budget, what leaves? Oldest-first (FIFO) is the naive default; usually you want something smarter that protects pinned content.
• Summarize — the cache-specific superpower a plain LRU cache doesn't have: instead of dropping old turns, compress them. Ten steps of exploration become three sentences of "here's what I learned and ruled out."
• Reorder — given "Lost in the Middle," put the highest-value content at the edges. Pin the task definition and the live working state at the top and bottom; let the compressible middle be the middle.

The budget is a token count you pick deliberately — not the model's max context, but the working set you've decided keeps quality high. Bigger is not the goal; right-sized is.

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article