Context Windows Are the New RAM: Managing Them Well

The Advertised-vs-Effective Gap: Why a Huge Context Window Isn't What You Think

Context windows have grown roughly 30x per year since mid-2023, and the pace is not slowing. According to data from Epoch AI, the input length at which top models maintain 80% accuracy on long-context benchmarks has risen more than 250x in just nine months. Huge windows are now the norm, not a premium feature.

The scale is genuinely staggering. BenchLM's tracker shows that 25% of tracked models — 115 of 465 — now offer windows of 1M+ tokens, with a median of 256K across the board. One million tokens is enough to fit War and Peace plus an average novel in a single query. On paper, you can stuff an entire codebase, contract archive, or research library into one prompt.

Here is the problem: the paper number is not the real number. Advertised limits often exceed effective usable context by more than 99% on complex tasks. In the NoLiMa benchmark (LMU Munich and Adobe Research, presented at ICML 2025), 11 of 13 leading models dropped below 50% of their baseline scores at just 32K tokens — a fraction of what their windows claim to support. GPT-4o fell from 99.3% to 69.7%, and that is a top-tier model on a modest input.

The gap shows up in three well-documented ways, per research synthesized by Atlan:

• Top models have failed with as little as 100 tokens in context, with clear degradation by 1,000 tokens.
• Accuracy drops more than 30% when the relevant information sits in the middle of the window rather than the edges (Stanford/UC Berkeley, 2023).
• Distractor content actively misleads — a 2025 Chroma study found irrelevant material degraded performance across 18 models tested.

Anthropic's own engineering team confirms the pattern, describing context as "a finite resource with diminishing marginal returns" with a limited attention budget. As token counts grow, attention gets stretched thin and recall accuracy falls — a phenomenon the industry now calls context rot.

The trap this creates for businesses is simple and expensive: teams see "1M tokens" on a model card and assume they can dump anything into the prompt — every document, every tool description, every skill instruction — and get good results. In practice, more low-signal context makes output worse, not better. As one vendor puts it bluntly: the best way to improve your agent is to give it less context, not more — but it has to be the right less.

That is why curation is becoming its own discipline. If you are equipping agents with skills and tools, what you leave out matters as much as what you include — which is why SkillzBench is building a free directory of vetted agent skills with per-agent loadouts, so an agent only ever sees what you deliberately chose for it.

Context Rot: Why More Tokens Make Your AI Dumber and Costlier

Here's the uncomfortable math behind those million-token context windows: stuffing more into them doesn't just cost more — it actively makes your AI worse. Researchers call it context rot, and it emerges across all models, not just weaker ones.

Three mechanisms compound the damage. First, lost-in-the-middle: a Stanford/UC Berkeley study found accuracy dropped more than 30% when relevant information sat in the middle of a long context rather than at the start or end. Models pay closest attention to what's at the edges. Second, attention dilution: as token counts grow, the model's attention gets stretched thin across everything in the window, so even correctly placed information gets weaker recall. Third, distractor interference — a 2025 Chroma study found that irrelevant but plausible-looking content actively misled 18 different models, not just wasted space.

The degradation shows up at surprisingly small scales. In the NoLiMa benchmark from LMU Munich and Adobe Research, 11 of 13 LLMs dropped below 50% of their baseline scores at just 32K tokens, and GPT-4o fell from 99.3% to 69.7%. Some top models have failed with as little as 100 tokens of distractor content in context.

Then there's the cost side, which scales worse than most people assume. Transformer attention creates pairwise relationships between every token, so doubling context quadruples compute — O(n²) scaling. One vendor analysis puts 128K tokens at roughly 16,384x the compute of 1K tokens, with 30+ second latency. Your "RAM" is expensive RAM, and the bill grows quadratically.

This is why Andrej Karpathy's framing has become the standard mental model: treat the LLM as a CPU and the context window as RAM — working memory you load deliberately, like an operating system loading just the right code for the task. Anthropic frames it the same way, describing context as a finite resource with an attention budget that every token you add spends from.

What drains that budget in practice:

• Dumping 200 raw document chunks when 10 ranked results would answer the question
• Bloated tool lists, where the agent has to reason through options a human couldn't distinguish either
• Accumulated chat history — chatbots begin contradicting earlier instructions after roughly 20 turns
• Edge-case examples stuffed into prompts instead of a few diverse, canonical ones

The practical takeaway for anyone building with agents: curation beats capacity. Anthropic's guidance is to find the smallest possible set of high-signal tokens for each task, and the emerging discipline of context engineering treats that curation as a core engineering skill, not an afterthought. For agent builders, that includes curating which skills and tools the agent can even see — which is exactly the problem SkillzBench is building for, with vetted agent skills and per-agent loadouts that keep the window lean. Waiting for a bigger window won't solve it; as Anthropic puts it, windows of all sizes will face the same relevance and dilution problems.

Context Engineering: Rank, Retrieve Just-in-Time, Never Stuff

Prompt engineering asked the model a better question. Context engineering decides what the model gets to see in the first place — and that shift is now the core skill in agent work. Anthropic's engineering guidance calls it finding "the smallest possible set of high-signal tokens" that maximizes the odds of the outcome you want, a discipline of curation rather than accumulation.

The patterns that work follow an operating-system logic: hold lightweight identifiers, load content at runtime. Anthropic's just-in-time retrieval approach keeps file paths, queries, and links in context while Claude Code fetches actual data only when needed. Progressive disclosure exposes detail on demand, and hybrid strategies pair upfront files like CLAUDE.md with runtime exploration via glob and grep.

Ranking beats stuffing. Shaped.ai's vendor analysis claims 10 ranked results at roughly 3,050 tokens cost about $0.015 per query versus 200 raw chunks at ~50,550 tokens for ~$0.16 — a claimed 10x reduction, though these are vendor figures, not independent measurements. The underlying math is less disputed: attention scales quadratically, so doubling context quadruples compute.

Long context versus retrieval isn't settled. An arXiv survey finds long-context models generally beat RAG on QA benchmarks, especially Wikipedia-style questions, while summarization-based retrieval performs comparably. Long context wins for self-contained information; RAG wins for fragmented information like dialogue. The authors flag context relevance as an overlooked gap, and findings across the field contradict each other.

For long-horizon tasks, three techniques carry the load:

• Compaction — summarizing history while preserving architectural decisions, unresolved bugs, and recently accessed files
• Structured note-taking — externalizing state the window can't hold
• Multi-agent architectures — splitting work across narrower windows
• Lean tool sets — if a human engineer can't say which tool applies, an agent can't either

Curating which skills and tools an agent sees is the same discipline applied to capabilities. SkillzBench is building a free directory of vetted agent skills plus a free MCP connector with per-agent loadouts, so an agent only sees what you choose — join the waitlist to get access at launch.

Your Context Management Playbook: Compaction, Notes, and Lean Tool Sets

When an agent's task stretches across hours instead of minutes, the context window fills up fast — and as we've seen, a full window is a degraded window. Anthropic's engineering guidance names three techniques for exactly this problem, and they're practical enough to adopt today.

The first is compaction: when context nears its limit, summarize the conversation and restart with the summary. The key is what survives the cut. Claude Code, for example, retains architectural decisions, unresolved bugs, implementation details, and the five most recently accessed files — the information the agent needs to keep working, not the noise it generated getting there. If your agent's summary drops the "why" behind a decision, it will re-litigate that decision an hour later.

The second is structured note-taking. Instead of keeping everything in the window, the agent persists key facts to a document or memory file outside the window and retrieves them when needed. This is the same just-in-time pattern Claude Code uses more broadly: hold lightweight identifiers — file paths, queries, links — and load the full data only at runtime.

The third is multi-agent architectures: a lead agent decomposes the task and delegates to subagents, each with its own clean, focused window. Every subagent works in parallel with fresh attention, and the lead sees only condensed results.

Then there's the tool problem. Anthropic's rule is blunt: "If a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better." Bloated tool sets and skill stacks dilute the model's attention just like bloated documents do — and with attention cost scaling quadratically, every low-signal token you load makes the high-signal ones harder to find. Their advice: "Do the simplest thing that works."

That's where an audit mindset pays off. Before your next agent deployment, ask:

• What exactly loads into the window at startup — and is each item earning its tokens?
• Which tools and skills does the agent actually see, and could a human on your team say which one to use in each situation?
• Does your compaction or summarization step preserve decisions, unresolved bugs, and recently touched files?
• Are you loading full documents when lightweight identifiers plus runtime retrieval would do?

Curation at this level is hard to do by hand, which is why SkillzBench is building per-agent loadouts into its free MCP connector — you toggle which skills and tools each agent sees, and the agent only gets what you chose. If a skill doesn't lift performance, it doesn't belong in the window. That's the discipline, in one line: give your agent less context — but the right less.

Curate What Your Agent Sees: The SkillzBench Approach to a Clean Context

If the context window is your agent's RAM, then every skill you load is a program running in that memory — and most agents are running far too many. Anthropic's guidance on effective context engineering is blunt: find the smallest possible set of high-signal tokens, and keep tool sets lean, because "if a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better."

Agent skills make this harder, not easier. Each SKILL.md package you add brings its own instructions and tools into the window, consuming attention budget whether or not the current task needs them. With attention cost scaling quadratically — doubling context tokens roughly quadruples compute — a bloated loadout isn't just noisy, it's expensive. And bloated loadouts can be dangerous: published research found that roughly 26% of agent skills carry at least one vulnerability, so every unvetted skill is a potential liability sitting directly in your agent's line of sight.

The curation discipline Anthropic recommends — just-in-time retrieval, lean tool selection, deliberate loading — implies a layer most teams don't have: a way to see what skills exist, judge which ones are safe and effective, and control exactly what each agent can access. That layer is what SkillzBench is building. Still in development and open for a waitlist, the free platform is designed around five pillars:

• Discover and verify — continuously crawl skills from GitHub, pin every version to a commit SHA, and run license checks and security scans, so the ~26% vulnerability rate doesn't become your problem.
• Benchmark before you trust — test every skill in a sandbox with the same model, with and without the skill, and publish the Lift in percentage points, kept separate from trust scores.
• Equip per agent — one free MCP connector with per-agent loadouts: you toggle skills and tools on or off, and the agent only sees what you deliberately chose.
• Stay current — track upstream changes, re-scan, re-benchmark, and notify you when a skill you rely on shifts under you.

This maps directly to the operating-system metaphor that has become the field's standard mental model. In Karpathy's framing, echoed by Addy Osmani, the LLM is the CPU and the context window is RAM — the engineer's job is to load "just the right code and data for the task." Per-agent loadouts are that loading discipline made concrete: nothing enters the window by default, and nothing enters without a reason.

It also answers the warning that keeps recurring across the research. Anthropic notes that "context windows of all sizes will be subject to context pollution and information relevance concerns" — waiting for a bigger window won't fix a curation problem. A research synthesis puts it another way: the model you pick matters less than the governance layer feeding it context.

For business owners and agent builders alike, the practical takeaway is the same. Audit what your agents currently load, cut anything low-signal, and treat skill selection as a security and performance decision, not a convenience. When SkillzBench launches, its directory, security scans, benchmarks, and MCP connector will all be free — join the waitlist to get early access and start curating your agents' loadouts from day one.