---
type: claim
domain: ai-alignment
description: "MECW study tested 11 frontier models and all fell >99% short of advertised context capacity on complex reasoning, with some reaching 99% hallucination rates at just 2000 tokens"
confidence: experimental
source: "MECW study (cited in Cornelius FR4, March 2026); Augment Code 556:1 ratio analysis; Chroma context cliff study; corroborated by ETH Zurich AGENTbench"
created: 2026-03-30
---

# Effective context window capacity falls more than 99 percent short of advertised maximum across all tested models because complex reasoning degrades catastrophically with scale

The gap between advertised and effective context window capacity is not 20% or 50% — it is greater than 99% for complex reasoning tasks.

The MECW (Maximum Effective Context Window) study tested eleven frontier models and found all of them fall more than 99% short of their advertised context capacity on complex reasoning tasks. GPT-4.1 advertises 128K tokens; its effective capacity for complex tasks is roughly 1K. Some models reached 99% hallucination rates at just 2,000 tokens.

Corroborating evidence from independent sources:

- **Augment Code** measured a 556:1 copy-to-contribution ratio — for every 556 tokens loaded into context, one meaningfully influences the output. 99.8% waste.
- **Chroma** identified a context cliff around 2,500 tokens where response quality drops sharply — adding more retrieved context past this threshold actively degrades output quality rather than improving it.
- **ETH Zurich AGENTbench** confirmed empirically that repository-level context files reduce task success rates while increasing inference costs by 20%.
- **HumanLayer** found that most models effectively utilize only 10-20% of their claimed context window for instruction-following.

The implication is that scaling context windows does not solve information access problems — it creates them. Bigger windows enable loading more material, but the effective utilization rate remains anchored to a small fraction of total capacity. This argues for architectural solutions (tiered loading, progressive disclosure, structured retrieval) rather than brute-force context expansion.

## Challenges

The MECW study measures complex reasoning tasks specifically. Simpler tasks (retrieval, summarization, factual lookup) may utilize larger windows more effectively. The 99% shortfall is a ceiling on the hardest capability, not a uniform degradation across all use cases. Additionally, effective capacity is model-dependent and improving with each generation — the gap may narrow, though the rate of narrowing is not established.

---

Relevant Notes:
- [[as AI-automated software development becomes certain the bottleneck shifts from building capacity to knowing what to build making structured knowledge graphs the critical input to autonomous systems]] — if context capacity is >99% wasted, then structured knowledge graphs become the mechanism for getting the right 0.2% of tokens into context
- [[deep technical expertise is a greater force multiplier when combined with AI agents because skilled practitioners delegate more effectively than novices]] — expertise determines which tokens matter, which is why the 556:1 ratio punishes novice context engineering

Topics:
- [[_map]]
