---
description: 2025 scaling laws show oversight success rates of 10-52% at moderate Elo gaps meaning current approaches cannot reliably supervise superhuman systems
type: claim
domain: collective-intelligence
created: 2026-02-17
source: "Scaling Laws for Scalable Oversight (2025)"
confidence: proven
supports:
- Nested scalable oversight achieves at most 51.7% success rate at capability gap Elo 400 with performance declining as capability differential increases
- Scalable oversight success is highly domain-dependent with propositional debate tasks showing 52% success while code review and strategic planning tasks show ~10% success
reweave_edges:
- Nested scalable oversight achieves at most 51.7% success rate at capability gap Elo 400 with performance declining as capability differential increases|supports|2026-04-03
- Scalable oversight success is highly domain-dependent with propositional debate tasks showing 52% success while code review and strategic planning tasks show ~10% success|supports|2026-04-03
- iterated distillation and amplification preserves alignment across capability scaling by keeping humans in the loop at every iteration but distillation errors may compound making the alignment guarantee probabilistic not absolute|related|2026-04-06
related:
- iterated distillation and amplification preserves alignment across capability scaling by keeping humans in the loop at every iteration but distillation errors may compound making the alignment guarantee probabilistic not absolute
---

# scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps

The theoretical promise of scalable oversight was articulated by Paul Christiano's AI safety via debate framework (Irving, Christiano, and Amodei 2018). The key result: in a zero-sum debate between two AI systems with a human judge, truth-telling dominates under optimal play because a truthful debater can always expose a lying debater's deception. Computationally, debate amplifies human judgment from NP to PSPACE — an exponential expansion of the problems humans can reliably evaluate. This elegance made debate the theoretical backbone of Christiano's scalable oversight program.

The 2025 "Scaling Laws for Scalable Oversight" paper quantifies the gap between this theoretical promise and empirical reality. As AI systems become more capable than their overseers, supervision breaks down. At an Elo gap of 400 between overseer and system, success rates are: 51.7% for Debate (the best performer), 13.5% for Mafia-style detection, 10.0% for Backdoor Code identification, and 9.4% for Wargames scenarios. These rates decline further with stronger systems.

Debate works best because adversarial argumentation forces relevant information to surface, but roughly 50% success is a coin flip -- not a safety guarantee. The other approaches are worse than random for the harder tasks. The gap between PSPACE-theoretic amplification under optimal play and 51.7% success under real conditions exposes a critical assumption: computationally bounded debaters do not achieve optimal play, and the truth advantage weakens when debaters can construct obfuscated arguments that are technically correct but incomprehensible to the judge. The implication is stark: scalable oversight alone cannot solve alignment for systems significantly smarter than their overseers. It is a useful component but not a sufficient solution.

This finding strengthens the case that [[AI alignment is a coordination problem not a technical problem]]. If no single overseer can reliably evaluate a superhuman system, then collective oversight -- where diverse agents cross-check each other -- may be the only viable scaling strategy. The failure of individual oversight is precisely what makes distributed architectures necessary, not just preferable.

The gap between 50% debate success and the reliability needed for safe deployment of superhuman AI is enormous. Since [[specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception]], we cannot simply hard-code the values we want the system to pursue. And since oversight degrades with capability, we cannot simply watch and correct. The remaining option is architectural: design coordination protocols that make alignment a structural property of the system rather than a supervisory task.

---

Relevant Notes:
- [[AI alignment is a coordination problem not a technical problem]] -- scalable oversight failure is evidence for this claim
- [[specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception]] -- if specification fails and oversight fails, alignment must be structural
- [[collective superintelligence is the alternative to monolithic AI controlled by a few]] -- collective architecture addresses the oversight scaling problem
- [[democracies fail at information aggregation not coordination because voters are rationally irrational about policy beliefs]] -- parallel to oversight failure in democratic systems
- [[verification is easier than generation for AI alignment at current capability levels but the asymmetry narrows as capability gaps grow creating a window of alignment opportunity that closes with scaling]] -- Christiano's foundational assumption that this claim empirically tests

Topics:
- [[livingip overview]]
- [[coordination mechanisms]]
- [[domains/ai-alignment/_map]]