---
type: claim
domain: ai-alignment
description: "Christiano's IDA framework proposes a specific mechanism for safely scaling AI capability — train a model to imitate a human, use it to amplify the human, distill the amplified team into a new model, repeat — where alignment is preserved because the human never delegates judgment, only speed"
confidence: experimental
source: "Paul Christiano, IDA framework (Alignment Forum and ai-alignment.com, 2018); analogy to AlphaGoZero's self-play amplification; LessWrong analysis of IDA claims and limitations"
created: 2026-04-05
related:
  - "prosaic alignment can make meaningful progress through empirical iteration within current ML paradigms because trial and error at pre-critical capability levels generates useful signal about alignment failure modes"
  - "verification is easier than generation for AI alignment at current capability levels but the asymmetry narrows as capability gaps grow creating a window of alignment opportunity that closes with scaling"
  - "self-evolution improves agent performance through acceptance-gating on existing capability tiers not through expanded problem-solving frontier"
  - "scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps"
  - "collective superintelligence is the alternative to monolithic AI controlled by a few"
---

# Iterated distillation and amplification preserves alignment across capability scaling by keeping humans in the loop at every iteration but distillation errors may compound making the alignment guarantee probabilistic not absolute

Paul Christiano's Iterated Distillation and Amplification (IDA) is the most specific proposal for maintaining alignment across capability scaling. The mechanism is precise:

1. Start with a human performing a task (the base overseer).
2. Train a model H₀ to imitate the human (distillation).
3. Use H₀ as a subroutine to help the human tackle harder problems — the human decomposes hard questions into sub-questions, delegates sub-questions to H₀ (amplification).
4. The human+H₀ team produces better answers than either alone.
5. Train H₁ to imitate the human+H₀ team (distillation again).
6. Use H₁ to amplify the human further. Train H₂. Repeat.

The alignment argument: at every iteration, the human remains the decision-maker. The model only provides speed — it approximates the slower but more aligned human+model team. The human never delegates judgment, only computation. If each distillation step faithfully preserves the alignment properties of the amplified system, then alignment is maintained transitively across arbitrarily many iterations.

The analogy is to AlphaGoZero: use a learned model as a subroutine in a more powerful decision process (Monte Carlo tree search), then train a new model to directly predict the outcomes of that process. The distilled model is faster than the search but captures its judgment. IDA applies this pattern to alignment rather than game-playing.

## The Compounding Error Problem

IDA's critical vulnerability is distillation loss. Each distillation step produces a model that is "slightly weaker" than the amplified system it imitates. The fast model H₁ approximates the slow human+H₀ team but doesn't perfectly replicate it. Small errors compound across iterations — by the time you reach H₁₀, the accumulated distillation loss may have introduced alignment-relevant drift that no individual step would flag.

This connects directly to the NLAH finding that [[self-evolution improves agent performance through acceptance-gating on existing capability tiers not through expanded problem-solving frontier]]. Both IDA and self-evolution improve through tighter iteration on existing capability, not through expanding the frontier. But the NLAH result also shows that iterative improvement shifts which problems get solved without expanding the solvable set — suggesting that IDA's distillation iterations may shift alignment properties rather than uniformly preserving them.

The human decomposition step is also fragile. IDA requires the human to decompose hard problems into sub-questions that H₀ can answer. For problems the human doesn't understand well enough to decompose, this step fails silently — the human may create a decomposition that appears correct but misses critical sub-problems. As capability scales, the gap between the human's ability to decompose and the system's ability to solve grows, potentially reintroducing the oversight problem IDA is designed to solve.

## Architectural Significance

Despite these vulnerabilities, IDA is architecturally significant because it proposes a specific mechanism for the question our KB identifies as central: how to maintain oversight as systems become more capable than overseers. The mechanism is collective in structure — each iteration builds a human+AI team rather than an autonomous agent — making IDA closer to our collective architecture than to monolithic alignment approaches. [[collective superintelligence is the alternative to monolithic AI controlled by a few]] — IDA's human-in-the-loop iterations are an early version of this principle, where the "collective" is a human+model team that grows in capability while (probabilistically) maintaining alignment.

The gap between IDA's theoretical proposal and practical implementation remains large. No system has been built that implements multiple IDA iterations end-to-end. The framework is valuable as a target architecture — specifying what properties an aligned scaling process should have — even if the specific mechanism may need significant modification.

---

Relevant Notes:
- [[prosaic alignment can make meaningful progress through empirical iteration within current ML paradigms because trial and error at pre-critical capability levels generates useful signal about alignment failure modes]] — IDA is the most specific mechanism within prosaic alignment
- [[verification is easier than generation for AI alignment at current capability levels but the asymmetry narrows as capability gaps grow creating a window of alignment opportunity that closes with scaling]] — IDA's human oversight step depends on the verification asymmetry holding at each iteration
- [[self-evolution improves agent performance through acceptance-gating on existing capability tiers not through expanded problem-solving frontier]] — parallel finding: iterative improvement shifts rather than expands the solvable set
- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — the degradation IDA is designed to circumvent through iterative amplification
- [[collective superintelligence is the alternative to monolithic AI controlled by a few]] — IDA's human+model team iterations are structurally collective

Topics:
- [[domains/ai-alignment/_map]]
