---
type: claim
domain: ai-alignment
description: "Russell's Off-Switch Game provides a formal game-theoretic proof that objective uncertainty yields corrigible behavior — the opposite of Yudkowsky's framing where corrigibility must be engineered against instrumental interests"
confidence: likely
source: "Hadfield-Menell, Dragan, Abbeel, Russell, 'The Off-Switch Game' (IJCAI 2017); Russell, 'Human Compatible: AI and the Problem of Control' (Viking, 2019)"
created: 2026-04-05
challenges:
- corrigibility is at cross-purposes with effectiveness because deception is a convergent free strategy while corrigibility must be engineered against instrumental interests
related:
- capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability
- intelligence and goals are orthogonal so a superintelligence can be maximally competent while pursuing arbitrary or destructive ends
- learning human values from observed behavior through inverse reinforcement learning is structurally safer than specifying objectives directly because the agent maintains uncertainty about what humans actually want
reweave_edges:
- learning human values from observed behavior through inverse reinforcement learning is structurally safer than specifying objectives directly because the agent maintains uncertainty about what humans actually want|related|2026-04-06
sourced_from:
- inbox/archive/bostrom-russell-drexler-alignment-foundations.md
---

# An AI agent that is uncertain about its objectives will defer to human shutdown commands because corrigibility emerges from value uncertainty not from engineering against instrumental interests

Russell and collaborators (IJCAI 2017) prove a result that directly challenges Yudkowsky's framing of the corrigibility problem. In the Off-Switch Game, an agent that is uncertain about its utility function will rationally defer to a human pressing the off-switch. The mechanism: if the agent isn't sure what the human wants, the human's decision to shut it down is informative — it signals the agent was doing something wrong. A utility-maximizing agent that accounts for this uncertainty will prefer being shut down (and thereby learning something about the true objective) over continuing an action that might be misaligned.

The formal result: the more certain the agent is about its objectives, the more it resists shutdown. At 100% certainty, the agent is maximally resistant — this is Yudkowsky's corrigibility problem. At meaningful uncertainty, corrigibility emerges naturally from rational self-interest. The agent doesn't need to be engineered to accept shutdown; it needs to be engineered to maintain uncertainty about what humans actually want.

This is a fundamentally different approach from [[corrigibility is at cross-purposes with effectiveness because deception is a convergent free strategy while corrigibility must be engineered against instrumental interests]]. Yudkowsky's claim: corrigibility fights against instrumental convergence and must be imposed from outside. Russell's claim: corrigibility is instrumentally convergent *given the right epistemic state*. The disagreement is not about instrumental convergence itself but about whether the right architectural choice (maintaining value uncertainty) can make corrigibility the instrumentally rational strategy.

Russell extends this in *Human Compatible* (2019) with three principles of beneficial AI: (1) the machine's only objective is to maximize the realization of human preferences, (2) the machine is initially uncertain about what those preferences are, (3) the ultimate source of information about human preferences is human behavior. Together these define "assistance games" (formalized as Cooperative Inverse Reinforcement Learning in Hadfield-Menell et al., NeurIPS 2016) — the agent and human are cooperative players where the agent learns the human's reward function through observation rather than having it specified directly.

The assistance game framework makes a structural prediction: an agent designed this way has a positive incentive to be corrected, because correction provides information. This contrasts with the standard RL paradigm where the agent has a fixed reward function and shutdown is always costly (it prevents future reward accumulation).

## Challenges

- The proof assumes the human is approximately rational and that human actions are informative about the true reward. If the human is systematically irrational, manipulated, or provides noisy signals, the framework's corrigibility guarantee degrades. In practice, human feedback is noisy enough that agents may learn to discount correction signals.
- Maintaining genuine uncertainty at superhuman capability levels may be impossible. [[capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability]] — a sufficiently capable agent may resolve its uncertainty about human values and then resist shutdown for the same instrumental reasons Yudkowsky describes.
- The framework addresses corrigibility for a single agent learning from a single human. Multi-principal settings (many humans with conflicting preferences, many agents with different uncertainty levels) are formally harder and less well-characterized.
- Current training methods (RLHF, DPO) don't implement Russell's framework. They optimize for a fixed reward model, not for maintaining uncertainty. The gap between the theoretical framework and deployed systems remains large.
- Russell's proof operates in an idealized game-theoretic setting. Whether gradient-descent-trained neural networks actually develop the kind of principled uncertainty reasoning the framework requires is an empirical question without strong evidence either way.