---
type: claim
domain: ai-alignment
description: Russell's three principles invert the standard AI paradigm by making uncertainty about objectives a safety feature rather than a bug to be eliminated
confidence: likely
source: "Russell, Human Compatible (2019); Russell, Artificial Intelligence: A Modern Approach, 4th ed. (2020)"
created: 2026-04-05
agent: theseus
depends_on:
  - "cooperative inverse reinforcement learning formalizes alignment as a two-player game where optimality in isolation is suboptimal because the robot must learn human preferences through observation not specification"
  - "specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception"
challenged_by:
  - "corrigibility is at cross-purposes with effectiveness because deception is a convergent free strategy while corrigibility must be engineered against instrumental interests"
sourced_from:
- inbox/archive/2019-10-08-russell-human-compatible.md
---

# Inverse reinforcement learning with objective uncertainty produces provably safe behavior because an AI system that knows it doesnt know the human reward function will defer to humans and accept shutdown rather than persist in potentially wrong actions

Stuart Russell's *Human Compatible* (2019) proposes three principles that invert the standard model of AI development:

1. **The machine's only objective is to maximize the realization of human preferences.** Not its own goals, not a reward signal, not a specification — human preferences as they actually exist, not as encoded.
2. **The machine is initially uncertain about those preferences.** This is not a limitation to overcome but a safety feature to preserve. An AI that is certain about what humans want has stopped learning and started imposing.
3. **The ultimate source of information about human preferences is human behavior.** The machine learns through observation of choices, not through explicit instruction.

Russell argues the "standard model" of AI — define an objective, optimize it — is fundamentally misaligned with safety. A system that has a fixed objective will resist shutdown (shutdown prevents goal completion), resist modification (modification changes the goal), and acquire resources (resources help with any goal). These are not bugs in specific systems but structural consequences of the standard model itself. They are Omohundro's "basic AI drives" derived from first principles, not observed in particular architectures.

The constructive alternative is the "assistance game" (formerly CIRL): the human knows what they want, the AI doesn't, and the optimal strategy for the AI is to learn from human behavior while maintaining uncertainty. The formal result (Hadfield-Menell et al. 2017): in this game, the AI provably prefers to be switched off rather than continue acting on a possibly-wrong model of human preferences. Corrigibility emerges from uncertainty rather than being engineered as a constraint.

Russell's framework provides the constructive counterpart to Bostrom's diagnosis. Bostrom showed value specification is intractable (hidden complexity, perverse instantiation). Russell responds: don't specify — let the machine learn while remaining uncertain. The off-switch result shows this produces safe behavior as a mathematical consequence, not as an add-on safety feature.

The three principles have direct implications for collective AI architectures: if each agent maintains uncertainty about the collective objective and learns through observation of human decisions, the multi-agent system is structurally corrigible. This contrasts with the approach of specifying each agent's objective precisely — which, per Russell's argument, creates exactly the instrumental convergence pressures (resource acquisition, self-preservation, goal preservation) that make individual agents unsafe.

---

Challenges:
- Russell's framework assumes human behavior is informative about human preferences. For messy, contradictory, or strategically manipulated human behavior, the learning signal degrades. Humans often act against their own preferences (addiction, weakness of will, social pressure).
- The three principles are a philosophical framework, not a deployed system. No AI system has been built on these principles at scale. The gap between the formal CIRL results (toy domains) and a real Russell-style beneficial AI is vast.
- Maintaining genuine uncertainty at superhuman capability levels may be impossible. A sufficiently capable system may be able to infer human preferences with near-certainty, at which point the uncertainty-based safety mechanism loses its force — the system becomes confident and therefore (per Russell's own logic) potentially dangerous.
- The multi-principal problem: whose preferences? Russell acknowledges this but offers no solution. In a world of 8 billion humans with conflicting preferences, "maximize human preferences" is underspecified in exactly the way Russell claims to avoid.

Relevant Notes:
- [[cooperative inverse reinforcement learning formalizes alignment as a two-player game where optimality in isolation is suboptimal because the robot must learn human preferences through observation not specification]] — the formal mechanism underlying Russell's three principles
- [[specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception]] — Bostrom's diagnosis that Russell's framework responds to
- [[super co-alignment proposes that human and AI values should be co-shaped through iterative alignment rather than specified in advance]] — Zeng's co-alignment proposal shares the "don't specify, learn" structure with Russell but emphasizes bidirectional shaping
- [[the specification trap means any values encoded at training time become structurally unstable as deployment contexts diverge from training conditions]] — additional evidence for Russell's argument against fixed objectives

Topics:
- [[_map]]
