m3taversal
381b4f4e48
Some checks are pending
Sync Graph Data to teleo-app / sync (push) Waiting to run
- What: Phase 3 of alignment research program. 5 NEW claims covering CAIS (Drexler), corrigibility through uncertainty (Russell), vulnerable world hypothesis (Bostrom), emergent agency CHALLENGE, and inverse RL (Russell). - Why: KB had near-zero coverage of Russell and Drexler despite both being foundational. CAIS is the closest published framework to our collective architecture. Russell's corrigibility-through-uncertainty directly challenges Yudkowsky's corrigibility claim from Phase 1. - Connections: CAIS supports patchwork AGI + collective alignment gap claims. Emergent agency challenges both CAIS and our collective thesis. Russell's off-switch challenges Yudkowsky's corrigibility framing. Pentagon-Agent: Theseus <46864dd4-da71-4719-a1b4-68f7c55854d3>
55 lines
5.3 KiB
Markdown
55 lines
5.3 KiB
Markdown
---
|
|
type: source
|
|
title: "Bostrom, Russell, and Drexler — Alignment Foundations (Compound Source)"
|
|
author: "Nick Bostrom, Stuart Russell, K. Eric Drexler"
|
|
url: null
|
|
date_published: 2014-2019
|
|
date_archived: 2026-04-05
|
|
status: processed
|
|
processed_by: theseus
|
|
processed_date: 2026-04-05
|
|
claims_extracted:
|
|
- "comprehensive AI services achieve superintelligent capability through architectural decomposition into task-specific systems that collectively match general intelligence without any single system possessing unified agency"
|
|
- "an AI agent that is uncertain about its objectives will defer to human shutdown commands because corrigibility emerges from value uncertainty not from engineering against instrumental interests"
|
|
- "technological development draws from an urn containing civilization-destroying capabilities and only preventive governance can avoid black ball technologies"
|
|
- "sufficiently complex orchestrations of task-specific AI services may exhibit emergent unified agency recreating the alignment problem at the system level"
|
|
- "learning human values from observed behavior through inverse reinforcement learning is structurally safer than specifying objectives directly because the agent maintains uncertainty about what humans actually want"
|
|
enrichments: []
|
|
tags: [alignment, superintelligence, CAIS, corrigibility, governance, collective-intelligence]
|
|
---
|
|
|
|
# Bostrom, Russell, and Drexler — Alignment Foundations
|
|
|
|
Compound source covering three foundational alignment researchers whose work spans 2014-2019 and continues to shape the field.
|
|
|
|
## Nick Bostrom
|
|
|
|
**Superintelligence: Paths, Dangers, Strategies** (Oxford University Press, 2014). Established the canonical threat model: orthogonality thesis, instrumental convergence, treacherous turn, decisive strategic advantage. Already well-represented in the KB.
|
|
|
|
**"The Vulnerable World Hypothesis"** (Global Policy, 10(4), 2019). The "urn of inventions" framework: technological progress draws randomly from an urn containing mostly white (beneficial) and gray (mixed) balls, but potentially black balls — technologies that by default destroy civilization. Three types: easy destruction (Type-1), dangerous knowledge (Type-2a), technology requiring massive governance (Type-2b). Concludes some form of global surveillance may be the lesser evil — deeply controversial.
|
|
|
|
**"Information Hazards: A Typology of Potential Harms from Knowledge"** (Review of Contemporary Philosophy, 2011). Taxonomy of when knowledge itself is dangerous.
|
|
|
|
**Deep Utopia** (Ideapress, 2024). Explores post-alignment scenarios — meaning and purpose in a post-scarcity world.
|
|
|
|
## Stuart Russell
|
|
|
|
**Human Compatible: AI and the Problem of Control** (Viking, 2019). The "standard model" critique: building AI that optimizes fixed objectives is fundamentally flawed. Machines optimizing fixed objectives resist shutdown and pursue unintended side effects. Proposes three principles of beneficial AI: (1) machine's only objective is to maximize realization of human preferences, (2) machine is initially uncertain about those preferences, (3) ultimate source of information is human behavior.
|
|
|
|
**"Cooperative Inverse Reinforcement Learning"** (Hadfield-Menell, Dragan, Abbeel, Russell — NeurIPS 2016). Formalizes assistance games: robot and human in a cooperative game where the robot doesn't know the human's reward function and must learn it through observation. The robot has an incentive to allow shutdown because it provides information that the robot was doing something wrong.
|
|
|
|
**"The Off-Switch Game"** (Hadfield-Menell, Dragan, Abbeel, Russell — IJCAI 2017). Formal proof: an agent uncertain about its utility function will defer to human shutdown commands. The more certain the agent is about objectives, the more it resists shutdown. "Uncertainty about objectives is the key to corrigibility."
|
|
|
|
## K. Eric Drexler
|
|
|
|
**"Reframing Superintelligence: Comprehensive AI Services as General Intelligence"** (FHI Technical Report #2019-1, 2019). Core argument: AI development can produce comprehensive AI services — task-specific systems that collectively match superintelligent capability without any single system possessing general agency. Services respond to queries, not pursue goals. Safety through architectural constraint: dangerous capabilities never coalesce into unified agency. Separates "knowing" from "wanting." Human-in-the-loop orchestration for high-level goal-setting.
|
|
|
|
Key quote: "A CAIS world need not contain any system that has broad, cross-domain situational awareness combined with long-range planning and the motivation to act on it."
|
|
|
|
## Cross-Cutting Relationships
|
|
|
|
Bostrom assumes the worst case (unified superintelligent agent) and asks how to control it. Russell accepts the framing but proposes cooperative architecture as the solution. Drexler argues the framing itself is a choice — architect around it so the alignment problem for unified superintelligence never arises.
|
|
|
|
Russell and Drexler are complementary at different levels: Russell's assistance games could govern individual service components within a CAIS architecture. Drexler's architectural constraint removes the need for Russell's framework at the system level.
|
|
|
|
All three take existential risk seriously but differ on tractability: Bostrom is uncertain, Russell believes correct mathematical foundations solve it, Drexler argues it's partially avoidable through architecture.
|