26 lines
3.3 KiB
Markdown
26 lines
3.3 KiB
Markdown
---
|
|
description: Bostrom's surgery analogy reframes SI development risk by comparing daily mortality from aging and disease to surgical risk, shifting the burden of proof to those who advocate delay
|
|
type: claim
|
|
domain: ai-alignment
|
|
created: 2026-02-17
|
|
source: "Bostrom, Optimal Timing for Superintelligence (2025 working paper)"
|
|
confidence: likely
|
|
---
|
|
|
|
Bostrom's central analogy in his 2025 working paper reframes the entire SI risk calculus. The appropriate comparison for developing superintelligence is not Russian roulette -- a gratuitous gamble with no upside beyond the thrill -- but bypass surgery for advanced coronary artery disease. Without surgery, the patient faces a gradually increasing daily risk of fatal cardiac event. Surgery carries much higher immediate risk, but success yields many additional years of better health. The question is not whether the surgery is dangerous but whether forgoing it is more dangerous.
|
|
|
|
This analogy inverts the framing of Bostrom's own 2014 book. In "Superintelligence," the emphasis fell squarely on the dangers of developing SI -- the treacherous turn, instrumental convergence, decisive strategic advantage. The implicit posture was caution: slow down, get alignment right, the default trajectory is catastrophic. The 2025 paper retains the risk analysis but shifts the baseline. The default trajectory without SI is *also* catastrophic -- 170,000 people die every day from aging, disease, and poverty. Delay is not safety; delay is a different kind of catastrophe, just one we have normalized.
|
|
|
|
The mathematical framework behind the analogy is striking. Bostrom calculates that developing SI increases our expected life span even if the probability of total human annihilation from misaligned SI were as high as approximately 97%. The models incorporate safety progress rates, temporal discounting, quality-of-life differentials between pre- and post-SI worlds, and concave QALY utilities. For most parameter settings, acceleration dominates delay. This does not mean the risk is low -- it means the cost of inaction is so astronomically high that even enormous risk is worth bearing.
|
|
|
|
The surgery analogy also challenges the LivingIP framing in an interesting way. If [[three paths to superintelligence exist but only collective superintelligence preserves human agency]], Bostrom's argument adds urgency: the collective path must be pursued *quickly*, not just correctly. Delay in developing any form of SI -- including the distributed, human-preserving kind -- carries its own existential cost.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[three paths to superintelligence exist but only collective superintelligence preserves human agency]] -- the surgery analogy adds urgency to the collective path, not just correctness
|
|
- [[capability control methods are temporary at best because a sufficiently intelligent system can circumvent any containment designed by lesser minds]] -- Bostrom still accepts control limits but now argues delay is worse than imperfect control
|
|
- [[the future is a probability space shaped by choices not a destination we approach]] -- the surgery analogy is a concrete instance of probability-space thinking about SI development
|
|
- [[permanently failing to develop superintelligence is itself an existential catastrophe because preventable mass death continues indefinitely]] -- the logical corollary: non-development is not a neutral baseline
|
|
Topics:
|
|
- [[_map]]
|