diff --git a/domains/ai-alignment/capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability.md b/domains/ai-alignment/capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability.md index 6b115fab9..e2f7292f3 100644 --- a/domains/ai-alignment/capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability.md +++ b/domains/ai-alignment/capabilities generalize further than alignment as systems scale because behavioral heuristics that keep systems aligned at lower capability cease to function at higher capability.md @@ -6,12 +6,12 @@ confidence: likely source: "Eliezer Yudkowsky / Nate Soares, 'AGI Ruin: A List of Lethalities' (2022), 'If Anyone Builds It, Everyone Dies' (2025), Soares 'sharp left turn' framing" created: 2026-04-05 challenged_by: - - "instrumental convergence risks may be less imminent than originally argued because current AI architectures do not exhibit systematic power-seeking behavior" - - "AI personas emerge from pre-training data as a spectrum of humanlike motivations rather than developing monomaniacal goals which makes AI behavior more unpredictable but less catastrophically focused than instrumental convergence predicts" +- instrumental convergence risks may be less imminent than originally argued because current AI architectures do not exhibit systematic power-seeking behavior +- AI personas emerge from pre-training data as a spectrum of humanlike motivations rather than developing monomaniacal goals which makes AI behavior more unpredictable but less catastrophically focused than instrumental convergence predicts related: - - "intelligence and goals are orthogonal so a superintelligence can be maximally competent while pursuing arbitrary or destructive ends" - - "capability and reliability are independent dimensions not correlated ones because a system can be highly capable at hard tasks while unreliable at easy ones and vice versa" - - "scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps" +- intelligence and goals are orthogonal so a superintelligence can be maximally competent while pursuing arbitrary or destructive ends +- capability and reliability are independent dimensions not correlated ones because a system can be highly capable at hard tasks while unreliable at easy ones and vice versa +- scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps - the shape of returns on cognitive reinvestment determines takeoff speed because constant or increasing returns on investing cognitive output into cognitive capability produce recursive self improvement reweave_edges: - the shape of returns on cognitive reinvestment determines takeoff speed because constant or increasing returns on investing cognitive output into cognitive capability produce recursive self improvement|related|2026-04-07