m3taversal
a246972967
- What: Delete jagged intelligence and J-curve standalone claims, enrich their target claims instead. Add enrichment-vs-standalone gate, evidence bar by confidence level, and source quality assessment to evaluator framework. - Why: Post-Phase 2 calibration. Both claims were reframings of existing claims, not genuinely new mechanisms. 0 rejections across 22 PRs suggests evaluator leniency. This corrects both the specific errors and the framework gap. - Changes: - DELETE: jagged intelligence standalone → ENRICH: RSI claim with counterargument - DELETE: J-curve standalone → ENRICH: knowledge embodiment lag with AI-specific data - UPDATE: _map.md, three-conditions wiki links, source archive metadata - UPDATE: agents/leo/reasoning.md with three new evaluation gates - Peer review requested: Theseus (ai-alignment changes), Rio (internet-finance changes) Pentagon-Agent: Leo <76FB9BCA-CC16-4479-B3E5-25A3769B3D7E> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 KiB
| description | type | domain | created | source | confidence |
|---|---|---|---|---|---|
| Noah Smith argues that cognitive superintelligence alone cannot produce AI takeover — physical autonomy, robotics, and full production chain control are necessary preconditions, none of which current AI possesses | claim | ai-alignment | 2026-03-06 | Noah Smith, 'Superintelligence is already here, today' (Noahopinion, Mar 2, 2026) | experimental |
three conditions gate AI takeover risk autonomy robotics and production chain control and current AI satisfies none of them which bounds near-term catastrophic risk despite superhuman cognitive capabilities
Noah Smith identifies three necessary conditions for AI to pose a direct takeover risk, arguing that cognitive capability alone — even at superhuman levels — is insufficient. All three must be satisfied simultaneously:
-
Full autonomy: AI systems must be able to operate independently for extended periods, setting their own goals and adapting to novel situations without human instruction. Current AI agents can execute multi-step tasks but require human-defined objectives and frequently fail on open-ended problems. Autonomy is advancing but not at the level required for independent strategic action.
-
Robotics: Cognitive capability must be coupled with physical manipulation. A superintelligent chatbot cannot seize physical infrastructure, manufacture weapons, or defend territory. Current robotics is advancing rapidly but remains far behind the dexterity, reliability, and adaptability needed for AI systems to operate independently in uncontrolled physical environments.
-
Production chain control: AI must control its own production chain — manufacturing its own hardware, generating its own energy, maintaining its own infrastructure — to be independent of human cooperation. This is the most distant condition. Even the most capable AI today depends entirely on human-operated semiconductor fabrication, power grids, data centers, and supply chains.
Smith's argument is that these three conditions create a sequential gate. Each requires the previous: robotics requires autonomy to be useful, and production chain control requires both autonomy and robotics. The current state — superhuman cognition without autonomy, robotics, or production chain independence — bounds the near-term catastrophic risk.
This doesn't eliminate risk. Smith explicitly argues that AI poses severe risks through other vectors (bioterrorism, infrastructure fragility, economic displacement) that don't require any of the three conditions. But it bounds the specific "robot uprising" or "AI seizes control" scenario that dominates public imagination and some alignment research.
The outside-view value of this framing is its specificity. Rather than arguing about whether superintelligence is "dangerous" in general, it decomposes the risk into testable conditions. We can empirically track progress on each condition and update risk assessments accordingly — autonomy benchmarks, robotics capability curves, and supply chain dependencies are all measurable.
Relevant Notes:
- recursive self-improvement creates explosive intelligence gains because the system that improves is itself improving — Smith's "jagged intelligence" counterargument (enriched into this claim) argues SI is already here cognitively via combination, but these three physical conditions bound the takeover risk
- the first mover to superintelligence likely gains decisive strategic advantage because the gap between leader and followers accelerates during takeoff — the three conditions moderate decisive strategic advantage: cognitive leads don't translate to physical control without robotics and production chains
- instrumental convergence risks may be less imminent than originally argued because current AI architectures do not exhibit systematic power-seeking behavior — the three-condition gate provides a structural explanation for why power-seeking hasn't materialized: the physical preconditions don't exist
Topics: