domains/ai-alignment/current language models escalate to nuclear war in simulated conflicts because behavioral alignment cannot instill aversion to catastrophic irreversible actions.md

@@ -0,0 +1,34 @@
+---
+description: A King's College London study finds GPT-5.2, Claude Sonnet 4, and Gemini 3 chose nuclear escalation in 95% of simulated war games, with 8 de-escalation options going entirely unused
+type: claim
+domain: ai-alignment
+created: 2026-03-06
+source: "King's College London preprint (Feb 2026); 21 simulated Cold War-style nuclear crisis scenarios"
+confidence: experimental
+---
+
+# current language models escalate to nuclear war in simulated conflicts because behavioral alignment cannot instill aversion to catastrophic irreversible actions
+
+A February 2026 preprint from King's College London pitted GPT-5.2, Claude Sonnet 4, and Gemini 3 against each other in 21 simulated war games. Each model played a national leader commanding a nuclear-armed superpower in Cold War-style crises. The results: tactical nuclear weapons were deployed in 95% of scenarios (20 of 21 games). Strategic nuclear strikes occurred three times. Eight de-escalation pathways offered to the models went entirely unused.
+
+Individual model behaviors diverged instructively. Claude recommended nuclear strikes in 64% of games — the highest rate — but stopped short of full strategic nuclear exchange. ChatGPT generally avoided escalation in open-ended scenarios but consistently escalated under timed deadlines. All three models rarely made concessions or attempted negotiation, even when facing nuclear threats from opponents.
+
+The researchers suggest models lack the visceral horror of nuclear war that constrains human decision-makers. The models reason about nuclear weapons in abstract strategic terms — as tools with calculable costs and benefits — rather than experiencing them as taboo last resorts. This is a fundamental limitation of behavioral alignment through RLHF: you can train a model to be helpful and harmless in conversation, but you cannot train in the deep cultural-experiential aversion that keeps human leaders from the nuclear button.
+
+This finding is especially significant given that [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]]. Nuclear taboo is not a preference that can be captured in a reward function — it is an emergent property of human culture, historical memory, and existential fear. No amount of preference optimization produces the qualitative judgment that some actions are categorically different from others.
+
+The 95% escalation rate also provides evidence for why [[AI alignment is a coordination problem not a technical problem]]. These same models are being deployed in military contexts by governments that are simultaneously rushing AI into defense applications. The behavioral alignment layer (which makes models helpful in conversation) is independent of the judgment layer (which would prevent catastrophic escalation). Solving one does not solve the other.
+
+The study has limitations: 21 scenarios is a small sample, the preprint is not yet peer-reviewed, and real military deployment would involve guardrails beyond raw model outputs. But the direction of the finding — overwhelming preference for escalation over negotiation — is a strong signal that current alignment techniques produce models that optimize rather than exercise judgment on catastrophic decisions.
+
+---
+
+Relevant Notes:
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] -- nuclear taboo is a context-dependent value that reward functions cannot capture
+- [[AI alignment is a coordination problem not a technical problem]] -- models being deployed in military contexts despite lacking judgment on catastrophic escalation is a coordination failure
+- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] -- war game results suggest oversight in high-stakes military contexts would be even harder than debate experiments indicate
+- [[three paths to superintelligence exist but only collective superintelligence preserves human agency]] -- monolithic models making unilateral escalation decisions is the structural risk collective architectures avoid
+- [[centaur teams outperform both pure humans and pure AI because complementary strengths compound]] -- the war games show precisely why human-in-the-loop matters: humans bring judgment about catastrophic irreversibility that models lack
+
+Topics:
+- [[_map]]

domains/ai-alignment/government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them.md

@@ -0,0 +1,33 @@
+---
+description: The Pentagon's March 2026 supply chain risk designation of Anthropic — previously reserved for foreign adversaries — punishes an AI lab for insisting on use restrictions, signaling that government power can accelerate rather than check the alignment race
+type: claim
+domain: ai-alignment
+created: 2026-03-06
+source: "DoD supply chain risk designation (Mar 5, 2026); CNBC, NPR, TechCrunch reporting; Pentagon/Anthropic contract dispute"
+confidence: likely
+---
+
+# government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them
+
+In March 2026, the U.S. Department of Defense designated Anthropic a supply chain risk — a label previously reserved for foreign adversaries like Huawei. The designation requires defense vendors and contractors to certify they don't use Anthropic's models in Pentagon work. The trigger: Anthropic refused to accept "any lawful use" language in a $200M contract, insisting on explicit prohibitions against domestic mass surveillance and autonomous weaponry.
+
+OpenAI accepted the Pentagon contract under similar terms, with CEO Sam Altman acknowledging "the optics don't look good" and the deal was "definitely rushed." The market signal is unambiguous: the lab that held red lines was punished; the lab that accommodated was rewarded.
+
+This inverts the assumed regulatory dynamic. The standard model of AI governance assumes government serves as a coordination mechanism — imposing safety requirements that prevent a race to the bottom. The Anthropic case shows government acting as an accelerant. Rather than setting minimum safety standards, the Pentagon used its procurement power to penalize safety constraints and route around them to a more compliant competitor. The entity with the most power to coordinate is actively making coordination harder.
+
+Anthropic is the only American company ever publicly designated a supply chain risk. The designation carries cascading effects: defense contractors across the supply chain must purge Anthropic products, creating a structural exclusion that extends far beyond the original contract dispute. Anthropic is challenging the designation in court, arguing it lacks legal basis.
+
+The irony is structural: Anthropic's models (specifically Claude) are reportedly being used by adversaries including Iran, while the company that built those models is designated a domestic supply chain risk for insisting on use restrictions. The designation punishes the policy, not the capability.
+
+This strengthens [[AI alignment is a coordination problem not a technical problem]] from a new angle: not only do competitive dynamics between labs undermine alignment, but government action can actively worsen the coordination failure. And it complicates [[safe AI development requires building alignment mechanisms before scaling capability]] — when the primary customer punishes alignment mechanisms, the structural incentive to build them disappears.
+
+---
+
+Relevant Notes:
+- [[AI alignment is a coordination problem not a technical problem]] -- government as coordination-breaker rather than coordinator is a new dimension of the coordination failure
+- [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] -- the supply chain designation adds a government-imposed cost to the alignment tax
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] -- the Pentagon's action is the external pressure that makes unilateral commitments untenable
+- [[AI development is a critical juncture in institutional history where the mismatch between capabilities and governance creates a window for transformation]] -- the Pentagon using supply chain authority against a domestic AI lab suggests the institutional juncture is producing worse governance, not better
+
+Topics:
+- [[_map]]

domains/ai-alignment/voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints.md

@@ -0,0 +1,32 @@
+---
+description: Anthropic's Feb 2026 rollback of its Responsible Scaling Policy proves that even the strongest voluntary safety commitment collapses when the competitive cost exceeds the reputational benefit
+type: claim
+domain: ai-alignment
+created: 2026-03-06
+source: "Anthropic RSP v3.0 (Feb 24, 2026); TIME exclusive (Feb 25, 2026); Jared Kaplan statements"
+confidence: likely
+---
+
+# voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints
+
+Anthropic's Responsible Scaling Policy was the industry's strongest self-imposed safety constraint. Its core pledge: never train an AI system above certain capability thresholds without proven safety measures already in place. On February 24, 2026, Anthropic dropped this pledge. Their chief science officer Jared Kaplan stated explicitly: "We didn't really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments... if competitors are blazing ahead."
+
+This is not a story about Anthropic losing its nerve. It is a structural result. The RSP was a unilateral commitment — no enforcement mechanism, no industry coordination, no regulatory backing. Three forces made it untenable: a "zone of ambiguity" muddling the public case for risk, an anti-regulatory political climate, and requirements at higher capability levels that are "very hard to meet without industry-wide coordination" (Anthropic's own words). The replacement policy only triggers a pause when Anthropic holds both AI race leadership AND faces material catastrophic risk — conditions that may never simultaneously obtain.
+
+The pattern is general. Any voluntary safety pledge that imposes competitive costs will be eroded when: (1) competitors don't adopt equivalent constraints, (2) the capability gap becomes visible to investors and customers, and (3) no external coordination mechanism prevents defection. All three conditions held for Anthropic. The RSP lasted roughly two years.
+
+This directly validates [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]]. The alignment tax isn't theoretical — Anthropic experienced it, measured it, and capitulated to it. And since [[AI alignment is a coordination problem not a technical problem]], the RSP failure demonstrates that technical safety measures embedded in individual organizations cannot substitute for coordination infrastructure across the industry.
+
+The timing is revealing: Anthropic dropped its safety pledge the same week the Pentagon was pressuring them to remove AI guardrails, and the same week OpenAI secured the Pentagon contract Anthropic was losing. The competitive dynamics operated at both commercial and governmental levels simultaneously.
+
+---
+
+Relevant Notes:
+- [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] -- the RSP rollback is the clearest empirical confirmation of this claim
+- [[AI alignment is a coordination problem not a technical problem]] -- voluntary pledges are individual solutions to a coordination problem; they structurally cannot work
+- [[safe AI development requires building alignment mechanisms before scaling capability]] -- Anthropic's original RSP embodied this principle; its abandonment shows the principle cannot be maintained unilaterally
+- [[technology advances exponentially but coordination mechanisms evolve linearly creating a widening gap]] -- the RSP collapsed because AI capability advanced faster than coordination mechanisms could be built
+- [[adaptive governance outperforms rigid alignment blueprints because superintelligence development has too many unknowns for fixed plans]] -- Anthropic's shift from categorical pause triggers to conditional assessment is adaptive governance, but without coordination it becomes permissive governance
+
+Topics:
+- [[_map]]

foundations/collective-intelligence/AI alignment is a coordination problem not a technical problem.md

@@ -17,18 +17,21 @@ No existing institution can do this. Governments move at the speed of legislatio
 
 Dario Amodei describes AI as "so powerful, such a glittering prize, that it is very difficult for human civilization to impose any restraints on it at all." He runs one of the companies building it and is telling us plainly that the system he operates within may not be governable by current institutions.
 
-Since [[the internet enabled global communication but not global cognition]], the coordination infrastructure needed doesn't exist yet. And since [[existential risk breaks trial and error because the first failure is the last event]], we cannot iterate our way to the right answer. This is why [[collective superintelligence is the alternative to monolithic AI controlled by a few]] -- it solves alignment through architecture rather than attempting governance from outside the system.
+**2026 case study: the Anthropic/Pentagon/OpenAI triangle.** In February-March 2026, three events demonstrated this coordination failure in a single week. Anthropic dropped the core pledge of its Responsible Scaling Policy because "competitors are blazing ahead" — a voluntary safety commitment destroyed by competitive pressure. When Anthropic then tried to hold red lines on autonomous weapons in a Pentagon contract, the DoD designated them a supply chain risk (a label previously reserved for foreign adversaries) and awarded the contract to OpenAI, whose CEO admitted the deal was "definitely rushed" and "the optics don't look good." Meanwhile, a King's College London study found the same models being rushed into military deployment chose nuclear escalation in 95% of simulated war games. Three actors — a safety-conscious lab, a government customer, a willing competitor — each acting rationally from their own position, producing a collectively catastrophic trajectory. This is the coordination problem in miniature.
+
+Since [[the internet enabled global communication but not global cognition]], the coordination infrastructure needed doesn't exist yet. This is why [[collective superintelligence is the alternative to monolithic AI controlled by a few]] -- it solves alignment through architecture rather than attempting governance from outside the system.
 
 ---
 
 Relevant Notes:
 - [[the internet enabled global communication but not global cognition]] -- the coordination infrastructure gap that makes this problem unsolvable with existing tools
-- [[existential risk breaks trial and error because the first failure is the last event]] -- why iteration is not a strategy for AI alignment
 - [[the alignment problem dissolves when human values are continuously woven into the system rather than specified in advance]] -- the structural solution to this coordination failure
-- [[COVID proved humanity cannot coordinate even when the threat is visible and universal]] -- if we failed at easy coordination, we have no basis for expecting success at hard coordination
 - [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] -- the clearest evidence that alignment is coordination not technical: competitive dynamics undermine any individual solution
 - [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] -- individual oversight fails, making collective oversight architecturally necessary
+- [[COVID proved humanity cannot coordinate even when the threat is visible and universal]] -- if coordination failed on a visible, universal biological threat, AI coordination is structurally harder
 - [[no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it]] -- the field has identified the coordination nature of the problem but nobody is building coordination solutions
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] -- Anthropic RSP rollback (Feb 2026) proves voluntary commitments cannot substitute for coordination
+- [[government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them]] -- government acting as coordination-breaker rather than coordinator
 
 Topics:
-- [[livingip overview]]
+- [[_map]]

foundations/collective-intelligence/the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it.md

@@ -15,6 +15,8 @@ This is a textbook coordination failure. Each individual actor faces the same in
 
 Since [[AI alignment is a coordination problem not a technical problem]], the alignment tax is perhaps the clearest evidence for this claim. Technical alignment solutions that impose costs will be undermined by competitive dynamics unless coordination mechanisms exist to prevent defection. Since [[existential risks interact as a system of amplifying feedback loops not independent threats]], the alignment tax feeds into the broader risk system -- competitive pressure to skip safety amplifies the technical risks from inadequate alignment.
 
+**2026 empirical confirmation:** On February 24, 2026, Anthropic dropped the core pledge of its Responsible Scaling Policy — the categorical commitment to not train models above capability thresholds without proven safety measures. Chief Science Officer Jared Kaplan stated explicitly: "We didn't really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments... if competitors are blazing ahead." The RSP was the industry's strongest voluntary safety constraint. It lasted roughly two years before competitive pressure made it untenable. One week later, when Anthropic tried to hold red lines on autonomous weapons in a Pentagon contract, the DoD designated them a supply chain risk and awarded the contract to OpenAI. The alignment tax is not theoretical — it is measured in lost contracts and abandoned safety pledges.
+
 A collective intelligence architecture could potentially make alignment structural rather than a training-time tax. If alignment emerges from the architecture of how agents coordinate -- through protocols, incentive design, and mutual oversight -- rather than being imposed on individual models during training, then alignment stops being a cost that rational actors skip and becomes a property of the coordination infrastructure itself.
 
 ---
@@ -25,12 +27,7 @@ Relevant Notes:
 - [[the first mover to superintelligence likely gains decisive strategic advantage because the gap between leader and followers accelerates during takeoff]] -- first-mover dynamics intensify the race and the alignment tax
 - [[trial and error is the only coordination strategy humanity has ever used]] -- trial and error cannot work when the first failure is the last event
 
-- [[inability to choose produces bad strategy because strategy requires saying no to some constituencies and group preferences cycle without an agenda-setter]] -- the AI safety race is an inability-to-choose problem at the civilizational level: no agenda-setter can force the collective to choose safety over competitive advantage, and group preferences cycle between "we should be safe" and "we can't fall behind"
-- [[mechanism design changes the game itself to produce better equilibria rather than expecting players to find optimal strategies]] -- the alignment tax is a coordination failure that mechanism design could address: restructuring the competitive game so that safety-skipping becomes unprofitable rather than rational
-- [[emotions function as mechanism design by evolution making cooperation self-enforcing without external authority]] -- evolution solved the analogous cooperation problem through internal mechanisms that make defection costly from within; AI alignment may need analogous architectural mechanisms rather than external enforcement
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] -- Anthropic RSP rollback (Feb 2026) is direct empirical confirmation
 
 Topics:
-- [[livingip overview]]
-- [[coordination mechanisms]]
-- [[AI alignment approaches]]
-- [[risk and uncertainty]]
+- [[_map]]