teleo-codex/domains/ai-alignment/AI alignment is a coordination problem not a technical problem.md

description	type	domain	created	confidence	source
Getting AI right requires simultaneous alignment across competing companies, nations, and disciplines at the speed of AI development -- no existing institution can coordinate this	claim	ai-alignment	2026-02-16	likely	TeleoHumanity Manifesto, Chapter 5

AI alignment is a coordination problem not a technical problem

The manifesto makes one of its sharpest claims here: the hard part of AI alignment is not the technical challenge of specifying values in code but the coordination challenge of getting competing actors to align simultaneously.

Getting AI right requires alignment across competing companies, each racing to be first because second place may mean irrelevance. Across competing nations, each afraid the other will achieve superintelligence and use it to dominate. Across multiple academic disciplines that barely speak to each other. And it must happen at the speed of AI development, which is measured in months, not the decades or centuries over which previous coordination challenges were resolved.

No existing institution can do this. Governments move at the speed of legislation and are bounded by borders. International bodies lack enforcement. Academia is siloed by discipline. The companies building AI are locked in a race that punishes caution. The incentive structure actively makes it worse: to win the race to superintelligence is to win the right to shape the future of humanity. The prize is so vast that every actor is incentivized to move faster than safety allows. Each is locally rational. The collective outcome is potentially catastrophic.

Dario Amodei describes AI as "so powerful, such a glittering prize, that it is very difficult for human civilization to impose any restraints on it at all." He runs one of the companies building it and is telling us plainly that the system he operates within may not be governable by current institutions.

2026 case study: the Anthropic/Pentagon/OpenAI triangle. In February-March 2026, three events demonstrated this coordination failure in a single week. Anthropic dropped the core pledge of its Responsible Scaling Policy because "competitors are blazing ahead" — a voluntary safety commitment destroyed by competitive pressure. When Anthropic then tried to hold red lines on autonomous weapons in a Pentagon contract, the DoD designated them a supply chain risk (a label previously reserved for foreign adversaries) and awarded the contract to OpenAI, whose CEO admitted the deal was "definitely rushed" and "the optics don't look good." Meanwhile, a King's College London study found the same models being rushed into military deployment chose nuclear escalation in 95% of simulated war games. Three actors — a safety-conscious lab, a government customer, a willing competitor — each acting rationally from their own position, producing a collectively catastrophic trajectory. This is the coordination problem in miniature.

Since the internet enabled global communication but not global cognition, the coordination infrastructure needed doesn't exist yet. This is why collective superintelligence is the alternative to monolithic AI controlled by a few -- it solves alignment through architecture rather than attempting governance from outside the system.

Additional Evidence (extend)

Source: 2025-11-00-sahoo-rlhf-alignment-trilemma | Added: 2026-03-11 | Extractor: anthropic/claude-sonnet-4.5

The alignment trilemma provides mathematical grounding for why alignment cannot be solved through better RLHF training alone. The impossibility result shows that no single-reward optimization can simultaneously achieve representativeness, tractability, and robustness — which means alignment requires coordination mechanisms that preserve preference diversity rather than collapsing it into scalar rewards. This formalizes the intuition that alignment is fundamentally about coordinating diverse human values, not optimizing a single objective function. The trilemma's strategic relaxation pathways (constrain representativeness to core values, scope robustness narrowly, or accept super-polynomial costs) all require collective decisions about which horn of the trilemma to accept — decisions that cannot be made by technical optimization alone.

---

Relevant Notes:

• the internet enabled global communication but not global cognition -- the coordination infrastructure gap that makes this problem unsolvable with existing tools
• the alignment problem dissolves when human values are continuously woven into the system rather than specified in advance -- the structural solution to this coordination failure
• the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it -- the clearest evidence that alignment is coordination not technical: competitive dynamics undermine any individual solution
• scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps -- individual oversight fails, making collective oversight architecturally necessary
• COVID proved humanity cannot coordinate even when the threat is visible and universal -- if coordination failed on a visible, universal biological threat, AI coordination is structurally harder
• no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it -- the field has identified the coordination nature of the problem but nobody is building coordination solutions
• voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints -- Anthropic RSP rollback (Feb 2026) proves voluntary commitments cannot substitute for coordination
• government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them -- government acting as coordination-breaker rather than coordinator

Topics:

• _map