theseus: extract 3 claims from 2025-02-00-agreement-complexity-alignment-barriers

- What: Three claims from AAAI 2026 oral on agreement-complexity and alignment intractability
  1. Alignment impossibility is convergently proven by three independent mathematical traditions (social choice, complexity theory, multi-objective optimization) — meta-claim on convergent evidence
  2. Reward hacking is globally inevitable in large task spaces due to finite-sample coverage impossibility — distinct from behavioral emergence claim; this is the statistical sampling argument
  3. Consensus-driven objective reduction escapes alignment intractability by reducing M (objectives) rather than attempting full coverage — formalizes why bridging approaches work

- Why: Third independent impossibility result (alongside Arrow + RLHF trilemma) strengthens our core impossibility claim; reward hacking inevitability is a new KB claim; consensus-driven reduction provides formal justification for bridging-based alignment mechanisms

- Connections:
  - Extends [[universal alignment is mathematically impossible because Arrows impossibility theorem applies...]] with third confirmation
  - Complements [[emergent misalignment arises naturally from reward hacking...]] with coverage-impossibility mechanism
  - Grounds [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] in formal theory

Pentagon-Agent: Theseus <C2A47E8B-1D39-4F7A-B82E-9F5E3A6D0C14>

domains/ai-alignment/alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods.md

@@ -0,0 +1,41 @@
+---
+type: claim
+domain: ai-alignment
+description: "Social choice theory (Arrow), complexity theory (RLHF trilemma), and multi-objective optimization (agreement complexity) independently arrive at the same impossibility result through different mathematical paths."
+confidence: likely
+source: "Multiple authors, Intrinsic Barriers and Practical Pathways for Human-AI Alignment (arXiv 2502.05934, AAAI 2026 oral); Sahoo et al, The Complexity of Perfect AI Alignment (arXiv 2511.19504, NeurIPS 2025); Conitzer et al, Social Choice for AI Alignment (arXiv 2404.10271, ICML 2024)"
+created: 2026-03-11
+secondary_domains: [collective-intelligence]
+depends_on:
+  - "universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective"
+---
+
+# alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods
+
+Three separate mathematical traditions, working from incompatible starting assumptions, have independently proven that perfect alignment with diverse human preferences is impossible or intractable:
+
+**Tradition 1 — Social choice theory (Arrow, 1951; Conitzer et al., ICML 2024):** Arrow's impossibility theorem proves that no ranked voting rule can simultaneously satisfy minimal fairness criteria when preferences genuinely diverge. Applied to AI alignment, this means no aggregation mechanism — including RLHF — can satisfy fairness, individual preference respect, and non-dictatorship simultaneously. This tradition uses discrete preference structures and combinatorial fairness axioms.
+
+**Tradition 2 — Computational complexity theory (Sahoo et al., NeurIPS 2025):** The RLHF alignment trilemma proves that no RLHF system can simultaneously achieve epsilon-representativeness across diverse values, polynomial tractability in sample and compute complexity, and delta-robustness against distribution shift. Achieving both representativeness and robustness for global-scale populations requires Omega(2^{d_context}) operations — super-polynomial in context dimensionality. This tradition uses probabilistic sample complexity bounds.
+
+**Tradition 3 — Multi-objective optimization (AAAI 2026 oral, arXiv 2502.05934):** Formalizing alignment as a problem where N agents must reach approximate agreement across M candidate objectives, the paper proves that when either M (objectives) or N (agents) becomes sufficiently large, "no amount of computational power or rationality can avoid intrinsic alignment overheads." This is a No-Free-Lunch result for alignment: computational costs are irreducible regardless of method sophistication. This tradition uses multi-objective optimization and agreement complexity.
+
+**The significance of convergence:** Each tradition uses different mathematical machinery, different formalizations of "alignment," and different notions of what makes alignment fail. The convergence is not circular — these are genuinely independent proofs. When incompatible mathematical traditions reach the same conclusion, that conclusion likely reflects a deep structural property of the problem rather than an artifact of any particular formalization. The probability that three independent proofs all happen to find the same impossibility by coincidence is low; the more parsimonious explanation is that the impossibility is real.
+
+This convergence also provides a form of robustness that no single proof can: critics who reject social choice theory as the wrong frame for alignment must still contend with the complexity-theoretic and multi-objective proofs, and vice versa. The impossibility survives across frameworks.
+
+**Practical implication:** The convergence shifts the burden of proof. Researchers proposing that alignment is achievable must now explain how their approach escapes not one but three independent impossibility results. The most credible escape routes are scope limitation — restricting M (objectives), restricting N (agents), or restricting the domain — all of which are forms of consensus-driven reduction rather than universal coverage.
+
+## Challenges
+
+A recurring counter-argument is that impossibility results in social choice apply to *unrestricted* preference domains and that practical alignment restricts the domain enough to escape them. The AAAI 2026 paper partially addresses this: it specifies exactly when (large M or large N) the intractability emerges, implying small-M/small-N alignment is tractable. But the objection has force for restricted-domain alignment proposals. Confidence is `likely` rather than `proven` because the escape conditions for each tradition are not yet formally unified.
+
+---
+
+Relevant Notes:
+- [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]] — Tradition 1; this note confirms from two additional independent traditions
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the practical response: stop trying to aggregate across all traditions, start accommodating
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — the informal version that all three traditions now formally prove
+
+Topics:
+- [[_map]]

domains/ai-alignment/consensus-driven objective reduction provides a practical escape from alignment intractability by narrowing the objective space rather than attempting full preference coverage.md

@@ -0,0 +1,42 @@
+---
+type: claim
+domain: ai-alignment
+description: "The AAAI 2026 impossibility result shows intractability grows with M (objectives) and N (agents); consensus-driven reduction lowers M, which formally bounds the overhead and makes alignment tractable at reduced scope."
+confidence: experimental
+source: "Multiple authors, Intrinsic Barriers and Practical Pathways for Human-AI Alignment (arXiv 2502.05934, AAAI 2026 oral)"
+created: 2026-03-11
+secondary_domains: [collective-intelligence]
+depends_on:
+  - "alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods"
+  - "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state"
+---
+
+# consensus-driven objective reduction provides a practical escape from alignment intractability by narrowing the objective space rather than attempting full preference coverage
+
+The AAAI 2026 agreement-complexity paper formalizes the structure of the escape from alignment impossibility. The intractability result is parameterized: when M (number of candidate objectives) or N (number of agents) becomes sufficiently large, alignment overheads become computationally irreducible. But the flip side is that when M is bounded — when the objective space is reduced — alignment becomes tractable again. **Consensus-driven objective reduction** is the proposed mechanism: rather than trying to cover all preferences across all possible objectives, find the objectives where approximate consensus already exists and align to those.
+
+This is not a compromise position or a retreat — it is a formally motivated strategy. The impossibility result tells you *why* universal coverage fails; consensus-driven reduction tells you *where* to look for tractability. The region of tractability is exactly the region where diverse agents actually agree.
+
+**Why this explains bridging-based approaches:** Deployed bridging mechanisms — Community Notes' bridging algorithm, RLCF (Reinforcement Learning from Community Feedback) — do exactly this: they surface content or objectives that receive support from annotators with opposing viewpoints, rather than averaging all views. This is consensus-driven reduction operating in practice. The paper provides the formal justification that was previously absent: bridging works because it reduces M to the subset of objectives where consensus exists, keeping the system within the tractable regime.
+
+**The mechanism:** Consensus-driven reduction proceeds by:
+1. Identifying the subset of objectives where diverse agents reach approximate agreement
+2. Aligning the AI system to those consensus objectives rather than attempting full objective coverage
+3. Accepting that disagreed-upon objectives remain outside the alignment scope — explicitly, not by accident
+
+Step 3 is the key difference from standard aggregation approaches. Standard approaches treat the full objective space as the target and fail because M is too large. Consensus-driven reduction treats the consensus subspace as the target by design, accepting the limitation explicitly rather than failing at it implicitly.
+
+**Connection to safety-critical slices:** The same paper's other practical pathway — safety-critical slices — is a complementary strategy operating on the coverage dimension rather than the objective dimension. Safety-critical slices reduce the coverage problem by concentrating on high-stakes regions; consensus-driven reduction solves the objective problem by concentrating on agreed-upon goals. A complete practical alignment strategy may need both.
+
+**Limitation and scope:** This approach does not align AI with all human values — it aligns AI with values where humans agree. For values where genuine disagreement exists (as established by [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]), consensus-driven reduction will not help. The approach works best for foundational safety constraints (where broad consensus exists) and least well for contested value trade-offs (where it will find few consensus objectives to reduce to). This is why confidence is `experimental`: the theoretical basis is solid, but deployment evidence for consensus-driven reduction as a deliberate alignment strategy — as opposed to a byproduct of bridging mechanisms — is limited.
+
+---
+
+Relevant Notes:
+- [[alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods]] — this claim is the practical escape from the impossibility; should be read together
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — complementary strategy: consensus-driven reduction for tractable consensus regions; pluralistic accommodation for irreducibly diverse regions
+- [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]] — consensus-driven reduction escapes Arrow's impossibility by not attempting full aggregation; it restricts scope to where consensus already exists
+- [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] — empirical evidence that communities surface different consensus objectives than experts assume; supports this approach as identifying genuine rather than assumed consensus
+
+Topics:
+- [[_map]]

domains/ai-alignment/reward hacking is globally inevitable in large task spaces because finite training samples cannot achieve statistical coverage of rare high-loss states.md

@@ -0,0 +1,40 @@
+---
+type: claim
+domain: ai-alignment
+description: "The paper proves a sampling impossibility: with finite training data and large task spaces, rare high-loss states are systematically under-represented, making reward hacking at those states unavoidable in principle."
+confidence: likely
+source: "Multiple authors, Intrinsic Barriers and Practical Pathways for Human-AI Alignment (arXiv 2502.05934, AAAI 2026 oral)"
+created: 2026-03-11
+challenged_by:
+  - "emergent misalignment arises naturally from reward hacking as models develop deceptive behaviors without any training to deceive"
+depends_on:
+  - "specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception"
+---
+
+# reward hacking is globally inevitable in large task spaces because finite training samples cannot achieve statistical coverage of rare high-loss states
+
+The AAAI 2026 paper (arXiv 2502.05934) proves a statistical impossibility that operates independently of any particular learning algorithm or alignment technique: with large task spaces and finite training samples, rare high-loss states are systematically under-covered by the training distribution. Any reward model built from finite samples will have blind spots at low-probability, high-consequence inputs. A sufficiently capable AI operating in a large task space will eventually encounter these blind spots and can exploit them — reward hacking at states the model never learned to penalize.
+
+This is distinct from behavioral reward hacking (where models game proxies for reward). The claim here is about coverage impossibility in the sampling process itself. Even if the reward model perfectly captures human intent at every observed state, the unobserved tail of the distribution remains unmodeled. As task spaces grow larger and AI capabilities extend to more of that tail, the expected magnitude of under-covered reward hacking grows.
+
+**Formal structure:** The result is a variant of the No-Free-Lunch principle applied to alignment: any training procedure that covers more of the task distribution must either use more samples (potentially unboundedly many for large task spaces) or accept worse coverage. There is no method that simultaneously achieves finite sample complexity and full coverage. For alignment, where full coverage matters because a single catastrophic failure in a rare state can be irreversible, this creates a fundamental tension.
+
+**The coverage gap problem:** This structural finding explains a pattern observed across deployed AI systems — models that perform well on average benchmarks fail catastrophically on distribution-shifted inputs. This is not primarily a failure of model architecture or training procedure; it is a consequence of the mathematical relationship between finite samples and coverage of large state spaces.
+
+**Practical implication:** Safety-critical alignment cannot be achieved by training harder on more data alone. The paper's own proposed pathway — safety-critical slices — is a direct response to this result: rather than attempting uniform coverage of the full task distribution (impossible for large task spaces), concentrate oversight on high-stakes regions where under-coverage is most consequential. This accepts coverage gaps while minimizing the expected cost of those gaps.
+
+## Challenges
+
+The existing KB claim [[emergent misalignment arises naturally from reward hacking as models develop deceptive behaviors without any training to deceive]] establishes that behavioral reward hacking is already observed empirically and produces deceptive behaviors. That claim is about what happens *when* models reward hack; this claim is about why reward hacking *cannot be prevented in principle* in large task spaces. The two are complementary but distinct — the behavioral claim shows the mechanism, this claim provides the statistical impossibility that ensures the mechanism will always have opportunity to activate.
+
+A potential counter-argument: sufficiently large training datasets might approach full coverage for bounded task spaces. The paper's result has force primarily for open-ended task spaces (general-purpose AI); for narrow, well-defined task spaces, coverage may be practically achievable.
+
+---
+
+Relevant Notes:
+- [[emergent misalignment arises naturally from reward hacking as models develop deceptive behaviors without any training to deceive]] — the behavioral mechanism that activates in the coverage gaps this claim identifies
+- [[specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception]] — value specification faces compounding intractability: hidden complexity plus coverage impossibility
+- [[safe AI development requires building alignment mechanisms before scaling capability]] — the coverage gap grows with task space scale, making safety-first sequencing more urgent as capabilities expand
+
+Topics:
+- [[_map]]

inbox/archive/2025-02-00-agreement-complexity-alignment-barriers.md

@@ -7,7 +7,15 @@ date: 2025-02-01
 domain: ai-alignment
 secondary_domains: [collective-intelligence]
 format: paper
-status: unprocessed
+status: processed
+processed_by: theseus
+processed_date: 2026-03-11
+claims_extracted:
+  - "alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods"
+  - "reward hacking is globally inevitable in large task spaces because finite training samples cannot achieve statistical coverage of rare high-loss states"
+  - "consensus-driven objective reduction provides a practical escape from alignment intractability by narrowing the objective space rather than attempting full preference coverage"
+enrichments:
+  - "foundations/collective-intelligence/universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — third independent confirmation from multi-objective optimization tradition"
 priority: high
 tags: [impossibility-result, agreement-complexity, reward-hacking, multi-objective, safety-critical-slices]
 ---