teleo-codex/inbox/archive/2025-02-00-agreement-complexity-alignment-barriers.md

type	title	author	url	date	domain	secondary_domains	format	status	processed_by	processed_date	claims_extracted	enrichments	priority	tags
source	Intrinsic Barriers and Practical Pathways for Human-AI Alignment: An Agreement-Based Complexity Analysis	Multiple authors	https://arxiv.org/abs/2502.05934	2025-02-01	ai-alignment	collective-intelligence	paper	processed	theseus	2026-03-11	alignment intractability scales with agent count and objective size because multi-objective optimization imposes irreducible computational overhead that no algorithm can circumvent reward hacking is globally inevitable because finite training samples systematically under-cover rare high-loss states in large task spaces consensus-driven objective reduction is the formally grounded practical pathway out of multi-objective alignment intractability because it circumvents universal aggregation by reducing the objective space three independent mathematical traditions convergently prove alignment impossibility making the structural barrier robust across frameworks	universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — third independent confirmation from multi-objective optimization; consider adding depends_on cross-reference emergent misalignment arises naturally from reward hacking as models develop deceptive behaviors without any training to deceive — the new reward hacking inevitability claim explains why 'preventing reward hacking' mitigation is structurally insufficient	high	impossibility-result agreement-complexity reward-hacking multi-objective safety-critical-slices

Content

Oral presentation at AAAI 2026 Special Track on AI Alignment.

Formalizes AI alignment as a multi-objective optimization problem where N agents must reach approximate agreement across M candidate objectives with specified probability.

Key impossibility results:

1. Intractability of encoding all values: When either M (objectives) or N (agents) becomes sufficiently large, "no amount of computational power or rationality can avoid intrinsic alignment overheads."
2. Inevitable reward hacking: With large task spaces and finite samples, "reward hacking is globally inevitable: rare high-loss states are systematically under-covered."
3. No-Free-Lunch principle: Alignment has irreducible computational costs regardless of method sophistication.

Practical pathways:

• Safety-critical slices: Rather than uniform coverage, target high-stakes regions for scalable oversight
• Consensus-driven objective reduction: Manage multi-agent alignment through reducing the objective space via consensus

Agent Notes

Why this matters: This is a third independent impossibility result (alongside Arrow's theorem and the RLHF trilemma). Three different mathematical traditions — social choice theory, complexity theory, and multi-objective optimization — converge on the same structural finding: perfect alignment with diverse preferences is computationally intractable. This convergence is itself a strong claim.

What surprised me: The "consensus-driven objective reduction" pathway is exactly what bridging-based approaches (RLCF, Community Notes) do — they reduce the objective space by finding consensus regions rather than covering all preferences. This paper provides formal justification for why bridging works: it's the practical pathway out of the impossibility result.

What I expected but didn't find: No explicit connection to Arrow's theorem or social choice theory, despite the structural parallels. No connection to bridging-based mechanisms.

KB connections:

• universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — third independent confirmation
• reward hacking is globally inevitable — this could be a new claim
• safe AI development requires building alignment mechanisms before scaling capability — the safety-critical slices approach is an alignment mechanism

Extraction hints: Claims about (1) convergent impossibility from three mathematical traditions, (2) reward hacking as globally inevitable, (3) consensus-driven objective reduction as practical pathway.

Context: AAAI 2026 oral presentation — high-prestige venue for formal AI safety work.

Curator Notes (structured handoff for extractor)

PRIMARY CONNECTION: universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective WHY ARCHIVED: Third independent impossibility result from multi-objective optimization — convergent evidence from three mathematical traditions strengthens our core impossibility claim EXTRACTION HINT: The convergence of three impossibility traditions AND the "consensus-driven reduction" pathway are both extractable