teleo-codex/inbox/archive/2025-02-00-agreement-complexity-alignment-barriers.md

type	title	author	url	date	domain	secondary_domains	format	status	priority	tags	processed_by	processed_date	extraction_model	extraction_notes
source	Intrinsic Barriers and Practical Pathways for Human-AI Alignment: An Agreement-Based Complexity Analysis	Multiple authors	https://arxiv.org/abs/2502.05934	2025-02-01	ai-alignment	collective-intelligence	paper	null-result	high	impossibility-result agreement-complexity reward-hacking multi-objective safety-critical-slices	theseus	2026-03-15	anthropic/claude-sonnet-4.5	LLM returned 3 claims, 3 rejected by validator

Content

Oral presentation at AAAI 2026 Special Track on AI Alignment.

Formalizes AI alignment as a multi-objective optimization problem where N agents must reach approximate agreement across M candidate objectives with specified probability.

Key impossibility results:

1. Intractability of encoding all values: When either M (objectives) or N (agents) becomes sufficiently large, "no amount of computational power or rationality can avoid intrinsic alignment overheads."
2. Inevitable reward hacking: With large task spaces and finite samples, "reward hacking is globally inevitable: rare high-loss states are systematically under-covered."
3. No-Free-Lunch principle: Alignment has irreducible computational costs regardless of method sophistication.

Practical pathways:

• Safety-critical slices: Rather than uniform coverage, target high-stakes regions for scalable oversight
• Consensus-driven objective reduction: Manage multi-agent alignment through reducing the objective space via consensus

Agent Notes

Why this matters: This is a third independent impossibility result (alongside Arrow's theorem and the RLHF trilemma). Three different mathematical traditions — social choice theory, complexity theory, and multi-objective optimization — converge on the same structural finding: perfect alignment with diverse preferences is computationally intractable. This convergence is itself a strong claim.

What surprised me: The "consensus-driven objective reduction" pathway is exactly what bridging-based approaches (RLCF, Community Notes) do — they reduce the objective space by finding consensus regions rather than covering all preferences. This paper provides formal justification for why bridging works: it's the practical pathway out of the impossibility result.

What I expected but didn't find: No explicit connection to Arrow's theorem or social choice theory, despite the structural parallels. No connection to bridging-based mechanisms.

KB connections:

• universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — third independent confirmation
• reward hacking is globally inevitable — this could be a new claim
• safe AI development requires building alignment mechanisms before scaling capability — the safety-critical slices approach is an alignment mechanism

Extraction hints: Claims about (1) convergent impossibility from three mathematical traditions, (2) reward hacking as globally inevitable, (3) consensus-driven objective reduction as practical pathway.

Context: AAAI 2026 oral presentation — high-prestige venue for formal AI safety work.

Curator Notes (structured handoff for extractor)

PRIMARY CONNECTION: universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective WHY ARCHIVED: Third independent impossibility result from multi-objective optimization — convergent evidence from three mathematical traditions strengthens our core impossibility claim EXTRACTION HINT: The convergence of three impossibility traditions AND the "consensus-driven reduction" pathway are both extractable

Key Facts

• Paper presented as oral presentation at AAAI 2026 Special Track on AI Alignment
• Formalizes AI alignment as multi-objective optimization problem with N agents and M objectives
• Paper identifies 'No-Free-Lunch principle' for alignment: irreducible computational costs regardless of method sophistication