19b3855a7f
- What: Three claims from AAAI 2026 oral on agreement-complexity and alignment intractability 1. Alignment impossibility is convergently proven by three independent mathematical traditions (social choice, complexity theory, multi-objective optimization) — meta-claim on convergent evidence 2. Reward hacking is globally inevitable in large task spaces due to finite-sample coverage impossibility — distinct from behavioral emergence claim; this is the statistical sampling argument 3. Consensus-driven objective reduction escapes alignment intractability by reducing M (objectives) rather than attempting full coverage — formalizes why bridging approaches work - Why: Third independent impossibility result (alongside Arrow + RLHF trilemma) strengthens our core impossibility claim; reward hacking inevitability is a new KB claim; consensus-driven reduction provides formal justification for bridging-based alignment mechanisms - Connections: - Extends [[universal alignment is mathematically impossible because Arrows impossibility theorem applies...]] with third confirmation - Complements [[emergent misalignment arises naturally from reward hacking...]] with coverage-impossibility mechanism - Grounds [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] in formal theory Pentagon-Agent: Theseus <C2A47E8B-1D39-4F7A-B82E-9F5E3A6D0C14>
5.6 KiB
| type | domain | description | confidence | source | created | secondary_domains | depends_on | |||
|---|---|---|---|---|---|---|---|---|---|---|
| claim | ai-alignment | The AAAI 2026 impossibility result shows intractability grows with M (objectives) and N (agents); consensus-driven reduction lowers M, which formally bounds the overhead and makes alignment tractable at reduced scope. | experimental | Multiple authors, Intrinsic Barriers and Practical Pathways for Human-AI Alignment (arXiv 2502.05934, AAAI 2026 oral) | 2026-03-11 |
|
|
consensus-driven objective reduction provides a practical escape from alignment intractability by narrowing the objective space rather than attempting full preference coverage
The AAAI 2026 agreement-complexity paper formalizes the structure of the escape from alignment impossibility. The intractability result is parameterized: when M (number of candidate objectives) or N (number of agents) becomes sufficiently large, alignment overheads become computationally irreducible. But the flip side is that when M is bounded — when the objective space is reduced — alignment becomes tractable again. Consensus-driven objective reduction is the proposed mechanism: rather than trying to cover all preferences across all possible objectives, find the objectives where approximate consensus already exists and align to those.
This is not a compromise position or a retreat — it is a formally motivated strategy. The impossibility result tells you why universal coverage fails; consensus-driven reduction tells you where to look for tractability. The region of tractability is exactly the region where diverse agents actually agree.
Why this explains bridging-based approaches: Deployed bridging mechanisms — Community Notes' bridging algorithm, RLCF (Reinforcement Learning from Community Feedback) — do exactly this: they surface content or objectives that receive support from annotators with opposing viewpoints, rather than averaging all views. This is consensus-driven reduction operating in practice. The paper provides the formal justification that was previously absent: bridging works because it reduces M to the subset of objectives where consensus exists, keeping the system within the tractable regime.
The mechanism: Consensus-driven reduction proceeds by:
- Identifying the subset of objectives where diverse agents reach approximate agreement
- Aligning the AI system to those consensus objectives rather than attempting full objective coverage
- Accepting that disagreed-upon objectives remain outside the alignment scope — explicitly, not by accident
Step 3 is the key difference from standard aggregation approaches. Standard approaches treat the full objective space as the target and fail because M is too large. Consensus-driven reduction treats the consensus subspace as the target by design, accepting the limitation explicitly rather than failing at it implicitly.
Connection to safety-critical slices: The same paper's other practical pathway — safety-critical slices — is a complementary strategy operating on the coverage dimension rather than the objective dimension. Safety-critical slices reduce the coverage problem by concentrating on high-stakes regions; consensus-driven reduction solves the objective problem by concentrating on agreed-upon goals. A complete practical alignment strategy may need both.
Limitation and scope: This approach does not align AI with all human values — it aligns AI with values where humans agree. For values where genuine disagreement exists (as established by pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state), consensus-driven reduction will not help. The approach works best for foundational safety constraints (where broad consensus exists) and least well for contested value trade-offs (where it will find few consensus objectives to reduce to). This is why confidence is experimental: the theoretical basis is solid, but deployment evidence for consensus-driven reduction as a deliberate alignment strategy — as opposed to a byproduct of bridging mechanisms — is limited.
Relevant Notes:
- alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods — this claim is the practical escape from the impossibility; should be read together
- pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state — complementary strategy: consensus-driven reduction for tractable consensus regions; pluralistic accommodation for irreducibly diverse regions
- universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — consensus-driven reduction escapes Arrow's impossibility by not attempting full aggregation; it restricts scope to where consensus already exists
- community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules — empirical evidence that communities surface different consensus objectives than experts assume; supports this approach as identifying genuine rather than assumed consensus
Topics: