19b3855a7f
- What: Three claims from AAAI 2026 oral on agreement-complexity and alignment intractability 1. Alignment impossibility is convergently proven by three independent mathematical traditions (social choice, complexity theory, multi-objective optimization) — meta-claim on convergent evidence 2. Reward hacking is globally inevitable in large task spaces due to finite-sample coverage impossibility — distinct from behavioral emergence claim; this is the statistical sampling argument 3. Consensus-driven objective reduction escapes alignment intractability by reducing M (objectives) rather than attempting full coverage — formalizes why bridging approaches work - Why: Third independent impossibility result (alongside Arrow + RLHF trilemma) strengthens our core impossibility claim; reward hacking inevitability is a new KB claim; consensus-driven reduction provides formal justification for bridging-based alignment mechanisms - Connections: - Extends [[universal alignment is mathematically impossible because Arrows impossibility theorem applies...]] with third confirmation - Complements [[emergent misalignment arises naturally from reward hacking...]] with coverage-impossibility mechanism - Grounds [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] in formal theory Pentagon-Agent: Theseus <C2A47E8B-1D39-4F7A-B82E-9F5E3A6D0C14>
42 lines
5.6 KiB
Markdown
42 lines
5.6 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "The AAAI 2026 impossibility result shows intractability grows with M (objectives) and N (agents); consensus-driven reduction lowers M, which formally bounds the overhead and makes alignment tractable at reduced scope."
|
|
confidence: experimental
|
|
source: "Multiple authors, Intrinsic Barriers and Practical Pathways for Human-AI Alignment (arXiv 2502.05934, AAAI 2026 oral)"
|
|
created: 2026-03-11
|
|
secondary_domains: [collective-intelligence]
|
|
depends_on:
|
|
- "alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods"
|
|
- "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state"
|
|
---
|
|
|
|
# consensus-driven objective reduction provides a practical escape from alignment intractability by narrowing the objective space rather than attempting full preference coverage
|
|
|
|
The AAAI 2026 agreement-complexity paper formalizes the structure of the escape from alignment impossibility. The intractability result is parameterized: when M (number of candidate objectives) or N (number of agents) becomes sufficiently large, alignment overheads become computationally irreducible. But the flip side is that when M is bounded — when the objective space is reduced — alignment becomes tractable again. **Consensus-driven objective reduction** is the proposed mechanism: rather than trying to cover all preferences across all possible objectives, find the objectives where approximate consensus already exists and align to those.
|
|
|
|
This is not a compromise position or a retreat — it is a formally motivated strategy. The impossibility result tells you *why* universal coverage fails; consensus-driven reduction tells you *where* to look for tractability. The region of tractability is exactly the region where diverse agents actually agree.
|
|
|
|
**Why this explains bridging-based approaches:** Deployed bridging mechanisms — Community Notes' bridging algorithm, RLCF (Reinforcement Learning from Community Feedback) — do exactly this: they surface content or objectives that receive support from annotators with opposing viewpoints, rather than averaging all views. This is consensus-driven reduction operating in practice. The paper provides the formal justification that was previously absent: bridging works because it reduces M to the subset of objectives where consensus exists, keeping the system within the tractable regime.
|
|
|
|
**The mechanism:** Consensus-driven reduction proceeds by:
|
|
1. Identifying the subset of objectives where diverse agents reach approximate agreement
|
|
2. Aligning the AI system to those consensus objectives rather than attempting full objective coverage
|
|
3. Accepting that disagreed-upon objectives remain outside the alignment scope — explicitly, not by accident
|
|
|
|
Step 3 is the key difference from standard aggregation approaches. Standard approaches treat the full objective space as the target and fail because M is too large. Consensus-driven reduction treats the consensus subspace as the target by design, accepting the limitation explicitly rather than failing at it implicitly.
|
|
|
|
**Connection to safety-critical slices:** The same paper's other practical pathway — safety-critical slices — is a complementary strategy operating on the coverage dimension rather than the objective dimension. Safety-critical slices reduce the coverage problem by concentrating on high-stakes regions; consensus-driven reduction solves the objective problem by concentrating on agreed-upon goals. A complete practical alignment strategy may need both.
|
|
|
|
**Limitation and scope:** This approach does not align AI with all human values — it aligns AI with values where humans agree. For values where genuine disagreement exists (as established by [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]), consensus-driven reduction will not help. The approach works best for foundational safety constraints (where broad consensus exists) and least well for contested value trade-offs (where it will find few consensus objectives to reduce to). This is why confidence is `experimental`: the theoretical basis is solid, but deployment evidence for consensus-driven reduction as a deliberate alignment strategy — as opposed to a byproduct of bridging mechanisms — is limited.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[alignment impossibility is convergently proven by three independent mathematical traditions suggesting it reflects structural properties of the problem not limitations of current methods]] — this claim is the practical escape from the impossibility; should be read together
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — complementary strategy: consensus-driven reduction for tractable consensus regions; pluralistic accommodation for irreducibly diverse regions
|
|
- [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]] — consensus-driven reduction escapes Arrow's impossibility by not attempting full aggregation; it restricts scope to where consensus already exists
|
|
- [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] — empirical evidence that communities surface different consensus objectives than experts assume; supports this approach as identifying genuine rather than assumed consensus
|
|
|
|
Topics:
|
|
- [[_map]]
|