008f2504d4
- Source: inbox/archive/2025-11-00-sahoo-rlhf-alignment-trilemma.md - Domain: ai-alignment - Extracted by: headless extraction cron (worker 4) Pentagon-Agent: Theseus <HEADLESS>
40 lines
3.2 KiB
Markdown
40 lines
3.2 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Formal complexity-theoretic proof that no RLHF system can simultaneously achieve epsilon-representativeness, polynomial tractability, and delta-robustness — an impossibility result analogous to CAP theorem"
|
|
confidence: likely
|
|
source: "Sahoo et al. (Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models"
|
|
created: 2026-03-11
|
|
tags: [alignment-trilemma, impossibility-result, complexity-theory, rlhf]
|
|
depends_on: ["RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values"]
|
|
---
|
|
|
|
# RLHF alignment trilemma proves no system can simultaneously achieve representativeness, tractability, and robustness
|
|
|
|
The alignment trilemma establishes a formal impossibility result: no RLHF system can simultaneously achieve all three of:
|
|
|
|
1. **Epsilon-representativeness** across diverse human values (epsilon ≤ 0.01)
|
|
2. **Polynomial tractability** in sample and compute complexity
|
|
3. **Delta-robustness** against adversarial perturbations and distribution shift (delta ≤ 0.001)
|
|
|
|
This is proven through complexity theory, not merely observed in practice. The core complexity bound shows that achieving both representativeness and robustness for global-scale populations requires Ω(2^{d_context}) operations — super-polynomial in context dimensionality. This makes the combination computationally intractable regardless of algorithmic improvements.
|
|
|
|
**Why this matters:** The trilemma provides independent confirmation from complexity theory of what Arrow's impossibility theorem suggests from social choice theory — aggregating diverse preferences into a single coherent objective faces fundamental mathematical barriers. The convergence of two independent intellectual traditions on compatible impossibility results constitutes strong evidence that the barrier is structural, not merely engineering-limited.
|
|
|
|
**Strategic relaxation pathways:** The paper identifies three ways to escape the trilemma by abandoning one vertex:
|
|
1. Constrain representativeness to K << |H| "core" human values (~30 universal principles)
|
|
2. Scope robustness narrowly to restricted adversarial classes targeting plausible threats
|
|
3. Accept super-polynomial costs for high-stakes applications where exponential compute is justified
|
|
|
|
Each pathway involves explicit tradeoffs that must be chosen before scaling, not retrofitted afterward.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — this trilemma formalizes our existing informal claim
|
|
- [[safe AI development requires building alignment mechanisms before scaling capability]] — the trilemma shows why pre-scaling alignment is necessary
|
|
- [[AI alignment is a coordination problem not a technical problem]] — the impossibility result constrains what technical solutions can achieve
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the trilemma proves why pluralism is structurally necessary
|
|
|
|
Topics:
|
|
- [[domains/ai-alignment/_map]]
|