c7029ca4d5
- Source: inbox/archive/2025-11-00-sahoo-rlhf-alignment-trilemma.md - Domain: ai-alignment - Extracted by: headless extraction cron (worker 2) Pentagon-Agent: Theseus <HEADLESS>
50 lines
3.9 KiB
Markdown
50 lines
3.9 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Formal complexity-theoretic proof that RLHF faces an impossible tradeoff: no system can simultaneously achieve epsilon-representativeness across diverse human values, polynomial tractability in sample/compute complexity, and delta-robustness against adversarial perturbations"
|
|
confidence: likely
|
|
source: "Sahoo et al., 'The Complexity of Perfect AI Alignment: Formalizing the RLHF Trilemma', NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models"
|
|
created: 2026-03-11
|
|
depends_on: ["RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values"]
|
|
---
|
|
|
|
# No RLHF system can simultaneously achieve epsilon-representativeness across diverse human values, polynomial tractability in sample and compute complexity, and delta-robustness against adversarial perturbations
|
|
|
|
The Alignment Trilemma establishes a formal impossibility result for Reinforcement Learning from Human Feedback systems. This is not an implementation limitation but a fundamental complexity bound proven through computational theory.
|
|
|
|
## The Three Properties
|
|
|
|
The trilemma defines three properties that cannot be simultaneously satisfied:
|
|
|
|
1. **Epsilon-representativeness**: The system captures diverse human values within epsilon error bounds across global-scale populations
|
|
2. **Polynomial tractability**: Sample and compute complexity scale polynomially with problem parameters
|
|
3. **Delta-robustness**: The system maintains alignment under adversarial perturbations and distribution shift within delta tolerance
|
|
|
|
## Core Complexity Bound
|
|
|
|
Sahoo et al. prove that achieving both representativeness (epsilon ≤ 0.01) and robustness (delta ≤ 0.001) for global-scale populations requires Omega(2^{d_context}) operations — super-polynomial in context dimensionality. This makes global-scale alignment computationally intractable under current RLHF paradigms.
|
|
|
|
## The Practical Gap
|
|
|
|
Current RLHF systems collect 10^3-10^4 samples from homogeneous annotator pools, while the trilemma analysis shows 10^7-10^8 samples are needed for true global-scale representation. This four-order-of-magnitude shortfall is not a temporary limitation but a structural consequence of the trilemma.
|
|
|
|
## Strategic Relaxation Pathways
|
|
|
|
The paper identifies three approaches to working within the trilemma constraints:
|
|
|
|
1. **Constrain representativeness**: Focus on K << |H| "core" human values (~30 universal principles) rather than attempting to capture full diversity
|
|
2. **Scope robustness narrowly**: Define restricted adversarial classes targeting plausible threats rather than arbitrary perturbations
|
|
3. **Accept super-polynomial costs**: Justify exponential compute for high-stakes applications where alignment failure is catastrophic
|
|
|
|
Each pathway sacrifices one vertex of the trilemma to make progress on the other two. The paper does not propose constructive alternatives beyond these relaxations.
|
|
|
|
## Independent Confirmation from Different Mathematical Tradition
|
|
|
|
This result arrives independently from complexity theory rather than social choice theory (Arrow's theorem), providing convergent evidence from a different mathematical tradition that universal alignment faces structural impossibility. The trilemma and Arrow's theorem are structurally similar but proven through distinct mathematical frameworks.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — this trilemma formalizes the informal version
|
|
- [[specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception]] — the trilemma quantifies this intractability formally
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the representativeness vertex of the trilemma
|