teleo-codex/domains/ai-alignment/no-rlhf-system-can-simultaneously-achieve-representativeness-tractability-and-robustness-alignment-trilemma.md

type	domain	description	confidence	source	created	depends_on
claim	ai-alignment	Formal complexity-theoretic proof that RLHF faces an impossible tradeoff: no system can simultaneously achieve epsilon-representativeness across diverse human values, polynomial tractability in sample/compute complexity, and delta-robustness against adversarial perturbations	likely	Sahoo et al., 'The Complexity of Perfect AI Alignment: Formalizing the RLHF Trilemma', NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models	2026-03-11	RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values

No RLHF system can simultaneously achieve epsilon-representativeness across diverse human values, polynomial tractability in sample and compute complexity, and delta-robustness against adversarial perturbations

The Alignment Trilemma establishes a formal impossibility result for Reinforcement Learning from Human Feedback systems. This is not an implementation limitation but a fundamental complexity bound proven through computational theory.

The Three Properties

The trilemma defines three properties that cannot be simultaneously satisfied:

1. Epsilon-representativeness: The system captures diverse human values within epsilon error bounds across global-scale populations
2. Polynomial tractability: Sample and compute complexity scale polynomially with problem parameters
3. Delta-robustness: The system maintains alignment under adversarial perturbations and distribution shift within delta tolerance

Core Complexity Bound

Sahoo et al. prove that achieving both representativeness (epsilon ≤ 0.01) and robustness (delta ≤ 0.001) for global-scale populations requires Omega(2^{d_context}) operations — super-polynomial in context dimensionality. This makes global-scale alignment computationally intractable under current RLHF paradigms.

The Practical Gap

Current RLHF systems collect 10^3-10^4 samples from homogeneous annotator pools, while the trilemma analysis shows 10^7-10^8 samples are needed for true global-scale representation. This four-order-of-magnitude shortfall is not a temporary limitation but a structural consequence of the trilemma.

Strategic Relaxation Pathways

The paper identifies three approaches to working within the trilemma constraints:

1. Constrain representativeness: Focus on K << |H| "core" human values (~30 universal principles) rather than attempting to capture full diversity
2. Scope robustness narrowly: Define restricted adversarial classes targeting plausible threats rather than arbitrary perturbations
3. Accept super-polynomial costs: Justify exponential compute for high-stakes applications where alignment failure is catastrophic

Each pathway sacrifices one vertex of the trilemma to make progress on the other two. The paper does not propose constructive alternatives beyond these relaxations.

Independent Confirmation from Different Mathematical Tradition

This result arrives independently from complexity theory rather than social choice theory (Arrow's theorem), providing convergent evidence from a different mathematical tradition that universal alignment faces structural impossibility. The trilemma and Arrow's theorem are structurally similar but proven through distinct mathematical frameworks.

---

Relevant Notes:

• RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values — this trilemma formalizes the informal version
• specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception — the trilemma quantifies this intractability formally
• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state — the representativeness vertex of the trilemma