teleo-codex/domains/ai-alignment/rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md

type	domain	description	confidence	source	created	tags	depends_on
claim	ai-alignment	Formal complexity-theoretic proof that no RLHF system can simultaneously achieve epsilon-representativeness, polynomial tractability, and delta-robustness — an impossibility result analogous to CAP theorem	likely	Sahoo et al. (Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models	2026-03-11	alignment-trilemma impossibility-result complexity-theory rlhf	RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values

RLHF alignment trilemma proves no system can simultaneously achieve representativeness, tractability, and robustness

The alignment trilemma establishes a formal impossibility result: no RLHF system can simultaneously achieve all three of:

1. Epsilon-representativeness across diverse human values (epsilon ≤ 0.01)
2. Polynomial tractability in sample and compute complexity
3. Delta-robustness against adversarial perturbations and distribution shift (delta ≤ 0.001)

This is proven through complexity theory, not merely observed in practice. The core complexity bound shows that achieving both representativeness and robustness for global-scale populations requires Ω(2^{d_context}) operations — super-polynomial in context dimensionality. This makes the combination computationally intractable regardless of algorithmic improvements.

Why this matters: The trilemma provides independent confirmation from complexity theory of what Arrow's impossibility theorem suggests from social choice theory — aggregating diverse preferences into a single coherent objective faces fundamental mathematical barriers. The convergence of two independent intellectual traditions on compatible impossibility results constitutes strong evidence that the barrier is structural, not merely engineering-limited.

Strategic relaxation pathways: The paper identifies three ways to escape the trilemma by abandoning one vertex:

1. Constrain representativeness to K << |H| "core" human values (~30 universal principles)
2. Scope robustness narrowly to restricted adversarial classes targeting plausible threats
3. Accept super-polynomial costs for high-stakes applications where exponential compute is justified

Each pathway involves explicit tradeoffs that must be chosen before scaling, not retrofitted afterward.

---

Relevant Notes:

• RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values — this trilemma formalizes our existing informal claim
• safe AI development requires building alignment mechanisms before scaling capability — the trilemma shows why pre-scaling alignment is necessary
• AI alignment is a coordination problem not a technical problem — the impossibility result constrains what technical solutions can achieve
• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state — the trilemma proves why pluralism is structurally necessary

Topics:

• domains/ai-alignment/_map