teleo-codex/domains/ai-alignment/rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md

type	domain	description	confidence	source	created	depends_on
claim	ai-alignment	Formal impossibility result: no RLHF system can simultaneously achieve epsilon-representativeness, polynomial tractability, and delta-robustness	likely	Sahoo et al. (Berkeley AI Safety Initiative, AWS, Meta, Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models	2026-03-11	RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values

RLHF alignment trilemma proves no system can simultaneously achieve representativeness, tractability, and robustness

The alignment trilemma establishes a formal impossibility result: no RLHF system can simultaneously achieve three critical properties:

1. Epsilon-representativeness across diverse human values
2. Polynomial tractability in sample and compute complexity
3. Delta-robustness against adversarial perturbations and distribution shift

This is proven through complexity theory, not an implementation limitation. The core complexity bound shows that achieving both representativeness (epsilon ≤ 0.01) and robustness (delta ≤ 0.001) for global-scale populations requires Ω(2^{d_context}) operations—super-polynomial in context dimensionality. This makes the combination computationally intractable for real-world deployment.

The paper identifies three strategic relaxation pathways, each abandoning one vertex of the trilemma:

1. Constrain representativeness: Focus on K << |H| "core" human values (~30 universal principles) rather than attempting global diversity
2. Scope robustness narrowly: Define restricted adversarial classes targeting only plausible threats rather than worst-case perturbations
3. Accept super-polynomial costs: Justify exponential compute for high-stakes applications where tractability can be relaxed

Critically, this result arrives at a compatible impossibility conclusion to universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective but through an independent mathematical tradition (complexity theory rather than social choice theory). This provides convergent evidence from different intellectual foundations that universal alignment faces fundamental mathematical barriers.

---

Relevant Notes:

• RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values
• universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective
• safe AI development requires building alignment mechanisms before scaling capability
• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state

Topics:

• domains/ai-alignment/_map