2.1 KiB
| type | domain | description | confidence | source | created |
|---|---|---|---|---|---|
| claim | ai-alignment | Formal impossibility result showing single reward functions fail when subpopulations have distinct preferences | likely | Chakraborty et al., MaxMin-RLHF (ICML 2024) | 2026-03-11 |
Single-reward RLHF cannot align models with diverse preferences because alignment gap grows proportional to minority distinctiveness
Chakraborty et al. (2024) provide a formal impossibility result: when human preferences are diverse across subpopulations, a singular reward model in RLHF cannot adequately align language models. The alignment gap is proportional to how distinct minority preferences are and inversely proportional to their representation in the training data.
Empirical validation at Tulu2-7B scale: single reward accuracy on minority groups drops from 70.4% (balanced data) to 42% (10:1 majority:minority ratio). This 28-point degradation demonstrates the structural failure mode.
The impossibility is independent of model scale or training compute—it's a fundamental property of aggregating diverse preferences into a single scalar reward signal.
Evidence
- Formal proof in Chakraborty et al. (2024) showing alignment gap scales with subpopulation diversity
- Tulu2-7B experiments: 70.4% → 42% minority accuracy as representation decreases from 50% to 9%
- GPT-2 experiments: single RLHF satisfied majority preference (positive sentiment) while completely ignoring minority preference (conciseness)
Relationship to Existing Work
This provides formal grounding for RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values. The impossibility result is convergent with but independent from Arrow's theorem applications to alignment.
Relevant Notes:
- RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values
- pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state
Topics:
- domains/ai-alignment/_map