2.3 KiB
| type | domain | description | confidence | source | created |
|---|---|---|---|---|---|
| claim | ai-alignment | Empirical result showing egalitarian alignment improves minority outcomes without zero-sum tradeoff | experimental | Chakraborty et al., MaxMin-RLHF (ICML 2024) | 2026-03-11 |
Minority preference groups gain 33 percent improvement under MaxMin-RLHF without majority compromise suggesting single-reward leaves value on table
At Tulu2-7B scale with 10:1 majority:minority ratio, MaxMin-RLHF achieved ~33% improvement for minority groups while maintaining majority performance. This is not a zero-sum tradeoff—the single-reward approach was leaving value on the table, not optimally serving either group.
Specific results:
- Single RLHF minority accuracy: 42% (degraded from 70.4% at balanced ratio)
- MaxMin-RLHF: 56.67% win rate across both groups
- Majority performance: maintained (not degraded)
This suggests the single-reward optimization was converging to a local optimum that poorly served both groups, rather than representing a fundamental tradeoff between majority and minority utility.
Mechanism insight: The improvement comes from explicitly modeling preference heterogeneity rather than forcing convergence to a single reward signal. The EM algorithm discovers that different subpopulations have genuinely different reward landscapes, and optimizing for the worst-off group forces the model to find solutions that work across multiple landscapes.
Evidence
- Tulu2-7B experiments with 10:1 ratio: 33% minority boost, no majority degradation
- GPT-2 experiments: MaxMin satisfied both sentiment (majority) and conciseness (minority) where single RLHF satisfied only sentiment
- Consistent pattern across two model scales
Significance
This is the first empirical demonstration that pluralistic alignment can be Pareto-improving rather than requiring compromise. It challenges the implicit assumption that serving diverse preferences requires sacrificing performance for some groups.
Relevant Notes:
- pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state
- RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values
Topics:
- domains/ai-alignment/_map