40 lines
No EOL
2.3 KiB
Markdown
40 lines
No EOL
2.3 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Empirical result showing egalitarian alignment improves minority outcomes without zero-sum tradeoff"
|
|
confidence: experimental
|
|
source: "Chakraborty et al., MaxMin-RLHF (ICML 2024)"
|
|
created: 2026-03-11
|
|
---
|
|
|
|
# Minority preference groups gain 33 percent improvement under MaxMin-RLHF without majority compromise suggesting single-reward leaves value on table
|
|
|
|
At Tulu2-7B scale with 10:1 majority:minority ratio, MaxMin-RLHF achieved ~33% improvement for minority groups while maintaining majority performance. This is not a zero-sum tradeoff—the single-reward approach was leaving value on the table, not optimally serving either group.
|
|
|
|
**Specific results:**
|
|
- Single RLHF minority accuracy: 42% (degraded from 70.4% at balanced ratio)
|
|
- MaxMin-RLHF: 56.67% win rate across both groups
|
|
- Majority performance: maintained (not degraded)
|
|
|
|
This suggests the single-reward optimization was converging to a local optimum that poorly served both groups, rather than representing a fundamental tradeoff between majority and minority utility.
|
|
|
|
**Mechanism insight**: The improvement comes from explicitly modeling preference heterogeneity rather than forcing convergence to a single reward signal. The EM algorithm discovers that different subpopulations have genuinely different reward landscapes, and optimizing for the worst-off group forces the model to find solutions that work across multiple landscapes.
|
|
|
|
## Evidence
|
|
|
|
- Tulu2-7B experiments with 10:1 ratio: 33% minority boost, no majority degradation
|
|
- GPT-2 experiments: MaxMin satisfied both sentiment (majority) and conciseness (minority) where single RLHF satisfied only sentiment
|
|
- Consistent pattern across two model scales
|
|
|
|
## Significance
|
|
|
|
This is the first empirical demonstration that pluralistic alignment can be Pareto-improving rather than requiring compromise. It challenges the implicit assumption that serving diverse preferences requires sacrificing performance for some groups.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]]
|
|
|
|
Topics:
|
|
- domains/ai-alignment/_map |