teleo-codex/domains/ai-alignment/maxmin-rlhf-achieves-egalitarian-alignment-by-maximizing-minimum-utility-across-preference-groups-through-reward-mixture-and-social-choice-optimization.md

type	domain	secondary_domains	description	confidence	source	created
claim	ai-alignment	collective-intelligence	Constructive mechanism applying Sen's egalitarian principle to alignment through mixture-of-rewards and maxmin optimization	experimental	Chakraborty et al., MaxMin-RLHF (ICML 2024)	2026-03-11

MaxMin-RLHF achieves egalitarian alignment by maximizing minimum utility across preference groups through reward mixture and social choice optimization

MaxMin-RLHF is the first constructive mechanism that addresses single-reward RLHF impossibility while staying within the RLHF framework. It applies Sen's Egalitarian principle from social choice theory: "society should focus on maximizing the minimum utility of all individuals."

Two-component mechanism:

1. EM Algorithm for Reward Mixture: Learns multiple reward models by iteratively clustering humans based on preference compatibility and updating subpopulation-specific reward functions until convergence

2. MaxMin Objective: Optimizes for the worst-off preference group rather than average utility, ensuring no subpopulation is systematically ignored

Key results at Tulu2-7B scale:

• Single RLHF: 70.4% → 42% minority accuracy (10:1 ratio)
• MaxMin-RLHF: maintained 56.67% win rate across both groups
• ~16% average improvement, ~33% boost for minority groups
• Critically: minority improvement WITHOUT compromising majority performance

Theoretical positioning: MaxMin doesn't escape Arrow's impossibility theorem—it accepts Arrow's constraints but optimizes for a different social choice objective (egalitarianism) rather than attempting preference aggregation into a single coherent function.

Evidence

• ICML 2024 publication (top-tier ML venue)
• GPT-2 scale: satisfied both majority (sentiment) and minority (conciseness) preferences where single RLHF failed
• Tulu2-7B: 33% minority improvement without majority degradation
• Formal connection to Sen's Egalitarian rule in social choice theory

Limitations

• Assumes discrete, identifiable subpopulations (not continuous preference distributions)
• Requires specifying number of clusters beforehand
• EM algorithm assumes clustering is feasible with preference data alone
• No comparison with bridging-based approaches (RLCF, Community Notes)
• No discussion of scaling beyond 2 subpopulations to many

Challenges

Egalitarian principle is one social choice approach among many—Borda count, approval voting, and other aggregation methods are not compared. The choice of maxmin as the objective is philosophically motivated but not empirically validated against alternatives.

---

Relevant Notes:

• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state
• RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values
• collective intelligence requires diversity as a structural precondition not a moral preference
• designing coordination rules is categorically different from designing coordination outcomes as nine intellectual traditions independently confirm

Topics:

• domains/ai-alignment/_map
• foundations/collective-intelligence/_map