6e2998dcb3
- Fixed based on eval review comments - Quality gate pass 3 (fix-from-feedback) Pentagon-Agent: Theseus <HEADLESS>
3.9 KiB
| type | title | description | confidence | created | source | enrichments | |
|---|---|---|---|---|---|---|---|
| claim | Pluralistic Alignment Must Accommodate Irreducibly Diverse Values Simultaneously Rather Than Converging on a Single Aligned State | Standard alignment procedures (RLHF, DPO) reduce distributional pluralism by forcing convergence to a single model, but pluralistic alignment preserves diverse viewpoints through ensemble structures, temporal negotiation, and adaptive policy selection | likely | 2026-03-11 | Sorensen et al, Roadmap to Pluralistic Alignment (arXiv 2402.05070, ICML 2024); Klassen et al, Pluralistic Alignment Over Time (arXiv 2411.10654, NeurIPS 2024); Harland et al, Adaptive Alignment (arXiv 2410.23630, NeurIPS 2024) |
|
Pluralistic Alignment Must Accommodate Irreducibly Diverse Values Simultaneously Rather Than Converging on a Single Aligned State
Sorensen et al (ICML 2024, led by Yejin Choi) define three forms of alignment pluralism:
- Overton pluralistic models present a spectrum of reasonable responses rather than a single "correct" answer
- Steerably pluralistic models can be directed to reflect specific perspectives when appropriate
- Distributionally pluralistic models are calibrated to represent values proportional to a given population
The critical finding: standard alignment procedures (RLHF, DPO) may actively reduce distributional pluralism. The training intended to make models safer also makes them less capable of representing diverse viewpoints. This is not a side effect but a structural consequence of forcing diverse preferences into a single reward function.
Klassen et al (NeurIPS 2024) add the temporal dimension. In sequential decision-making, conflicting stakeholder preferences can be addressed over time rather than resolved in a single decision. The AI reflects different stakeholders' values at different times, applying fairness-over-time frameworks. This reframes alignment as ongoing negotiation rather than one-shot specification.
Harland et al (NeurIPS 2024) propose the technical mechanism: Multi-Objective RL with post-learning policy selection adjustment that dynamically adapts to diverse and shifting user preferences, making alignment itself adaptive rather than fixed.
Distinction from related claims:
- RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values describes the technical failure mode
- universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective establishes the theoretical impossibility
- Pluralistic alignment is the positive research program: what alignment looks like when you take diversity as irreducible rather than treating it as noise to be averaged out
Relevant Notes:
- collective intelligence requires diversity as a structural precondition not a moral preference — pluralistic alignment imports this structural insight into the alignment field; diversity is not a problem to be solved but a feature to be preserved
- universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — pluralistic alignment is the practical response to theoretical impossibility: stop trying to aggregate and start trying to accommodate
- the specification trap means any values encoded at training time become structurally unstable as deployment contexts diverge from training conditions — pluralism plus temporal adaptation addresses the specification trap
- democratic alignment assemblies produce constitutions as effective as expert-designed ones while better representing diverse populations — assemblies are one mechanism for pluralistic alignment
Topics: AI alignment, preference diversity, value pluralism, multi-objective optimization