39 lines
No EOL
2.7 KiB
Markdown
39 lines
No EOL
2.7 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Current RLHF implementations make social choice decisions about evaluator selection and preference aggregation without applying formal social choice theory"
|
|
confidence: likely
|
|
source: "Conitzer et al. 2024, 'Social Choice Should Guide AI Alignment' (ICML 2024)"
|
|
created: 2026-03-11
|
|
---
|
|
|
|
# RLHF is implicit social choice without normative scrutiny
|
|
|
|
Reinforcement Learning from Human Feedback (RLHF) necessarily makes social choice decisions—which humans provide input, what feedback is collected, how it's aggregated, and how it's used—but does so without applying the normative frameworks and formal mechanisms that social choice theory has developed over 70 years. The field has effectively reinvented (badly) what social choice already solved.
|
|
|
|
Every RLHF implementation must decide:
|
|
1. **Evaluator selection**: Who provides the feedback? (convenience sampling vs. representative sampling vs. deliberative assemblies)
|
|
2. **Input mode**: What form does feedback take? (rankings, ratings, approval votes, free-form text)
|
|
3. **Aggregation method**: How are diverse preferences combined? (implicit averaging, majority rule, or formal social welfare functions)
|
|
4. **Deployment**: How is the aggregated signal used in training?
|
|
|
|
These are precisely the questions social choice theory addresses, yet current RLHF practice treats them as engineering choices rather than normative decisions requiring theoretical justification. The paper argues this is a critical oversight: "Methods from social choice theory should guide AI alignment decisions."
|
|
|
|
The practical consequence is that RLHF systems embed unexamined assumptions about preference aggregation—often implicitly assuming a single reward function can capture context-dependent human values, which [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]].
|
|
|
|
## Evidence
|
|
|
|
- Conitzer et al. (2024) position paper at ICML 2024, co-authored by Stuart Russell (Berkeley CHAI), Nathan Lambert, and leading social choice theorists
|
|
- The paper explicitly frames RLHF as making "social choice decisions without normative scrutiny"
|
|
- Proposes RLCHF (Reinforcement Learning from Collective Human Feedback) as the formal alternative that applies social welfare functions before training reward models
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]]
|
|
- [[AI alignment is a coordination problem not a technical problem]]
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
|
|
Topics:
|
|
- domains/ai-alignment/_map
|
|
- core/mechanisms/_map |