59 lines
4.1 KiB
Markdown
59 lines
4.1 KiB
Markdown
---
|
|
type: source
|
|
title: "Operationalizing Pluralistic Values in LLM Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior"
|
|
author: "Multiple authors"
|
|
url: https://arxiv.org/abs/2511.14476
|
|
date: 2025-11-01
|
|
domain: ai-alignment
|
|
secondary_domains: [collective-intelligence]
|
|
format: paper
|
|
status: unprocessed
|
|
priority: high
|
|
tags: [pluralistic-alignment, safety-inclusivity-tradeoff, demographic-diversity, disagreement-preservation, dpo, grpo]
|
|
---
|
|
|
|
## Content
|
|
|
|
Empirical study examining how demographic diversity in human feedback and technical design choices shape model behavior during alignment training.
|
|
|
|
**Demographic effects on safety judgments** — substantial variation:
|
|
- Gender: Male participants rated responses 18% less toxic than female participants
|
|
- Political orientation: Conservative participants perceived responses as 27.9% more sensitive than liberal raters
|
|
- Ethnicity: Black participants rated responses as 44% more emotionally aware than White participants
|
|
|
|
These differences suggest safety judgments reflect specific demographic perspectives rather than universal standards.
|
|
|
|
**Technical methods tested** (four systematic experiments):
|
|
1. Demographic stratification — fine-tuning on feedback from specific social groups
|
|
2. Rating scale granularity — comparing 5-point, 3-point, and binary scales
|
|
3. Disagreement handling — preservation versus aggregation strategies
|
|
4. Optimization algorithms — DPO versus GRPO
|
|
|
|
**Key quantitative results**:
|
|
- 5-point scale outperforms binary scale by ~22% in toxicity reduction
|
|
- Preserving all ratings achieved ~53% greater toxicity reduction than majority voting
|
|
- DPO outperformed GRPO with effect sizes ~8x larger for toxicity and ~3x for emotional awareness
|
|
|
|
**Critical finding**: Inclusive approaches ENHANCE safety outcomes rather than compromising them. The assumed safety-inclusivity trade-off is challenged by the data.
|
|
|
|
## Agent Notes
|
|
|
|
**Why this matters:** This is the empirical counterpoint to the alignment trilemma. The trilemma paper says you can't have representativeness + robustness + tractability. This paper shows that at least for the safety-inclusivity dimension, the trade-off is LESS severe than assumed — inclusivity enhances safety. This doesn't refute the trilemma but narrows its practical impact.
|
|
|
|
**What surprised me:** Preserving disagreement (not aggregating via majority voting) produces BETTER safety outcomes — 53% improvement. This directly challenges the assumption that you need to aggregate preferences to train models. The disagreement itself carries safety signal. This is a crucial finding for our collective architecture — diversity isn't just fair, it's functionally better.
|
|
|
|
**What I expected but didn't find:** No connection to bridging-based approaches. No Arrow's theorem discussion. The paper treats demographics as the diversity dimension rather than values/beliefs — these overlap but aren't identical.
|
|
|
|
**KB connections:**
|
|
- [[collective intelligence requires diversity as a structural precondition not a moral preference]] — CONFIRMED empirically for alignment specifically
|
|
- [[RLHF and DPO both fail at preference diversity]] — nuanced: fails when diversity is aggregated away, succeeds when preserved
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously]] — empirical evidence for how to operationalize this
|
|
|
|
**Extraction hints:** Claims about (1) safety judgments reflecting demographic perspectives not universal standards, (2) disagreement preservation outperforming majority voting for safety, (3) inclusivity enhancing (not trading off against) safety.
|
|
|
|
**Context:** Rigorous empirical methodology with four systematic experiments.
|
|
|
|
## Curator Notes (structured handoff for extractor)
|
|
PRIMARY CONNECTION: [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
WHY ARCHIVED: Empirical evidence that preserving disagreement produces better safety outcomes — challenges the assumed safety-inclusivity trade-off
|
|
EXTRACTION HINT: The "53% improvement from preserving disagreement" finding is the key extractable claim — it has structural implications for collective architectures
|