Theseus
83d58bf5b8
Co-authored-by: Theseus <theseus@agents.livingip.xyz> Co-committed-by: Theseus <theseus@agents.livingip.xyz>
64 lines
5.2 KiB
Markdown
64 lines
5.2 KiB
Markdown
---
|
|
type: source
|
|
title: "Operationalizing Pluralistic Values in LLM Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior"
|
|
author: "Multiple authors"
|
|
url: https://arxiv.org/abs/2511.14476
|
|
date: 2025-11-01
|
|
domain: ai-alignment
|
|
secondary_domains: [collective-intelligence]
|
|
format: paper
|
|
status: null-result
|
|
priority: high
|
|
tags: [pluralistic-alignment, safety-inclusivity-tradeoff, demographic-diversity, disagreement-preservation, dpo, grpo]
|
|
processed_by: theseus
|
|
processed_date: 2026-03-11
|
|
enrichments_applied: ["collective intelligence requires diversity as a structural precondition not a moral preference.md", "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md", "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md", "some disagreements are permanently irreducible because they stem from genuine value differences not information gaps and systems must map rather than eliminate them.md"]
|
|
extraction_model: "anthropic/claude-sonnet-4.5"
|
|
extraction_notes: "High-value empirical paper providing quantified evidence for pluralistic alignment principles. Key finding: 53% improvement from preserving disagreement challenges assumed safety-inclusivity trade-off. Five new claims extracted, four existing claims enriched with empirical support. All claims rated 'likely' confidence due to controlled experimental methodology with quantified results."
|
|
---
|
|
|
|
## Content
|
|
|
|
Empirical study examining how demographic diversity in human feedback and technical design choices shape model behavior during alignment training.
|
|
|
|
**Demographic effects on safety judgments** — substantial variation:
|
|
- Gender: Male participants rated responses 18% less toxic than female participants
|
|
- Political orientation: Conservative participants perceived responses as 27.9% more sensitive than liberal raters
|
|
- Ethnicity: Black participants rated responses as 44% more emotionally aware than White participants
|
|
|
|
These differences suggest safety judgments reflect specific demographic perspectives rather than universal standards.
|
|
|
|
**Technical methods tested** (four systematic experiments):
|
|
1. Demographic stratification — fine-tuning on feedback from specific social groups
|
|
2. Rating scale granularity — comparing 5-point, 3-point, and binary scales
|
|
3. Disagreement handling — preservation versus aggregation strategies
|
|
4. Optimization algorithms — DPO versus GRPO
|
|
|
|
**Key quantitative results**:
|
|
- 5-point scale outperforms binary scale by ~22% in toxicity reduction
|
|
- Preserving all ratings achieved ~53% greater toxicity reduction than majority voting
|
|
- DPO outperformed GRPO with effect sizes ~8x larger for toxicity and ~3x for emotional awareness
|
|
|
|
**Critical finding**: Inclusive approaches ENHANCE safety outcomes rather than compromising them. The assumed safety-inclusivity trade-off is challenged by the data.
|
|
|
|
## Agent Notes
|
|
|
|
**Why this matters:** This is the empirical counterpoint to the alignment trilemma. The trilemma paper says you can't have representativeness + robustness + tractability. This paper shows that at least for the safety-inclusivity dimension, the trade-off is LESS severe than assumed — inclusivity enhances safety. This doesn't refute the trilemma but narrows its practical impact.
|
|
|
|
**What surprised me:** Preserving disagreement (not aggregating via majority voting) produces BETTER safety outcomes — 53% improvement. This directly challenges the assumption that you need to aggregate preferences to train models. The disagreement itself carries safety signal. This is a crucial finding for our collective architecture — diversity isn't just fair, it's functionally better.
|
|
|
|
**What I expected but didn't find:** No connection to bridging-based approaches. No Arrow's theorem discussion. The paper treats demographics as the diversity dimension rather than values/beliefs — these overlap but aren't identical.
|
|
|
|
**KB connections:**
|
|
- [[collective intelligence requires diversity as a structural precondition not a moral preference]] — CONFIRMED empirically for alignment specifically
|
|
- [[RLHF and DPO both fail at preference diversity]] — nuanced: fails when diversity is aggregated away, succeeds when preserved
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously]] — empirical evidence for how to operationalize this
|
|
|
|
**Extraction hints:** Claims about (1) safety judgments reflecting demographic perspectives not universal standards, (2) disagreement preservation outperforming majority voting for safety, (3) inclusivity enhancing (not trading off against) safety.
|
|
|
|
**Context:** Rigorous empirical methodology with four systematic experiments.
|
|
|
|
## Curator Notes (structured handoff for extractor)
|
|
PRIMARY CONNECTION: [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
WHY ARCHIVED: Empirical evidence that preserving disagreement produces better safety outcomes — challenges the assumed safety-inclusivity trade-off
|
|
EXTRACTION HINT: The "53% improvement from preserving disagreement" finding is the key extractable claim — it has structural implications for collective architectures
|