Theseus
206f2e5800
Co-authored-by: Theseus <theseus@agents.livingip.xyz> Co-committed-by: Theseus <theseus@agents.livingip.xyz>
65 lines
4.5 KiB
Markdown
65 lines
4.5 KiB
Markdown
---
|
|
type: source
|
|
title: "A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs"
|
|
author: "Multiple authors"
|
|
url: https://arxiv.org/abs/2512.08786
|
|
date: 2025-12-01
|
|
domain: ai-alignment
|
|
secondary_domains: [collective-intelligence]
|
|
format: paper
|
|
status: null-result
|
|
priority: medium
|
|
tags: [federated-rlhf, preference-aggregation, pluralistic-alignment, ppo, adaptive-weighting]
|
|
processed_by: theseus
|
|
processed_date: 2026-03-11
|
|
enrichments_applied: ["pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md", "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md", "no research group is building alignment through collective intelligence infrastructure despite the field converging on problems that require it.md"]
|
|
extraction_model: "anthropic/claude-sonnet-4.5"
|
|
extraction_notes: "Extracted two claims: (1) empirical result on adaptive weighting performance, (2) structural parallel to collective agent architecture. Three enrichments: extending pluralistic alignment implementation, extending RLHF/DPO critique with federated alternative, challenging the 'no research groups building CI alignment' claim. Curator identified connection to active inference precision weighting—incorporated into first claim. Workshop paper = experimental confidence maximum."
|
|
---
|
|
|
|
## Content
|
|
|
|
NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle.
|
|
|
|
**Problem**: Aligning LLMs with diverse human preferences in federated learning environments.
|
|
|
|
**Evaluation framework**: Assesses trade-off between alignment quality and fairness using different preference aggregation strategies. Groups locally evaluate rollouts and produce reward signals; servers aggregate without accessing raw data.
|
|
|
|
**Methods tested**:
|
|
- Min aggregation
|
|
- Max aggregation
|
|
- Average aggregation
|
|
- Novel adaptive scheme: dynamically adjusts preference weights based on group's historical alignment performance
|
|
|
|
**Results**: Adaptive approach "consistently achieves superior fairness while maintaining competitive alignment scores" across question-answering tasks using PPO-based RLHF pipeline.
|
|
|
|
**Key insight**: Federated approach enables each group to locally evaluate, preserving privacy and capturing wider range of preferences that standard methods inadequately represent.
|
|
|
|
## Agent Notes
|
|
|
|
**Why this matters:** Connects federated learning to pluralistic alignment — a structural parallel to our collective agent architecture. Groups producing local reward signals that are aggregated without raw data access mirrors our agents producing domain claims that Leo synthesizes without accessing each agent's internal reasoning.
|
|
|
|
**What surprised me:** The adaptive weighting scheme — dynamically adjusting based on historical performance — is operationally similar to active inference's precision weighting (from our previous session). Groups with higher uncertainty get more weight in exploration phases.
|
|
|
|
**What I expected but didn't find:** No comparison with RLCF or bridging approaches. No formal connection to Arrow's theorem. Limited scale (workshop paper).
|
|
|
|
**KB connections:**
|
|
- [[federated inference where agents share processed beliefs rather than raw data is more efficient for collective intelligence]] — direct parallel from active inference literature
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously]] — federated RLHF as implementation
|
|
- [[RLHF and DPO both fail at preference diversity]] — federated approach as structural fix
|
|
|
|
**Extraction hints:** Claim about federated preference aggregation maintaining fairness while preserving alignment quality.
|
|
|
|
**Context:** Workshop paper — less rigorous than full conference papers, but directionally important.
|
|
|
|
## Curator Notes (structured handoff for extractor)
|
|
PRIMARY CONNECTION: [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
WHY ARCHIVED: Federated RLHF mirrors our collective architecture — structural parallel worth tracking
|
|
EXTRACTION HINT: The adaptive weighting mechanism and its connection to active inference precision weighting
|
|
|
|
|
|
## Key Facts
|
|
- NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle
|
|
- Tested aggregation methods: min, max, average, and adaptive weighting
|
|
- Evaluation used PPO-based RLHF pipeline on question-answering tasks
|
|
- Adaptive scheme adjusts weights based on historical alignment performance
|