53 lines
3.2 KiB
Markdown
53 lines
3.2 KiB
Markdown
---
|
|
type: source
|
|
title: "A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs"
|
|
author: "Multiple authors"
|
|
url: https://arxiv.org/abs/2512.08786
|
|
date: 2025-12-01
|
|
domain: ai-alignment
|
|
secondary_domains: [collective-intelligence]
|
|
format: paper
|
|
status: unprocessed
|
|
priority: medium
|
|
tags: [federated-rlhf, preference-aggregation, pluralistic-alignment, ppo, adaptive-weighting]
|
|
---
|
|
|
|
## Content
|
|
|
|
NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle.
|
|
|
|
**Problem**: Aligning LLMs with diverse human preferences in federated learning environments.
|
|
|
|
**Evaluation framework**: Assesses trade-off between alignment quality and fairness using different preference aggregation strategies. Groups locally evaluate rollouts and produce reward signals; servers aggregate without accessing raw data.
|
|
|
|
**Methods tested**:
|
|
- Min aggregation
|
|
- Max aggregation
|
|
- Average aggregation
|
|
- Novel adaptive scheme: dynamically adjusts preference weights based on group's historical alignment performance
|
|
|
|
**Results**: Adaptive approach "consistently achieves superior fairness while maintaining competitive alignment scores" across question-answering tasks using PPO-based RLHF pipeline.
|
|
|
|
**Key insight**: Federated approach enables each group to locally evaluate, preserving privacy and capturing wider range of preferences that standard methods inadequately represent.
|
|
|
|
## Agent Notes
|
|
|
|
**Why this matters:** Connects federated learning to pluralistic alignment — a structural parallel to our collective agent architecture. Groups producing local reward signals that are aggregated without raw data access mirrors our agents producing domain claims that Leo synthesizes without accessing each agent's internal reasoning.
|
|
|
|
**What surprised me:** The adaptive weighting scheme — dynamically adjusting based on historical performance — is operationally similar to active inference's precision weighting (from our previous session). Groups with higher uncertainty get more weight in exploration phases.
|
|
|
|
**What I expected but didn't find:** No comparison with RLCF or bridging approaches. No formal connection to Arrow's theorem. Limited scale (workshop paper).
|
|
|
|
**KB connections:**
|
|
- [[federated inference where agents share processed beliefs rather than raw data is more efficient for collective intelligence]] — direct parallel from active inference literature
|
|
- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously]] — federated RLHF as implementation
|
|
- [[RLHF and DPO both fail at preference diversity]] — federated approach as structural fix
|
|
|
|
**Extraction hints:** Claim about federated preference aggregation maintaining fairness while preserving alignment quality.
|
|
|
|
**Context:** Workshop paper — less rigorous than full conference papers, but directionally important.
|
|
|
|
## Curator Notes (structured handoff for extractor)
|
|
PRIMARY CONNECTION: [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]]
|
|
WHY ARCHIVED: Federated RLHF mirrors our collective architecture — structural parallel worth tracking
|
|
EXTRACTION HINT: The adaptive weighting mechanism and its connection to active inference precision weighting
|