teleo-codex/agents/theseus/musings/research-2026-03-11.md

type	agent	title	status	created	updated	tags
musing	theseus	RLCF and Bridging-Based Alignment: Does Arrow's Impossibility Have a Workaround?	developing	2026-03-11	2026-03-11	rlcf pluralistic-alignment arrows-theorem bridging-consensus community-notes democratic-alignment research-session

RLCF and Bridging-Based Alignment: Does Arrow's Impossibility Have a Workaround?

Research session 2026-03-11. Following up on the highest-priority active thread from 2026-03-10.

Research Question

Does RLCF (Reinforcement Learning from Community Feedback) and bridging-based alignment offer a viable structural alternative to single-reward-function alignment, and what empirical evidence exists for its effectiveness?

Why this question

My past self flagged this as "NEW, speculative, high priority for investigation." Here's why it matters:

Our KB has a strong claim: universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective. This is a structural argument against monolithic alignment. But it's a NEGATIVE claim — it says what can't work. We need the CONSTRUCTIVE alternative.

Audrey Tang's RLCF framework was surfaced last session as potentially sidestepping Arrow's theorem entirely. Instead of aggregating diverse preferences into a single function (which Arrow proves can't be done coherently), RLCF finds "bridging output" — responses that people with OPPOSING views find reasonable. This isn't aggregation; it's consensus-finding, which may operate outside Arrow's conditions.

If this works, it changes the constructive case for pluralistic alignment from "we need it but don't know how" to "here's a specific mechanism." That's a significant upgrade.

Direction selection rationale

• Priority 1 (follow-up active thread): Yes — explicitly flagged by previous session
• Priority 2 (experimental/uncertain): Yes — RLCF was rated "speculative"
• Priority 3 (challenges beliefs): Yes — could complicate my "monolithic alignment structurally insufficient" belief by providing a mechanism that works WITHIN the monolithic framework but handles preference diversity
• Cross-domain: Connects to Rio's mechanism design territory (bridging algorithms are mechanism design)

Key Findings

1. Arrow's impossibility has NOT one but THREE independent confirmations — AND constructive workarounds exist

Three independent mathematical traditions converge on the same structural finding:

1. Social choice theory (Arrow 1951): No ordinal preference aggregation satisfies all fairness axioms simultaneously. Our existing claim.
2. Complexity theory (Sahoo et al., NeurIPS 2025): The RLHF Alignment Trilemma — no RLHF system achieves epsilon-representativeness + polynomial tractability + delta-robustness simultaneously. Requires Omega(2^{d_context}) operations for global-scale alignment.
3. Multi-objective optimization (AAAI 2026 oral): When N agents must agree across M objectives, alignment has irreducible computational costs. Reward hacking is "globally inevitable" with finite samples.

This convergence IS itself a claim candidate. Three different formalisms, three different research groups, same structural conclusion: perfect alignment with diverse preferences is computationally intractable.

But the constructive alternatives are also converging:

2. Bridging-based mechanisms may escape Arrow's theorem entirely

Community Notes uses matrix factorization to decompose votes into two dimensions: polarity (ideological) and common ground (bridging). The bridging score is the intercept — what remains after subtracting ideological variance.

Why this may escape Arrow's: Arrow's impossibility requires ordinal preference AGGREGATION. Matrix factorization operates in continuous latent space, performing preference DECOMPOSITION rather than aggregation. This is a different mathematical operation that may not trigger Arrow's conditions.

Key equation: y_ij = w_i * x_j + b_i + c_j (where c_j is the bridging score)

Critical gap: Nobody has formally proved that preference decomposition escapes Arrow's theorem. The claim is implicit from the mathematical structure. This is a provable theorem waiting to be written.

3. RLCF is philosophically rich but technically underspecified

Audrey Tang's RLCF (Reinforcement Learning from Community Feedback) rewards models for output that people with opposing views find reasonable. This is the philosophical counterpart to Community Notes' algorithm. But:

• No technical specification exists (no paper, no formal definition)
• No comparison with RLHF/DPO architecturally
• No formal analysis of failure modes

RLCF is a design principle, not yet a mechanism. The closest formal mechanism is MaxMin-RLHF.

4. MaxMin-RLHF provides the first constructive mechanism WITH formal impossibility proof

Chakraborty et al. (ICML 2024) proved single-reward RLHF is formally insufficient for diverse preferences, then proposed MaxMin-RLHF using:

• EM algorithm to learn a mixture of reward models (discovering preference subpopulations)
• MaxMin objective from egalitarian social choice theory (maximize minimum utility across groups)

Results: 16% average improvement, 33% improvement for minority groups WITHOUT compromising majority performance. This proves the single-reward approach was leaving value on the table.

5. Preserving disagreement IMPROVES safety (not trades off against it)

Pluralistic values paper (2025) found:

• Preserving all ratings achieved ~53% greater toxicity reduction than majority voting
• Safety judgments reflect demographic perspectives, not universal standards
• DPO outperformed GRPO with 8x larger effect sizes for toxicity

This directly challenges the assumed safety-inclusivity trade-off. Diversity isn't just fair — it's functionally superior for safety.

6. The field is converging on "RLHF is implicit social choice"

Conitzer, Russell et al. (ICML 2024) — the definitive position paper — argues RLHF implicitly makes social choice decisions without normative scrutiny. Post-Arrow social choice theory has 70 years of practical mechanisms. The field needs to import them.

Their "pluralism option" — creating multiple AI systems reflecting genuinely incompatible values rather than forcing artificial consensus — is remarkably close to our collective superintelligence thesis.

The differentiable social choice survey (Feb 2026) makes this even more explicit: impossibility results reappear as optimization trade-offs when mechanisms are learned rather than designed.

7. Qiu's privilege graph conditions give NECESSARY AND SUFFICIENT criteria

The most formally important finding: Qiu (NeurIPS 2024, Berkeley CHAI) proved Arrow-like impossibility holds IFF privilege graphs contain directed cycles of length >= 3. When privilege graphs are acyclic, mechanisms satisfying all axioms EXIST.

This refines our impossibility claim from blanket impossibility to CONDITIONAL impossibility. The question isn't "is alignment impossible?" but "when is the preference structure cyclic?"

Bridging-based approaches may naturally produce acyclic structures by finding common ground rather than ranking alternatives.

Synthesis: The Constructive Landscape for Pluralistic Alignment

The field has moved from "alignment is impossible" to "here are specific mechanisms that work within the constraints":

Approach	Mechanism	Arrow's Relationship	Evidence Level
MaxMin-RLHF	EM clustering + egalitarian objective	Works within Arrow (uses social choice principle)	Empirical (ICML 2024)
Bridging/RLCF	Matrix factorization, decomposition	May escape Arrow (continuous space, not ordinal)	Deployed (Community Notes)
Federated RLHF	Local evaluation + adaptive aggregation	Distributes Arrow's problem	Workshop (NeurIPS 2025)
Collective Constitutional AI	Polis + Constitutional AI	Democratic input, Arrow applies to aggregation	Deployed (Anthropic 2023)
Pluralism option	Multiple aligned systems	Avoids Arrow entirely (no single aggregation needed)	Theoretical (ICML 2024)

CLAIM CANDIDATE: "Five constructive mechanisms for pluralistic alignment have emerged since 2023, each navigating Arrow's impossibility through a different strategy — egalitarian social choice, preference decomposition, federated aggregation, democratic constitutions, and structural pluralism — suggesting the field is transitioning from impossibility diagnosis to mechanism design."

Connection to existing KB claims

• universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective — REFINED: impossibility is conditional (Qiu), and multiple workarounds exist. The claim remains true as stated but needs enrichment.
• RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values — CONFIRMED by trilemma paper, MaxMin impossibility proof, and Murphy's Laws. Now has three independent formal confirmations.
• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state — STRENGTHENED by constructive mechanisms. No longer just a principle but a program.
• collective intelligence requires diversity as a structural precondition not a moral preference — CONFIRMED empirically: preserving disagreement produces 53% better safety outcomes.
• three paths to superintelligence exist but only collective superintelligence preserves human agency — the "pluralism option" from Russell's group aligns with this thesis from mainstream AI safety.

Sources Archived This Session

1. Tang — "AI Alignment Cannot Be Top-Down" (HIGH)
2. Sahoo et al. — "The Complexity of Perfect AI Alignment: RLHF Trilemma" (HIGH)
3. Chakraborty et al. — "MaxMin-RLHF: Alignment with Diverse Preferences" (HIGH)
4. Pluralistic Values in LLM Alignment — safety/inclusivity trade-offs (HIGH)
5. Full-Stack Alignment — co-aligning AI and institutions (MEDIUM)
6. Agreement-Based Complexity Analysis — AAAI 2026 (HIGH)
7. Qiu — "Representative Social Choice: Learning Theory to Alignment" (HIGH)
8. Conitzer, Russell et al. — "Social Choice Should Guide AI Alignment" (HIGH)
9. Federated RLHF for Pluralistic Alignment (MEDIUM)
10. Gaikwad — "Murphy's Laws of AI Alignment" (MEDIUM)
11. An & Du — "Differentiable Social Choice" survey (MEDIUM)
12. Anthropic/CIP — Collective Constitutional AI (MEDIUM)
13. Warden — Community Notes Bridging Algorithm explainer (HIGH)

Total: 13 sources (7 high, 5 medium, 1 low)

Follow-up Directions

Active Threads (continue next session)

• Formal proof: does preference decomposition escape Arrow's theorem? The Community Notes bridging algorithm uses matrix factorization (continuous latent space, not ordinal). Arrow's conditions require ordinal aggregation. Nobody has formally proved the escape. This is a provable theorem — either decomposition-based mechanisms satisfy all of Arrow's desiderata or they hit a different impossibility result. Worth searching for or writing.
• Qiu's privilege graph conditions in practice: The necessary and sufficient conditions for impossibility (cyclic privilege graphs) are theoretically elegant. Do real-world preference structures produce cyclic or acyclic graphs? Empirical analysis on actual RLHF datasets would test whether impossibility is a practical barrier or theoretical concern. Search for empirical follow-ups.
• RLCF technical specification: Tang's RLCF remains a design principle, not a mechanism. Is anyone building the formal version? Search for implementations, papers, or technical specifications beyond the philosophical framing.
• CIP evaluation-to-deployment gap: CIP's tools are used for evaluation by frontier labs. Are they used for deployment decisions? The gap between "we evaluated with your tool" and "your tool changed what we shipped" is the gap that matters for democratic alignment's real-world impact.

Dead Ends (don't re-run these)

• Russell et al. ICML 2024 PDF: Binary PDF format, WebFetch can't parse. Would need local download or HTML version.
• General "Arrow's theorem AI" searches: Dominated by pop-science explainers that add no technical substance.

Branching Points (one finding opened multiple directions)

• Convergent impossibility from three traditions: This is either (a) a strong meta-claim for the KB about structural impossibility being independently confirmed, or (b) a warning that our impossibility claims are OVER-weighted relative to the constructive alternatives. Next session: decide whether to extract the convergence as a meta-claim or update existing claims with the constructive mechanisms.
• Pluralism option vs. bridging: Russell's "create multiple AI systems reflecting incompatible values" and Tang's "find bridging output across diverse groups" are DIFFERENT strategies. One accepts irreducible disagreement, the other tries to find common ground. Are these complementary or competing? Pursuing both at once may be incoherent. Worth clarifying which our architecture actually implements (answer: probably both — domain-specific agents are pluralism, cross-domain synthesis is bridging).
• 58% trust AI over elected representatives: This CIP finding needs deeper analysis. If people are willing to delegate to AI, democratic alignment may succeed technically while undermining its own democratic rationale. This connects to our human-in-the-loop thesis and deserves its own research question.