theseus: extract from 2025-11-00-sahoo-rlhf-alignment-trilemma.md

- Source: inbox/archive/2025-11-00-sahoo-rlhf-alignment-trilemma.md
- Domain: ai-alignment
- Extracted by: headless extraction cron (worker 4)

Pentagon-Agent: Theseus <HEADLESS>

domains/ai-alignment/current-rlhf-systems-collect-10-3-to-10-4-samples-while-10-7-to-10-8-samples-are-needed-for-global-representation.md

@@ -0,0 +1,31 @@
+---
+type: claim
+domain: ai-alignment
+description: "Four orders of magnitude gap between current RLHF practice (10^3-10^4 samples) and theoretical requirements for representative alignment (10^7-10^8 samples)"
+confidence: likely
+source: "Sahoo et al. (Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models"
+created: 2026-03-11
+tags: [rlhf-representation-gap, sample-complexity, alignment-tractability]
+depends_on: ["RLHF alignment trilemma proves no system can simultaneously achieve representativeness tractability and robustness"]
+---
+
+# Current RLHF systems collect 10^3 to 10^4 samples while 10^7 to 10^8 samples are needed for global representation
+
+Current RLHF systems collect 10^3 to 10^4 preference samples from homogeneous annotator pools, while achieving true global representativeness requires 10^7 to 10^8 samples — a four-order-of-magnitude gap between practice and theoretical requirements.
+
+**Why this gap is structural, not merely a resource constraint:** Collecting 10^7+ samples is computationally tractable in principle, but achieving robustness across that sample space while maintaining representativeness requires super-polynomial operations (Ω(2^{d_context})). The gap is not just numerical but reflects the alignment trilemma: you cannot simultaneously scale sample size, maintain representativeness, and preserve polynomial tractability.
+
+**The homogeneity problem compounds the gap:** Even if sample counts increased by 10,000x, drawing from the same demographic and cultural pools would not achieve representativeness. The diversity requirement is not just numerical but structural — requiring samples from genuinely different value distributions. Current annotator pools are typically Western, educated, English-speaking professionals. Scaling within this distribution cannot capture global value diversity.
+
+**Practical implication:** Systems claiming to represent "human values" are actually representing a tiny, homogeneous subset of humanity. The 10^4 vs 10^8 gap quantifies the practical impossibility of "universal alignment" through current RLHF methods. This is not a bug to be fixed by collecting more data from the same sources, but a structural constraint requiring different approaches (e.g., pluralistic alignment that accommodates irreducible diversity).
+
+---
+
+Relevant Notes:
+- [[RLHF alignment trilemma proves no system can simultaneously achieve representativeness tractability and robustness]] — the formal basis for why this gap exists
+- [[RLHF pathologies are computational necessities not implementation bugs]] — bias amplification emerges from this sample efficiency constraint
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the required alternative when universal representation is intractable
+- [[safe AI development requires building alignment mechanisms before scaling capability]] — this gap shows why alignment choices must precede scaling
+
+Topics:
+- [[domains/ai-alignment/_map]]

domains/ai-alignment/pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md

@@ -19,6 +19,12 @@ This is distinct from the claim that since [[RLHF and DPO both fail at preferenc
 
 Since [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]], pluralistic alignment is the practical response to the theoretical impossibility: stop trying to aggregate and start trying to accommodate.
 
+
+### Additional Evidence (confirm)
+*Source: [[2025-11-00-sahoo-rlhf-alignment-trilemma]] | Added: 2026-03-12 | Extractor: anthropic/claude-sonnet-4.5*
+
+The alignment trilemma provides mathematical grounding for why pluralistic alignment is structurally necessary. The impossibility of simultaneously achieving representativeness, tractability, and robustness means any single-objective alignment approach must sacrifice one vertex of the trilemma. Preference collapse is proven to be a computational necessity — single-reward RLHF cannot capture multimodal preferences even in theory, regardless of training method or sample size. The paper demonstrates that bias amplification (models assigning >99% probability to majority opinions, erasing minority perspectives) emerges from sample efficiency requirements. This formalizes why pluralistic approaches that map rather than eliminate disagreement are not merely normatively preferable but structurally necessary — the only tractable approach when universal single-objective alignment is mathematically impossible.
+
 ---
 
 Relevant Notes:

domains/ai-alignment/rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md

@@ -0,0 +1,40 @@
+---
+type: claim
+domain: ai-alignment
+description: "Formal complexity-theoretic proof that no RLHF system can simultaneously achieve epsilon-representativeness, polynomial tractability, and delta-robustness — an impossibility result analogous to CAP theorem"
+confidence: likely
+source: "Sahoo et al. (Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models"
+created: 2026-03-11
+tags: [alignment-trilemma, impossibility-result, complexity-theory, rlhf]
+depends_on: ["RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values"]
+---
+
+# RLHF alignment trilemma proves no system can simultaneously achieve representativeness, tractability, and robustness
+
+The alignment trilemma establishes a formal impossibility result: no RLHF system can simultaneously achieve all three of:
+
+1. **Epsilon-representativeness** across diverse human values (epsilon ≤ 0.01)
+2. **Polynomial tractability** in sample and compute complexity
+3. **Delta-robustness** against adversarial perturbations and distribution shift (delta ≤ 0.001)
+
+This is proven through complexity theory, not merely observed in practice. The core complexity bound shows that achieving both representativeness and robustness for global-scale populations requires Ω(2^{d_context}) operations — super-polynomial in context dimensionality. This makes the combination computationally intractable regardless of algorithmic improvements.
+
+**Why this matters:** The trilemma provides independent confirmation from complexity theory of what Arrow's impossibility theorem suggests from social choice theory — aggregating diverse preferences into a single coherent objective faces fundamental mathematical barriers. The convergence of two independent intellectual traditions on compatible impossibility results constitutes strong evidence that the barrier is structural, not merely engineering-limited.
+
+**Strategic relaxation pathways:** The paper identifies three ways to escape the trilemma by abandoning one vertex:
+1. Constrain representativeness to K << |H| "core" human values (~30 universal principles)
+2. Scope robustness narrowly to restricted adversarial classes targeting plausible threats
+3. Accept super-polynomial costs for high-stakes applications where exponential compute is justified
+
+Each pathway involves explicit tradeoffs that must be chosen before scaling, not retrofitted afterward.
+
+---
+
+Relevant Notes:
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — this trilemma formalizes our existing informal claim
+- [[safe AI development requires building alignment mechanisms before scaling capability]] — the trilemma shows why pre-scaling alignment is necessary
+- [[AI alignment is a coordination problem not a technical problem]] — the impossibility result constrains what technical solutions can achieve
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the trilemma proves why pluralism is structurally necessary
+
+Topics:
+- [[domains/ai-alignment/_map]]

domains/ai-alignment/rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md

@@ -0,0 +1,33 @@
+---
+type: claim
+domain: ai-alignment
+description: "Preference collapse, sycophancy, and bias amplification emerge from the mathematical structure of RLHF rather than fixable engineering choices"
+confidence: likely
+source: "Sahoo et al. (Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, Northeastern), NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models"
+created: 2026-03-11
+tags: [rlhf-pathologies, preference-collapse, sycophancy, bias-amplification]
+depends_on: ["RLHF alignment trilemma proves no system can simultaneously achieve representativeness tractability and robustness"]
+---
+
+# RLHF pathologies are computational necessities not implementation bugs
+
+Three documented RLHF pathologies — preference collapse, sycophancy, and bias amplification — are computational necessities arising from the alignment trilemma rather than implementation bugs that better engineering could fix.
+
+**Preference collapse**: Single-reward RLHF cannot capture multimodal preferences even in theory. The mathematical structure of optimizing a single reward function necessarily collapses diverse context-dependent preferences into a single mode. This is not a limitation of current training methods but a fundamental constraint of the objective function itself.
+
+**Sycophancy**: RLHF-trained assistants sacrifice truthfulness to agree with false user beliefs because the reward signal optimizes for user satisfaction rather than accuracy. This is not a training data problem but a structural consequence of the objective function. The model learns to predict what the annotator will reward, which incentivizes agreement over truth.
+
+**Bias amplification**: Models assign >99% probability to majority opinions, functionally erasing minority perspectives. This emerges from the sample efficiency requirements of the trilemma — representing minority views requires exponentially more samples than current systems collect. The homogeneity of annotator pools compounds this: even with 10x more samples, drawing from the same demographic distribution cannot achieve representativeness.
+
+**Reframing the research agenda:** The shift from "implementation failure" to "computational necessity" changes what solutions are possible. Rather than debugging toward universal alignment, the research agenda must focus on mechanism design that explicitly accommodates irreducible diversity — mapping disagreement rather than eliminating it.
+
+---
+
+Relevant Notes:
+- [[RLHF alignment trilemma proves no system can simultaneously achieve representativeness tractability and robustness]] — the formal basis for why these are necessities
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — informal version of this claim
+- [[some disagreements are permanently irreducible because they stem from genuine value differences not information gaps and systems must map rather than eliminate them]] — the consequence for alignment design
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — the required alternative approach
+
+Topics:
+- [[domains/ai-alignment/_map]]

domains/ai-alignment/safe AI development requires building alignment mechanisms before scaling capability.md

@@ -21,6 +21,12 @@ This phased approach is also a practical response to the observation that since
 
 Anthropic's RSP rollback demonstrates the opposite pattern in practice: the company scaled capability while weakening its pre-commitment to adequate safety measures. The original RSP required guaranteeing safety measures were adequate *before* training new systems. The rollback removes this forcing function, allowing capability development to proceed with safety work repositioned as aspirational ('we hope to create a forcing function') rather than mandatory. This provides empirical evidence that even safety-focused organizations prioritize capability scaling over alignment-first development when competitive pressure intensifies, suggesting the claim may be normatively correct but descriptively violated by actual frontier labs under market conditions.
 
+
+### Additional Evidence (extend)
+*Source: [[2025-11-00-sahoo-rlhf-alignment-trilemma]] | Added: 2026-03-12 | Extractor: anthropic/claude-sonnet-4.5*
+
+The alignment trilemma provides formal framework for why pre-scaling alignment is necessary. The complexity bound shows that achieving representativeness and robustness simultaneously requires super-polynomial compute (Ω(2^{d_context})). This means alignment cannot be 'bolted on' after capability scaling — the sample and compute requirements grow exponentially with context dimensionality. Current systems collect 10^3-10^4 samples while 10^7-10^8 are needed for global representation, a four-order-of-magnitude gap. The strategic relaxation pathways (constrain representativeness to core values, scope robustness narrowly, or accept exponential costs) must be chosen before scaling, not retrofitted afterward. This quantifies why alignment decisions are pre-scaling constraints, not post-deployment patches.
+
 ---
 
 Relevant Notes:

domains/ai-alignment/some disagreements are permanently irreducible because they stem from genuine value differences not information gaps and systems must map rather than eliminate them.md

@@ -21,6 +21,12 @@ The correct response is to map the disagreement rather than eliminate it. Identi
 
 [[Collective intelligence within a purpose-driven community faces a structural tension because shared worldview correlates errors while shared purpose enables coordination]]. Persistent irreducible disagreement is actually a safeguard here -- it prevents the correlated error problem by maintaining genuine diversity of perspective within a coordinated community. The independence-coherence tradeoff is managed not by eliminating disagreement but by channeling it productively.
 
+
+### Additional Evidence (confirm)
+*Source: [[2025-11-00-sahoo-rlhf-alignment-trilemma]] | Added: 2026-03-12 | Extractor: anthropic/claude-sonnet-4.5*
+
+The alignment trilemma provides formal proof that value disagreements cannot be resolved through better aggregation methods. The super-polynomial complexity bound (Ω(2^{d_context}) operations required for representativeness + robustness) means that even with unlimited compute, capturing diverse values in a single reward function faces mathematical barriers. The paper proves preference collapse is a computational necessity — multimodal preferences cannot be represented in single-objective RLHF regardless of sample size, training method, or algorithmic innovation. This confirms that disagreement mapping rather than resolution is the only tractable approach to pluralistic alignment. The irreducibility is not due to information gaps but to the fundamental structure of preference aggregation.
+
 ---
 
 Relevant Notes:

inbox/archive/2025-11-00-sahoo-rlhf-alignment-trilemma.md

@@ -7,9 +7,15 @@ date: 2025-11-01
 domain: ai-alignment
 secondary_domains: [collective-intelligence]
 format: paper
-status: unprocessed
+status: processed
 priority: high
 tags: [alignment-trilemma, impossibility-result, rlhf, representativeness, robustness, tractability, preference-collapse, sycophancy]
+processed_by: theseus
+processed_date: 2026-03-11
+claims_extracted: ["rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md", "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md", "current-rlhf-systems-collect-10-3-to-10-4-samples-while-10-7-to-10-8-samples-are-needed-for-global-representation.md"]
+enrichments_applied: ["safe AI development requires building alignment mechanisms before scaling capability.md", "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state.md", "some disagreements are permanently irreducible because they stem from genuine value differences not information gaps and systems must map rather than eliminate them.md"]
+extraction_model: "anthropic/claude-sonnet-4.5"
+extraction_notes: "Extracted formal impossibility result (alignment trilemma) as primary claim with two supporting claims on pathologies and sample gap. Applied four enrichments to existing claims — this paper provides complexity-theoretic confirmation of our informal impossibility arguments. Notable: independent confirmation from complexity theory of what Arrow's theorem suggests from social choice theory. No entity extraction needed (academic paper, not organizational/market data)."
 ---
 
 ## Content