extract: 2025-11-00-sahoo-rlhf-alignment-trilemma

Pentagon-Agent: Ganymede <F99EBFA6-547B-4096-BEEA-1D59C3E4028A>

domains/ai-alignment/rlhf-is-implicit-social-choice-without-normative-scrutiny.md

@@ -49,7 +49,7 @@ EM-DPO makes the social choice function explicit by using MinMax Regret Aggregat
 ### Additional Evidence (extend)
 *Source: [[2025-11-00-sahoo-rlhf-alignment-trilemma]] | Added: 2026-03-16*
 
-The trilemma formalizes why implicit social choice in RLHF is problematic: the computational constraints force strategic relaxation of either representativeness, robustness, or tractability. Current RLHF implementations implicitly choose tractability, which mathematically necessitates sacrificing representativeness (homogeneous annotator pools) and robustness (vulnerability to distribution shift). This makes the normative choices explicit: which property are we willing to sacrifice?
+The trilemma formalizes why RLHF's implicit social choice is problematic: the system must choose between representativeness, tractability, and robustness, but this choice is hidden in hyperparameters and training procedures rather than made explicit. Current RLHF implementations sacrifice representativeness for tractability without acknowledging this is a normative choice about whose values matter.
 
 ---
 

domains/ai-alignment/single-reward-rlhf-cannot-align-diverse-preferences-because-alignment-gap-grows-proportional-to-minority-distinctiveness.md

@@ -46,10 +46,10 @@ An & Du's survey reveals the mechanism behind single-reward failure: RLHF is doi
 EM-DPO provides formal proof that binary comparisons are mathematically insufficient for preference type identification, explaining WHY single-reward RLHF fails: the training signal format cannot contain the information needed to discover heterogeneity, regardless of dataset size. Rankings over 3+ responses are necessary.
 
 
-### Additional Evidence (extend)
+### Additional Evidence (confirm)
 *Source: [[2025-11-00-sahoo-rlhf-alignment-trilemma]] | Added: 2026-03-16*
 
-The alignment gap is not just proportional to minority distinctiveness — it's super-polynomial in context dimensionality. Sahoo et al. prove that achieving epsilon <= 0.01 representativeness and delta <= 0.001 robustness requires Omega(2^{d_context}) operations. Current systems use 10^3-10^4 samples while 10^7-10^8 are needed for global representation. The gap compounds exponentially with the dimensionality of human values, making it structurally impossible to close through incremental improvements.
+Formal proof that achieving epsilon <= 0.01 representativeness requires 10^7-10^8 samples for global populations, while current systems use 10^3-10^4 samples from homogeneous pools — a 3-4 order of magnitude gap. The paper quantifies the alignment gap: models assign >99% probability to majority opinions, functionally erasing minority perspectives. This is not fixable through better sampling because the computational complexity is super-polynomial.
 
 ---
 

inbox/archive/.extraction-debug/2025-11-00-sahoo-rlhf-alignment-trilemma.json

@@ -7,7 +7,7 @@
       ]
     },
     {
-      "filename": "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md",
+      "filename": "rlhf-pathologies-are-computational-necessities-not-implementation-bugs-because-preference-collapse-sycophancy-and-bias-amplification-follow-from-the-trilemma.md",
       "issues": [
         "missing_attribution_extractor"
       ]
@@ -22,13 +22,13 @@
       "rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md:set_created:2026-03-16",
       "rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md:stripped_wiki_link:universal-alignment-is-mathematically-impossible-because-Arr",
       "rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md:stripped_wiki_link:single-reward-rlhf-cannot-align-diverse-preferences-because-",
-      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md:set_created:2026-03-16",
-      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md:stripped_wiki_link:single-reward-rlhf-cannot-align-diverse-preferences-because-",
-      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md:stripped_wiki_link:emergent-misalignment-arises-naturally-from-reward-hacking-a"
+      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs-because-preference-collapse-sycophancy-and-bias-amplification-follow-from-the-trilemma.md:set_created:2026-03-16",
+      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs-because-preference-collapse-sycophancy-and-bias-amplification-follow-from-the-trilemma.md:stripped_wiki_link:emergent-misalignment-arises-naturally-from-reward-hacking-a",
+      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs-because-preference-collapse-sycophancy-and-bias-amplification-follow-from-the-trilemma.md:stripped_wiki_link:rlhf-is-implicit-social-choice-without-normative-scrutiny.md"
     ],
     "rejections": [
       "rlhf-alignment-trilemma-proves-no-system-can-simultaneously-achieve-representativeness-tractability-and-robustness.md:missing_attribution_extractor",
-      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs.md:missing_attribution_extractor"
+      "rlhf-pathologies-are-computational-necessities-not-implementation-bugs-because-preference-collapse-sycophancy-and-bias-amplification-follow-from-the-trilemma.md:missing_attribution_extractor"
     ]
   },
   "model": "anthropic/claude-sonnet-4.5",

inbox/archive/2025-11-00-sahoo-rlhf-alignment-trilemma.md

@@ -64,8 +64,7 @@ EXTRACTION HINT: The trilemma is the key claim. Also extract the practical gap (
 
 ## Key Facts
 - Paper presented at NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models
-- Authors affiliated with Berkeley AI Safety Initiative, AWS, Stanford, Meta, and Northeastern
-- Current RLHF systems collect 10^3-10^4 samples from annotator pools
-- True global representation would require 10^7-10^8 samples
-- Models assign >99% probability to majority opinions in current implementations
-- Paper proposes three strategic relaxation pathways: constrain representativeness to ~30 core values, scope robustness to plausible threats, or accept super-polynomial costs for high-stakes applications
+- Authors affiliated with Berkeley AI Safety Initiative, AWS/Stanford, Meta/Stanford, and Northeastern
+- Current RLHF systems collect 10^3-10^4 samples from homogeneous annotator pools
+- True global representation requires 10^7-10^8 samples
+- Models trained with standard RLHF assign >99% probability to majority opinions