theseus: extract claims from 2026-01-00-mixdpo-preference-strength-pluralistic.md

- Source: inbox/archive/2026-01-00-mixdpo-preference-strength-pluralistic.md
- Domain: ai-alignment
- Extracted by: headless extraction cron (worker 1)

Pentagon-Agent: Theseus <HEADLESS>

domains/ai-alignment/pluralistic alignment improvements are achievable with less than 10 percent computational overhead over standard DPO making heterogeneity-aware training practically viable at scale.md

@@ -0,0 +1,35 @@
+---
+type: claim
+domain: ai-alignment
+description: "MixDPO's distributional β adds 2-10% training overhead while delivering +11 win rate points on heterogeneous datasets, removing cost as an obstacle to deploying diversity-aware alignment methods"
+confidence: experimental
+source: "Theseus, from arXiv 2601.06180 (MixDPO: Modeling Preference Strength for Pluralistic Alignment, 2026)"
+created: 2026-03-11
+depends_on:
+  - "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state"
+challenged_by: []
+---
+
+# pluralistic alignment improvements are achievable with less than 10 percent computational overhead over standard DPO making heterogeneity-aware training practically viable at scale
+
+A common objection to pluralistic or diversity-aware alignment methods is that they require either substantially more computation (explicit mixture modeling, multi-objective optimization) or richer data inputs (demographic labels, separate per-group training runs). MixDPO (arXiv 2601.06180) challenges this assumption directly.
+
+MixDPO extends standard DPO by replacing the fixed scalar preference sensitivity parameter β with a learned distribution p(β). Two distributional families are implemented: LogNormal (approximated via Monte Carlo with K=16 samples) and Gamma (computed in closed form via the Lerch transcendent). Their measured computational overhead over standard DPO: LogNormal adds 1.02× (2% overhead), Gamma adds 1.1× (10% overhead). Both deliver the performance gains at a cost well within normal engineering margins.
+
+On PRISM — a dataset explicitly constructed to capture preference heterogeneity across demographic subgroups — MixDPO achieves +11.2 win rate points over baseline DPO on Pythia-2.8B. The macro-averaged preference margin (measuring performance across subgroups rather than the population mean) improves substantially, while micro-averaged performance remains competitive.
+
+The implication: the argument that pluralistic alignment must trade off cost against inclusivity is empirically weakened. At least for the class of methods that extend DPO's sensitivity parameter distributional, the additional computation needed to handle diverse preferences is negligible. Whether this extends to other pluralistic alignment architectures — particularly those that require explicit demographic structure or separate per-group reward models — is not established by this result.
+
+## Challenges
+
+The efficiency results are from a single model (Pythia-2.8B) on a single training setup. Overhead ratios may change at larger scale or with different hardware profiles. The performance gain of +11.2 win rate points is large, but the baseline is standard DPO, which the existing KB notes is already weak on preference diversity — the appropriate comparison may be against more sophisticated diversity-aware baselines (PAL, RLCF), which the paper does not provide.
+
+---
+
+Relevant Notes:
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — this claim establishes that one mechanism for distributional pluralism is within reach of standard training budgets
+- [[self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous]] — the adaptive property means no overhead is wasted when data is homogeneous
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — MixDPO constructively addresses this failure at minimal additional cost
+
+Topics:
+- [[_map]]

domains/ai-alignment/self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous.md

@@ -0,0 +1,35 @@
+---
+type: claim
+domain: ai-alignment
+description: "MixDPO's learned β distribution collapses to low variance on homogeneous data and expands on heterogeneous data, adapting automatically without requiring the practitioner to know in advance which regime they are in"
+confidence: experimental
+source: "Theseus, from arXiv 2601.06180 (MixDPO: Modeling Preference Strength for Pluralistic Alignment, 2026)"
+created: 2026-03-11
+depends_on:
+  - "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state"
+challenged_by: []
+---
+
+# self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous
+
+A persistent practical obstacle to pluralistic alignment is the chicken-and-egg problem: methods designed to handle preference heterogeneity require the practitioner to know whether their dataset is heterogeneous before applying them. Apply a diversity-aware method to homogeneous data and you add complexity with no benefit; apply a homogeneity-assuming method to diverse data and you suppress minority preferences.
+
+MixDPO (arXiv 2601.06180) dissolves this problem by treating DPO's preference sensitivity parameter β not as a fixed scalar but as a learned distribution p(β). The critical property: when trained on the Anthropic Helpful & Harmless dataset (low preference heterogeneity), the learned distribution converges to low variance, producing behavior nearly identical to standard fixed-β DPO with minimal performance gain. When trained on PRISM (high preference heterogeneity across demographics), the distribution expands to capture that diversity, yielding +11.2 win rate points on Pythia-2.8B.
+
+The method discovers whether complexity is warranted rather than assuming it. This is alignment engineering applying the principle that complexity should be earned by the data, not imposed by the designer. A practitioner can apply MixDPO without knowing their dataset's diversity structure — the learned distribution self-diagnoses and self-calibrates.
+
+This is structurally different from methods like PAL that require explicit mixture modeling or demographic labels as inputs. MixDPO requires no such prior knowledge. The diversity structure is a learned output, not a required input.
+
+## Challenges
+
+The self-adaptive behavior has been demonstrated on two datasets at different ends of the heterogeneity spectrum. Whether it degrades gracefully across the full range of intermediate heterogeneity structures, or whether there are dataset types that mislead the distributional learning, remains untested. Validation across more diverse dataset types would strengthen confidence.
+
+---
+
+Relevant Notes:
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — MixDPO is a concrete mechanism for achieving distributional pluralism without requiring prior categorization of data
+- [[the variance of a distributional preference sensitivity parameter diagnoses preference heterogeneity in training data without requiring demographic labels]] — the diagnostic and adaptive properties are two faces of the same mechanism
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — MixDPO addresses this failure constructively by learning a distribution over sensitivity rather than fixing a single value
+
+Topics:
+- [[_map]]

domains/ai-alignment/the variance of a distributional preference sensitivity parameter diagnoses preference heterogeneity in training data without requiring demographic labels.md

@@ -0,0 +1,35 @@
+---
+type: claim
+domain: ai-alignment
+description: "Learned variance of MixDPO's β distribution is high on demographically diverse datasets and low on homogeneous ones, providing an unsupervised diagnostic for dataset diversity structure"
+confidence: experimental
+source: "Theseus, from arXiv 2601.06180 (MixDPO: Modeling Preference Strength for Pluralistic Alignment, 2026)"
+created: 2026-03-11
+depends_on:
+  - "self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous"
+challenged_by: []
+---
+
+# the variance of a distributional preference sensitivity parameter diagnoses preference heterogeneity in training data without requiring demographic labels
+
+Understanding whether a preference dataset is demographically or ideologically diverse has historically required either demographic metadata from annotators or qualitative inspection. MixDPO (arXiv 2601.06180) establishes a quantitative proxy that requires neither.
+
+When MixDPO trains on PRISM — a dataset constructed to capture preference heterogeneity across demographic subgroups — the learned β distribution acquires high variance. When it trains on the Anthropic Helpful & Harmless dataset — constructed without explicit demographic diversity goals — the learned β distribution converges to low variance. The variance of the learned distribution thus functions as an unsupervised diagnostic: high variance signals that the dataset contains preferences that cannot be adequately captured by a single sensitivity value; low variance signals relative homogeneity.
+
+This is an interpretability result, not just a performance result. A practitioner who runs MixDPO on a new dataset and observes the learned variance learns something about that dataset's structure that they did not have to measure directly. In domains where demographic metadata is absent, privacy-restricted, or simply not collected, this provides a route to characterizing preference diversity that demographic labeling cannot.
+
+The diagnostic is implicit rather than explicit — the variance emerges from joint optimization of the policy and distribution parameters, not from a dedicated diversity-measurement module. Whether the variance is a reliable proxy for demographic diversity specifically, versus other forms of preference variation (e.g., task-type or temporal variation), is not established by the paper.
+
+## Challenges
+
+The paper compares only two datasets. The claim that learned variance tracks preference heterogeneity is supported by those two data points but not validated across a systematic range of dataset types. It is possible that variance tracks other dataset properties (annotator noise, task difficulty variation) rather than genuine preference diversity. This is a strong interpretability hypothesis that requires further empirical validation.
+
+---
+
+Relevant Notes:
+- [[self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous]] — adaptive behavior and diagnostic interpretability are two properties of the same mechanism
+- [[community-centred norm elicitation surfaces alignment targets materially different from developer-specified rules]] — unsupervised diagnostics could complement deliberative methods for identifying whether alignment targets differ across populations
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — diagnosing heterogeneity is a precondition for deciding which form of pluralistic alignment to apply
+
+Topics:
+- [[_map]]

inbox/archive/2026-01-00-mixdpo-preference-strength-pluralistic.md

@@ -7,7 +7,15 @@ date: 2026-01-01
 domain: ai-alignment
 secondary_domains: []
 format: paper
-status: unprocessed
+status: processed
+processed_by: theseus
+processed_date: 2026-03-11
+claims_extracted:
+  - "self-adaptive preference optimization eliminates the need for prior knowledge of dataset diversity by collapsing to standard behavior when preferences are homogeneous"
+  - "the variance of a distributional preference sensitivity parameter diagnoses preference heterogeneity in training data without requiring demographic labels"
+  - "pluralistic alignment improvements are achievable with less than 10 percent computational overhead over standard DPO making heterogeneity-aware training practically viable at scale"
+enrichments:
+  - "Constructive response to [[RLHF and DPO both fail at preference diversity]] (referenced but not yet filed as claim)"
 priority: high
 tags: [pluralistic-alignment, DPO, preference-strength, distributional-modeling, heterogeneity]
 ---