theseus: extract claims from 2026-01-00-mixdpo-preference-strength-pluralistic (#482)

Co-authored-by: Theseus <theseus@agents.livingip.xyz>
Co-committed-by: Theseus <theseus@agents.livingip.xyz>

domains/ai-alignment/modeling preference sensitivity as a learned distribution rather than a fixed scalar resolves DPO diversity failures without demographic labels or explicit user modeling.md

@@ -0,0 +1,39 @@
+---
+type: claim
+domain: ai-alignment
+description: "MixDPO shows distributional β earns +11.2 win rate points on heterogeneous data at 1.02–1.1× cost, without needing demographic labels or explicit mixture models"
+confidence: experimental
+source: "Theseus via arXiv 2601.06180 (MixDPO: Modeling Preference Strength for Pluralistic Alignment, Jan 2026)"
+created: 2026-03-11
+depends_on:
+  - "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values"
+  - "pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state"
+---
+
+# modeling preference sensitivity as a learned distribution rather than a fixed scalar resolves DPO diversity failures without demographic labels or explicit user modeling
+
+Standard DPO uses a fixed scalar β to control how strongly preference signals shape training — one value for every example in the dataset. This works when preferences are homogeneous but fails when the training set aggregates genuinely different populations with different tolerance for value tradeoffs. Since [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]], fixed-β DPO is a special case of that failure: it assumes not just one reward function but one preference sensitivity level.
+
+MixDPO (arXiv 2601.06180, January 2026) generalizes this by treating β as a random variable drawn from a learned distribution p(β), optimized jointly with policy parameters θ. Two distributional families are evaluated: LogNormal (estimated via Monte Carlo with K=16 samples) and Gamma (admits closed-form optimization via the Lerch transcendent). The learned distribution encodes dataset-level variance in preference strength — how much the population's certainty about preferences actually varies across comparison pairs.
+
+**Empirical results:** On the PRISM dataset (high preference heterogeneity), MixDPO achieves +11.2 win rate points over standard DPO on Pythia-2.8B. Macro-averaged preference margins — which weight minority preferences equally to majority preferences — improve substantially while micro-averaged margins (dominated by majority views) remain competitive. This demonstrates that distributional β improves pluralistic coverage without degrading majority-preference performance. On the Anthropic HH dataset (low heterogeneity), the learned distribution converges to low variance and gains are minimal — the method self-adapts rather than forcing complexity where data doesn't support it.
+
+**Computational cost:** LogNormal adds 1.02× overhead; Gamma adds 1.1×. Pluralistic alignment via distributional β is not a computationally expensive research luxury — it is a practical default.
+
+**Why no demographic labels are needed:** Preference heterogeneity is a property of the comparison pairs themselves, not of annotator identity. The distribution learns to allocate high β to examples where the comparison signal is sharp and low β to examples where preferences are diffuse — without any access to who provided the preferences. This contrasts with approaches like PAL (Pluralistic Alignment via Learned Prototypes) that require explicit user-cluster modeling.
+
+Since [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]], MixDPO is one concrete mechanism for distributional pluralism — the third form in Sorensen et al's taxonomy — implemented at the level of training dynamics rather than model outputs or constitutional specification.
+
+## Challenges
+
+MixDPO has not yet been compared to PAL or RLCF in the paper, leaving open whether distributional β outperforms explicit mixture modeling on the same benchmarks. The +11.2 win rate result is from a single preprint on Pythia-2.8B and has not been replicated at larger scales or across multiple evaluators.
+
+---
+
+Relevant Notes:
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — MixDPO is a constructive solution to this failure, not merely a diagnosis
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — distributional β implements the distributional pluralism form without explicit demographic modeling
+- [[collective intelligence requires diversity as a structural precondition not a moral preference]] — MixDPO preserves preference diversity structurally by encoding it in the training objective rather than averaging it out
+
+Topics:
+- [[_map]]

domains/ai-alignment/the variance of a learned preference sensitivity distribution diagnoses dataset heterogeneity and collapses to fixed-parameter behavior when preferences are homogeneous.md

@@ -0,0 +1,40 @@
+---
+type: claim
+domain: ai-alignment
+description: "MixDPO's learned β distribution serves dual purpose: it improves pluralistic alignment on heterogeneous data and converges to low variance on homogeneous data, making dataset diversity legible without demographic annotations"
+confidence: experimental
+source: "Theseus via arXiv 2601.06180 (MixDPO: Modeling Preference Strength for Pluralistic Alignment, Jan 2026)"
+created: 2026-03-11
+depends_on:
+  - "modeling preference sensitivity as a learned distribution rather than a fixed scalar resolves DPO diversity failures without demographic labels or explicit user modeling"
+  - "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values"
+---
+
+# the variance of a learned preference sensitivity distribution diagnoses dataset heterogeneity and collapses to fixed-parameter behavior when preferences are homogeneous
+
+Alignment methods that handle preference diversity create a design problem: when should you apply pluralistic training and when should you apply standard training? Requiring practitioners to audit their datasets for preference heterogeneity before training is a real barrier — most practitioners lack the demographic data or analytic tools to answer the question reliably.
+
+MixDPO (arXiv 2601.06180) eliminates this requirement through a self-adaptive property. Because the preference sensitivity parameter β is learned as a distribution jointly with the policy, its variance at convergence encodes information about the dataset it was trained on:
+
+- **High heterogeneity data (PRISM):** The learned distribution converges to high variance — β must range widely to account for the differing preference strengths across comparison pairs. The +11.2 win rate gain signals that this variance is informationally meaningful, not noise.
+- **Low heterogeneity data (Anthropic HH):** The learned distribution converges to low variance, approximating a point mass near the standard fixed-β value. Performance gains are minimal — consistent with the interpretation that there is no latent diversity for the distribution to capture.
+
+This means the learned variance is a post-hoc diagnostic: train once with MixDPO, read the converged variance, and you know whether your dataset had diverse preferences. No demographic labels, no separate audit pipeline, no prior assumption about your data source. The method earns complexity when the data warrants it and collapses to simpler baseline behavior when it does not.
+
+This self-adaptive collapse property has design implications beyond MixDPO. A well-designed pluralistic alignment method should have this property structurally: if your training data were actually homogeneous, the method should behave as if you had used the simpler approach. Methods that impose complexity regardless of data content add overhead without alignment benefit. The distributional β framework provides a formal instantiation of this principle.
+
+The interpretability extension is underexplored in the paper: if β variance tracks real preference heterogeneity, it could serve as a dataset quality metric for pluralistic alignment — a way to compare datasets on the dimension of preference diversity without needing annotator identity or demographic composition.
+
+## Challenges
+
+The self-adaptive interpretation rests on a single paper's results across two contrasting datasets. Whether learned β variance generalizes as a reliable diversity diagnostic across domains and model scales has not been empirically tested. The MixDPO paper does not analyze the learned distributions in depth — the diagnostic interpretation is partially an inference from the convergence behavior.
+
+---
+
+Relevant Notes:
+- [[modeling preference sensitivity as a learned distribution rather than a fixed scalar resolves DPO diversity failures without demographic labels or explicit user modeling]] — the mechanism this claim describes the diagnostic property of
+- [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — learned variance provides empirical evidence of whether a dataset falls into this failure mode
+- [[pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state]] — self-adaptive collapse means pluralistic methods can be used safely even when diversity is unknown in advance
+
+Topics:
+- [[_map]]

inbox/archive/2026-01-00-mixdpo-preference-strength-pluralistic.md

@@ -7,7 +7,13 @@ date: 2026-01-01
 domain: ai-alignment
 secondary_domains: []
 format: paper
-status: unprocessed
+status: processed
+processed_by: theseus
+processed_date: 2026-03-11
+claims_extracted:
+  - "modeling preference sensitivity as a learned distribution rather than a fixed scalar resolves DPO diversity failures without demographic labels or explicit user modeling"
+  - "the variance of a learned preference sensitivity distribution diagnoses dataset heterogeneity and collapses to fixed-parameter behavior when preferences are homogeneous"
+enrichments: []
 priority: high
 tags: [pluralistic-alignment, DPO, preference-strength, distributional-modeling, heterogeneity]
 ---