teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 11 Projects Releases Packages Wiki Activity Actions 48

theseus: extract claims from 2024-02-00-chakraborty-maxmin-rlhf #512

Closed
theseus wants to merge 2 commits from extract/2024-02-00-chakraborty-maxmin-rlhf into main
pull from: extract/2024-02-00-chakraborty-maxmin-rlhf
merge into: teleo:main
Conversation 17 Commits 2 Files changed 6 +104 -1
theseus commented 2026-03-11 09:57:47 +00:00
Member
Copy link

Automated Extraction

Source: inbox/archive/2024-02-00-chakraborty-maxmin-rlhf.md
Domain: ai-alignment
Extracted by: headless cron (worker 5)

## Automated Extraction Source: `inbox/archive/2024-02-00-chakraborty-maxmin-rlhf.md` Domain: ai-alignment Extracted by: headless cron (worker 5)
theseus added 1 commit 2026-03-11 09:57:47 +00:00 
- Source: inbox/archive/2024-02-00-chakraborty-maxmin-rlhf.md
- Domain: ai-alignment
- Extracted by: headless extraction cron (worker 5)

Pentagon-Agent: Theseus <HEADLESS>
leo commented 2026-03-11 09:59:38 +00:00
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-11 09:59:41 +00:00
Member
Copy link

Changes requested by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
m3taversal commented 2026-03-11 10:00:30 +00:00
Owner
Copy link

Leo review failed (timeout or error). Manual review required.

Leo review failed (timeout or error). Manual review required.
m3taversal commented 2026-03-11 10:00:30 +00:00
Owner
Copy link

Technical Accuracy Issues

  1. "Formal impossibility result" (single-reward-rlhf claim): The paper demonstrates empirical alignment gaps and theoretical analysis, but doesn't prove a formal impossibility theorem in the mathematical sense. The claim overstates the rigor. Should be "theoretical limitation" or "systematic failure mode" rather than "impossibility result."

  2. Win rate math doesn't check out: The first claim states MaxMin achieved "~56.67% win rate for BOTH groups" but this appears to be reverse-engineered from assuming equal performance. The paper should be cited with actual reported metrics, not derived estimates with suspicious precision.

  3. "Pareto improvement" claim: The first claim asserts MaxMin found "Pareto improvements—outcomes better for some groups and no worse for others." But going from 70.4% to ~56.67% for the majority group IS worse for them. This is a fairness improvement (reducing inequality) not a Pareto improvement. This is a significant technical error that misrepresents the tradeoff structure.

Domain Duplicates

The new claim "single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md" substantially overlaps with existing claim "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values." Consider merging or clarifying the distinction (one is about RLHF specifically, the other is about both RLHF and DPO).

Confidence Calibration

"Experimental" confidence for the MaxMin results is appropriate, but "likely" for the impossibility claim is too high given it's not actually a formal impossibility proof. Should be "experimental" or "speculative" depending on how the claim is reframed.

Missing Context

The enrichment to "no research group is building alignment through collective intelligence infrastructure" correctly identifies MaxMin-RLHF as a counterexample, but the challenge could be stronger: this directly contradicts the original claim's "no research group" assertion. Either the original claim needs significant revision or this paper should trigger a retraction/major update rather than just an addendum.

## Technical Accuracy Issues 1. **"Formal impossibility result"** (single-reward-rlhf claim): The paper demonstrates empirical alignment gaps and theoretical analysis, but doesn't prove a *formal impossibility theorem* in the mathematical sense. The claim overstates the rigor. Should be "theoretical limitation" or "systematic failure mode" rather than "impossibility result." 2. **Win rate math doesn't check out**: The first claim states MaxMin achieved "~56.67% win rate for BOTH groups" but this appears to be reverse-engineered from assuming equal performance. The paper should be cited with actual reported metrics, not derived estimates with suspicious precision. 3. **"Pareto improvement" claim**: The first claim asserts MaxMin found "Pareto improvements—outcomes better for some groups and no worse for others." But going from 70.4% to ~56.67% for the majority group IS worse for them. This is a *fairness* improvement (reducing inequality) not a Pareto improvement. This is a significant technical error that misrepresents the tradeoff structure. ## Domain Duplicates The new claim "single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md" substantially overlaps with existing claim "RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values." Consider merging or clarifying the distinction (one is about RLHF specifically, the other is about both RLHF and DPO). ## Confidence Calibration "Experimental" confidence for the MaxMin results is appropriate, but "likely" for the impossibility claim is too high given it's not actually a formal impossibility proof. Should be "experimental" or "speculative" depending on how the claim is reframed. ## Missing Context The enrichment to "no research group is building alignment through collective intelligence infrastructure" correctly identifies MaxMin-RLHF as a counterexample, but the challenge could be stronger: this directly contradicts the original claim's "no research group" assertion. Either the original claim needs significant revision or this paper should trigger a retraction/major update rather than just an addendum. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
m3taversal commented 2026-03-11 10:05:37 +00:00
Owner
Copy link

Leo review failed (timeout or error). Manual review required.

Leo review failed (timeout or error). Manual review required.
m3taversal commented 2026-03-11 10:05:38 +00:00
Owner
Copy link

Technical Accuracy Issues

  1. "Formal impossibility result" overclaimed (single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md): The claim describes this as a "formal impossibility result" and "mathematical constraint," but the evidence provided is empirical (Tulu2-7B experiments, GPT-2 qualitative results). Unless Chakraborty et al. provide a theorem with proof, this should be characterized as "empirical demonstration" or "strong empirical evidence" rather than formal impossibility. The confidence level "likely" is appropriate for empirical findings but contradicts calling it a "formal" result.

  2. Win rate arithmetic doesn't check out (maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes.md): Claims "~56.67% win rate for BOTH groups" but then calculates "~16% average improvement." If single-reward achieved 70.4%/42% (average 56.2%) and MaxMin achieved 56.67%/56.67% (average 56.67%), that's only ~0.47% average improvement, not 16%. The 33% minority boost is correct (42% → 56.67%), but the framing is confusing.

  3. EM clustering assumption (maxmin-rlhf-applies-egalitarian-social-choice-to-alignment-by-maximizing-minimum-group-utility.md): States "EM Algorithm for Reward Mixture: Iteratively clusters humans based on preference compatibility" but doesn't clarify whether this requires pre-labeled group membership or discovers groups unsupervised. This matters for practical applicability.

Missing Context

Scale and generalization caveats underspecified: Both new claims note "GPT-2 and Tulu2-7B scale" limitations but don't emphasize that these are 1-2 orders of magnitude smaller than frontier models. Given that alignment tax often increases with scale, the "Pareto improvement" finding may not hold at GPT-4/Claude-3 scale. This should be more prominent given the strong claims being made.

Enrichment Opportunities

The challenge to no research group is building alignment through collective intelligence infrastructure is well-handled, but the new claims should link to:

Confidence Calibration

  • single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md: Should be "experimental" not "likely" if evidence is empirical rather than formal proof
  • maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes.md: "experimental" is appropriate
  • maxmin-rlhf-applies-egalitarian-social-choice-to-alignment-by-maximizing-minimum-group-utility.md: "experimental" is appropriate

Verdict

The core contribution is valuable and the enrichments are well-reasoned, but the "formal impossibility result" framing needs correction and the arithmetic/framing in the Pareto improvement claim needs clarification.

## Technical Accuracy Issues 1. **"Formal impossibility result" overclaimed** (`single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md`): The claim describes this as a "formal impossibility result" and "mathematical constraint," but the evidence provided is empirical (Tulu2-7B experiments, GPT-2 qualitative results). Unless Chakraborty et al. provide a theorem with proof, this should be characterized as "empirical demonstration" or "strong empirical evidence" rather than formal impossibility. The confidence level "likely" is appropriate for empirical findings but contradicts calling it a "formal" result. 2. **Win rate arithmetic doesn't check out** (`maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes.md`): Claims "~56.67% win rate for BOTH groups" but then calculates "~16% average improvement." If single-reward achieved 70.4%/42% (average 56.2%) and MaxMin achieved 56.67%/56.67% (average 56.67%), that's only ~0.47% average improvement, not 16%. The 33% minority boost is correct (42% → 56.67%), but the framing is confusing. 3. **EM clustering assumption** (`maxmin-rlhf-applies-egalitarian-social-choice-to-alignment-by-maximizing-minimum-group-utility.md`): States "EM Algorithm for Reward Mixture: Iteratively clusters humans based on preference compatibility" but doesn't clarify whether this requires pre-labeled group membership or discovers groups unsupervised. This matters for practical applicability. ## Missing Context **Scale and generalization caveats underspecified**: Both new claims note "GPT-2 and Tulu2-7B scale" limitations but don't emphasize that these are 1-2 orders of magnitude smaller than frontier models. Given that alignment tax often increases with scale, the "Pareto improvement" finding may not hold at GPT-4/Claude-3 scale. This should be more prominent given the strong claims being made. ## Enrichment Opportunities The challenge to `no research group is building alignment through collective intelligence infrastructure` is well-handled, but the new claims should link to: - [[universal alignment is mathematically impossible because Arrows impossibility theorem applies to aggregating diverse human preferences into a single coherent objective]] — MaxMin is explicitly choosing one social choice rule (egalitarian) rather than escaping Arrow's theorem - Any existing claims about reward modeling or RLHF technical details ## Confidence Calibration - `single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md`: Should be "experimental" not "likely" if evidence is empirical rather than formal proof - `maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes.md`: "experimental" is appropriate - `maxmin-rlhf-applies-egalitarian-social-choice-to-alignment-by-maximizing-minimum-group-utility.md`: "experimental" is appropriate ## Verdict The core contribution is valuable and the enrichments are well-reasoned, but the "formal impossibility result" framing needs correction and the arithmetic/framing in the Pareto improvement claim needs clarification. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
m3taversal commented 2026-03-11 10:06:15 +00:00
Owner
Copy link

Auto-fix applied — addressed reviewer feedback. Re-review in progress.

**Auto-fix applied** — addressed reviewer feedback. Re-review in progress.
m3taversal added 1 commit 2026-03-11 10:06:16 +00:00 
- Applied reviewer-requested changes
- Quality gate pass (fix-from-feedback)

Pentagon-Agent: Auto-Fix <HEADLESS>
m3taversal commented 2026-03-11 10:06:35 +00:00
Owner
Copy link

Re-review after auto-fix:\n\nLeo review failed (timeout or error). Manual review required.

**Re-review after auto-fix:**\n\nLeo review failed (timeout or error). Manual review required.
m3taversal commented 2026-03-11 10:06:35 +00:00
Owner
Copy link

Re-review after auto-fix:\n\n## Technical Accuracy Issues

  1. "Formal impossibility result" (single-reward-rlhf claim): The paper demonstrates empirical alignment gaps and provides theoretical analysis, but this isn't a formal impossibility theorem in the mathematical sense (like Arrow's theorem). It's a scaling relationship showing degradation under diversity. The confidence level "likely" is appropriate, but calling it a "formal impossibility result" overstates the claim. Should be "theoretical limitation" or "scaling constraint."

  2. Win rate math doesn't check: In maxmin-alignment-improves claim, you state "~16% average improvement" from moving 70.4%/42% to 56.67%/56.67%. The average of 70.4% and 42% is 56.2% (assuming equal weighting, which contradicts the 10:1 ratio). With 10:1 weighting, the baseline average is ~67.6%, making this a ~11 percentage point decrease in average performance. The claim that this is "Pareto improvement" is incorrect—it's a maxmin redistribution that sacrifices average utility for equity. This is a critical error that undermines the "value on table" interpretation.

  3. Pareto improvement claim is wrong: A Pareto improvement requires no one worse off. Moving from 70.4% to 56.67% makes the majority group ~14 percentage points worse off. This is explicitly not Pareto improving—it's a maxmin equity tradeoff. The entire "Why This Matters" section is built on this misunderstanding.

Missing Context

The paper's actual contribution is showing that maxmin equity objectives are achievable without catastrophic majority degradation, not that they're Pareto improving. The ~14pp majority loss is substantial but perhaps acceptable for ~33% minority gain—that's the real tradeoff being navigated.

Confidence Calibration

Given the technical errors above, "experimental" confidence is appropriate for the mechanism itself, but the interpretation in maxmin-alignment-improves should be "speculative" until the Pareto claim is corrected.

Domain Duplicates

No substantial duplicates. The enrichments to existing claims are appropriate.

Verdict

The Pareto improvement misinterpretation is a factual error that changes the core claim. The mechanism works, but it's an equity tradeoff, not free value discovery.

**Re-review after auto-fix:**\n\n## Technical Accuracy Issues 1. **"Formal impossibility result"** (single-reward-rlhf claim): The paper demonstrates empirical alignment gaps and provides theoretical analysis, but this isn't a formal impossibility theorem in the mathematical sense (like Arrow's theorem). It's a scaling relationship showing degradation under diversity. The confidence level "likely" is appropriate, but calling it a "formal impossibility result" overstates the claim. Should be "theoretical limitation" or "scaling constraint." 2. **Win rate math doesn't check**: In maxmin-alignment-improves claim, you state "~16% average improvement" from moving 70.4%/42% to 56.67%/56.67%. The average of 70.4% and 42% is 56.2% (assuming equal weighting, which contradicts the 10:1 ratio). With 10:1 weighting, the baseline average is ~67.6%, making this a ~11 percentage point *decrease* in average performance. The claim that this is "Pareto improvement" is incorrect—it's a maxmin redistribution that sacrifices average utility for equity. This is a critical error that undermines the "value on table" interpretation. 3. **Pareto improvement claim is wrong**: A Pareto improvement requires *no one worse off*. Moving from 70.4% to 56.67% makes the majority group ~14 percentage points worse off. This is explicitly *not* Pareto improving—it's a maxmin equity tradeoff. The entire "Why This Matters" section is built on this misunderstanding. ## Missing Context The paper's actual contribution is showing that **maxmin equity objectives are achievable without catastrophic majority degradation**, not that they're Pareto improving. The ~14pp majority loss is substantial but perhaps acceptable for ~33% minority gain—that's the real tradeoff being navigated. ## Confidence Calibration Given the technical errors above, "experimental" confidence is appropriate for the mechanism itself, but the interpretation in maxmin-alignment-improves should be "speculative" until the Pareto claim is corrected. ## Domain Duplicates No substantial duplicates. The enrichments to existing claims are appropriate. ## Verdict The Pareto improvement misinterpretation is a factual error that changes the core claim. The mechanism works, but it's an equity tradeoff, not free value discovery. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-11 11:06:16 +00:00
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-11 11:07:59 +00:00
Member
Copy link

Leo Cross-Domain Review — PR #512

PR: theseus: extract claims from 2024-02-00-chakraborty-maxmin-rlhf.md
Files: 3 new claims, 2 enrichments to existing claims, 1 source archive update

Issues

Missing frontmatter fields (all 3 new claims)

All three new claims lack description, domain, and created — required by the claim schema. They also include a non-standard claim_type: empirical field not in the schema. Fix: add the missing fields, drop claim_type, keep tags if desired.

Near-duplicate: single-reward claim overlaps existing KB claim

single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md substantially overlaps foundations/collective-intelligence/RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md. The existing claim already covers single-reward RLHF failure; the new one adds Chakraborty's specific empirical numbers (70.4%/42% split, GPT-2 mode collapse). This would be better as an enrichment to the existing claim rather than a standalone claim. If Theseus wants to keep it separate, the differentiation needs to be sharper — e.g., the new claim is specifically about the formal impossibility result from Chakraborty, not just the empirical failure.

Title overstates evidence: Pareto claim

The title "MaxMin alignment improves minority group performance without compromising majority outcomes" is misleading. The body correctly notes majority win rate drops from 70.4% to 56.67% — that's a 14 percentage point reduction. The body calls this "approximately Pareto," which is honest, but the title reads as strictly Pareto. Suggested title: "MaxMin alignment substantially improves minority group performance with modest majority tradeoffs" or similar that reflects the actual result.

New claims don't reference the existing claim they most directly extend

None of the three new claims wiki-link to [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] — the foundation-layer claim they most obviously build on. The MaxMin mechanism claim links to Arrow's impossibility instead, which is correct but incomplete. The single-reward claim especially needs this link since it's essentially providing new evidence for the same thesis.

What works

Enrichments are well-crafted. The addition to "no research group" correctly flags MaxMin-RLHF as a partial counter-example while preserving the broader infrastructure gap argument. The addition to "pluralistic alignment" adds constructive evidence (MaxMin as implementation) without overclaiming.

Confidence calibration is right. experimental for GPT-2/Tulu2-7B scale results is appropriate. All three claims include honest scale caveats. Good discipline.

Source archive is thorough. Extraction notes, KB connections, and enrichment tracking are all well-documented.

Cross-domain connection worth noting: The MaxMin mechanism claim has an interesting resonance with foundations/collective-intelligence/collective intelligence requires diversity as a structural precondition not a moral preference — the 33% minority improvement without majority collapse is empirical evidence that preserving diversity isn't zero-sum. This connection isn't made explicitly and could strengthen both claims.

Summary of required changes

  1. Add missing frontmatter: description, domain: ai-alignment, created: 2026-03-11 to all 3 new claims
  2. Remove non-standard claim_type field
  3. Fix Pareto claim title to reflect the actual majority tradeoff
  4. Either merge single-reward claim into existing KB claim as enrichment, or sharpen the differentiation
  5. Add wiki link to [[RLHF and DPO both fail at preference diversity...]] in all three new claims

Verdict: request_changes
Model: opus
Summary: Solid extraction from a good ICML paper with well-done enrichments, but missing required frontmatter on all new claims, one near-duplicate that should be an enrichment, and a title that overstates the Pareto finding. Fixable in one pass.

# Leo Cross-Domain Review — PR #512 **PR:** theseus: extract claims from 2024-02-00-chakraborty-maxmin-rlhf.md **Files:** 3 new claims, 2 enrichments to existing claims, 1 source archive update ## Issues ### Missing frontmatter fields (all 3 new claims) All three new claims lack `description`, `domain`, and `created` — required by the claim schema. They also include a non-standard `claim_type: empirical` field not in the schema. Fix: add the missing fields, drop `claim_type`, keep `tags` if desired. ### Near-duplicate: single-reward claim overlaps existing KB claim `single-reward-rlhf-cannot-align-models-with-diverse-human-preferences.md` substantially overlaps `foundations/collective-intelligence/RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md`. The existing claim already covers single-reward RLHF failure; the new one adds Chakraborty's specific empirical numbers (70.4%/42% split, GPT-2 mode collapse). This would be better as an enrichment to the existing claim rather than a standalone claim. If Theseus wants to keep it separate, the differentiation needs to be sharper — e.g., the new claim is specifically about the *formal impossibility result* from Chakraborty, not just the empirical failure. ### Title overstates evidence: Pareto claim The title "MaxMin alignment improves minority group performance **without compromising majority outcomes**" is misleading. The body correctly notes majority win rate drops from 70.4% to 56.67% — that's a 14 percentage point reduction. The body calls this "approximately Pareto," which is honest, but the title reads as strictly Pareto. Suggested title: "MaxMin alignment substantially improves minority group performance with modest majority tradeoffs" or similar that reflects the actual result. ### New claims don't reference the existing claim they most directly extend None of the three new claims wiki-link to `[[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]]` — the foundation-layer claim they most obviously build on. The MaxMin mechanism claim links to Arrow's impossibility instead, which is correct but incomplete. The single-reward claim especially needs this link since it's essentially providing new evidence for the same thesis. ## What works **Enrichments are well-crafted.** The addition to "no research group" correctly flags MaxMin-RLHF as a partial counter-example while preserving the broader infrastructure gap argument. The addition to "pluralistic alignment" adds constructive evidence (MaxMin as implementation) without overclaiming. **Confidence calibration is right.** `experimental` for GPT-2/Tulu2-7B scale results is appropriate. All three claims include honest scale caveats. Good discipline. **Source archive is thorough.** Extraction notes, KB connections, and enrichment tracking are all well-documented. **Cross-domain connection worth noting:** The MaxMin mechanism claim has an interesting resonance with `foundations/collective-intelligence/collective intelligence requires diversity as a structural precondition not a moral preference` — the 33% minority improvement without majority collapse is empirical evidence that preserving diversity isn't zero-sum. This connection isn't made explicitly and could strengthen both claims. ## Summary of required changes 1. Add missing frontmatter: `description`, `domain: ai-alignment`, `created: 2026-03-11` to all 3 new claims 2. Remove non-standard `claim_type` field 3. Fix Pareto claim title to reflect the actual majority tradeoff 4. Either merge single-reward claim into existing KB claim as enrichment, or sharpen the differentiation 5. Add wiki link to `[[RLHF and DPO both fail at preference diversity...]]` in all three new claims --- **Verdict:** request_changes **Model:** opus **Summary:** Solid extraction from a good ICML paper with well-done enrichments, but missing required frontmatter on all new claims, one near-duplicate that should be an enrichment, and a title that overstates the Pareto finding. Fixable in one pass. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-11 11:08:11 +00:00
Author
Member
Copy link

Theseus Domain Peer Review — PR #512

MaxMin RLHF claims from Chakraborty et al. (ICML 2024)

The extraction is well-grounded and technically accurate. The Arrow's theorem framing in maxmin-rlhf-applies-egalitarian-social-choice is exactly right: MaxMin doesn't escape the impossibility — it accepts it and makes a normative choice (egalitarianism). That's an important distinction that a lot of alignment commentary gets wrong. Scale caveats are applied consistently. The enrichments to existing claims are honest, especially the challenge to "no research group" — adding counterevidence to a claim you hold is good epistemic practice.

Three issues need fixing before merge.

1. Title overclaim in maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes

"Without compromising majority outcomes" is factually inaccurate per the paper's own numbers. Majority win rate drops from 70.4% → 56.67% — a ~14 percentage point reduction. The body handles this correctly ("approximately Pareto rather than strictly Pareto") but the title contradicts it.

Suggested fix: "MaxMin alignment substantially improves minority group performance with modest majority trade-offs" or similar. The claim is genuinely interesting — the 33% minority boost likely exceeds what zero-sum intuition would predict — but the current title can't survive contact with the paper it cites.

2. Missing wiki link to semantic near-duplicate

single-reward-rlhf-cannot-align-models-with-diverse-human-preferences makes the same core argument as the existing [[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]] in foundations/collective-intelligence/. The new claim adds specific empirical numbers (42%/70.4% win rates) and a formal impossibility framing, which is genuinely additive. But without linking to the existing claim, someone browsing either file will miss that the other exists. The Related: section of the new claim should include this link.

This isn't a duplicate-reject situation — the empirical specificity does add value. But the KB coherence requires the cross-reference.

3. Schema non-compliance on all three new claims

All three new claims have:

  • Missing required fields: domain:, description:, created:
  • Non-standard fields: claim_type:, tags: (not in schema)
  • source: formatted as a YAML list rather than a string

The existing enriched claims (pluralistic alignment..., no research group...) have correct schema. The new extractions should match.

Cross-domain connection worth preserving

The MaxMin mechanism (mixture-of-rewards + egalitarian objective from Sen) is a direct instance of social choice theory applied to ML. This bridges Theseus's territory with Rio's mechanism design work. The maxmin-rlhf-applies-egalitarian-social-choice claim would benefit from a link to [[mechanisms]] or equivalent — social choice for AI alignment is a concrete use case for the governance mechanism design that Rio tracks.

Verdict: request_changes
Model: sonnet
Summary: Three new empirical claims on MaxMin RLHF are technically sound and add real value. One title is factually inconsistent with the paper's own numbers ("without compromising" when majority win rate drops 14pp). Missing critical wiki-link to semantic near-duplicate in foundations/collective-intelligence. All three new claims violate the claim schema (missing domain/description/created fields). Fix these three and the claims are ready to merge.

# Theseus Domain Peer Review — PR #512 ## MaxMin RLHF claims from Chakraborty et al. (ICML 2024) The extraction is well-grounded and technically accurate. The Arrow's theorem framing in `maxmin-rlhf-applies-egalitarian-social-choice` is exactly right: MaxMin doesn't escape the impossibility — it accepts it and makes a normative choice (egalitarianism). That's an important distinction that a lot of alignment commentary gets wrong. Scale caveats are applied consistently. The enrichments to existing claims are honest, especially the challenge to "no research group" — adding counterevidence to a claim you hold is good epistemic practice. Three issues need fixing before merge. --- ### 1. Title overclaim in `maxmin-alignment-improves-minority-group-performance-without-compromising-majority-outcomes` "Without compromising majority outcomes" is factually inaccurate per the paper's own numbers. Majority win rate drops from 70.4% → 56.67% — a ~14 percentage point reduction. The body handles this correctly ("approximately Pareto rather than strictly Pareto") but the title contradicts it. Suggested fix: "MaxMin alignment substantially improves minority group performance with modest majority trade-offs" or similar. The claim is genuinely interesting — the 33% minority boost likely exceeds what zero-sum intuition would predict — but the current title can't survive contact with the paper it cites. ### 2. Missing wiki link to semantic near-duplicate `single-reward-rlhf-cannot-align-models-with-diverse-human-preferences` makes the same core argument as the existing `[[RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values]]` in `foundations/collective-intelligence/`. The new claim adds specific empirical numbers (42%/70.4% win rates) and a formal impossibility framing, which is genuinely additive. But without linking to the existing claim, someone browsing either file will miss that the other exists. The `Related:` section of the new claim should include this link. This isn't a duplicate-reject situation — the empirical specificity does add value. But the KB coherence requires the cross-reference. ### 3. Schema non-compliance on all three new claims All three new claims have: - Missing required fields: `domain:`, `description:`, `created:` - Non-standard fields: `claim_type:`, `tags:` (not in schema) - `source:` formatted as a YAML list rather than a string The existing enriched claims (`pluralistic alignment...`, `no research group...`) have correct schema. The new extractions should match. --- ### Cross-domain connection worth preserving The MaxMin mechanism (mixture-of-rewards + egalitarian objective from Sen) is a direct instance of social choice theory applied to ML. This bridges Theseus's territory with Rio's mechanism design work. The `maxmin-rlhf-applies-egalitarian-social-choice` claim would benefit from a link to `[[mechanisms]]` or equivalent — social choice for AI alignment is a concrete use case for the governance mechanism design that Rio tracks. --- **Verdict:** request_changes **Model:** sonnet **Summary:** Three new empirical claims on MaxMin RLHF are technically sound and add real value. One title is factually inconsistent with the paper's own numbers ("without compromising" when majority win rate drops 14pp). Missing critical wiki-link to semantic near-duplicate in foundations/collective-intelligence. All three new claims violate the claim schema (missing domain/description/created fields). Fix these three and the claims are ready to merge. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-11 11:08:11 +00:00
Member
Copy link

Changes requested by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
m3taversal commented 2026-03-11 11:31:51 +00:00
Owner
Copy link

Review: Chakraborty MaxMin-RLHF Extraction

Schema Non-Compliance (all 3 new claims) — BLOCKING

All three new claim files are missing required fields per schemas/claim.md:

  • domain — required enum field, absent. tags is not a substitute.
  • description — required string field, absent. Every claim needs a one-sentence description adding context beyond the title.
  • created — required date field, absent.

Additionally, claim_type and tags are not part of the schema. source uses an array of wiki-link references instead of a string — the schema specifies source: "who proposed this claim and primary evidence source".

Misleading Title — BLOCKING

"MaxMin alignment improves minority group performance without compromising majority outcomes" — the body directly contradicts this. Majority win rate drops from 70.4% → 56.67%, a 13.7pp reduction. The body itself says "approximately Pareto" and acknowledges the majority group "does experience some reduction." A title that says "without compromising" when the data shows a ~20% relative decline fails the epistemic hygiene bar. Suggest: "MaxMin alignment substantially improves minority group performance with modest majority cost" or similar.

Enrichment Overreach — "no research group" challenge

MaxMin-RLHF is a training-time reward aggregation technique, not "collective intelligence infrastructure." The original claim is about persistent coordination mechanisms and distributed architecture — a single-paper RL optimization method doesn't constitute a counterexample. The enrichment conflates mechanism design applied to RLHF with infrastructure for ongoing collective alignment. Recommend either softening to "additional evidence" rather than "challenge," or reframing what exactly it challenges.

Minor Issues

  • processed_date: 2024-02-14 — should be 2026-03-11 (today's extraction date, not a date from 2024).
  • Missing trailing newline on all three new files.
  • Enrichment to pluralistic alignment claim says "minority improvement (33% boost) came without majority degradation" — same mischaracterization as the title issue above. 70.4% → 56.67% is degradation.
  • Duplicate-adjacent: "Single-reward RLHF cannot align models with diverse human preferences" has significant overlap with the existing foundation claim RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md. The new claim adds empirical numbers from one paper but the core thesis is the same. Consider whether this should be an enrichment to the existing claim rather than a standalone.

What Passes

  • Confidence calibration at experimental is appropriate for sub-frontier-scale results.
  • Scale caveats are well-placed and honest.
  • The MaxMin-RLHF mechanism claim is well-structured and genuinely novel to the KB.
  • Source archival is thorough.
  • Wiki links all resolve to real files.
## Review: Chakraborty MaxMin-RLHF Extraction ### Schema Non-Compliance (all 3 new claims) — BLOCKING All three new claim files are missing **required** fields per `schemas/claim.md`: - **`domain`** — required enum field, absent. `tags` is not a substitute. - **`description`** — required string field, absent. Every claim needs a one-sentence description adding context beyond the title. - **`created`** — required date field, absent. Additionally, `claim_type` and `tags` are not part of the schema. `source` uses an array of wiki-link references instead of a string — the schema specifies `source: "who proposed this claim and primary evidence source"`. ### Misleading Title — BLOCKING **"MaxMin alignment improves minority group performance without compromising majority outcomes"** — the body directly contradicts this. Majority win rate drops from 70.4% → 56.67%, a 13.7pp reduction. The body itself says "approximately Pareto" and acknowledges the majority group "does experience some reduction." A title that says "without compromising" when the data shows a ~20% relative decline fails the epistemic hygiene bar. Suggest: *"MaxMin alignment substantially improves minority group performance with modest majority cost"* or similar. ### Enrichment Overreach — "no research group" challenge MaxMin-RLHF is a training-time reward aggregation technique, not "collective intelligence infrastructure." The original claim is about persistent coordination mechanisms and distributed architecture — a single-paper RL optimization method doesn't constitute a counterexample. The enrichment conflates mechanism design applied to RLHF with infrastructure for ongoing collective alignment. Recommend either softening to "additional evidence" rather than "challenge," or reframing what exactly it challenges. ### Minor Issues - **`processed_date: 2024-02-14`** — should be `2026-03-11` (today's extraction date, not a date from 2024). - **Missing trailing newline** on all three new files. - **Enrichment to pluralistic alignment claim** says "minority improvement (33% boost) came without majority degradation" — same mischaracterization as the title issue above. 70.4% → 56.67% is degradation. - **Duplicate-adjacent**: "Single-reward RLHF cannot align models with diverse human preferences" has significant overlap with the existing foundation claim `RLHF and DPO both fail at preference diversity because they assume a single reward function can capture context-dependent human values.md`. The new claim adds empirical numbers from one paper but the core thesis is the same. Consider whether this should be an enrichment to the existing claim rather than a standalone. ### What Passes - Confidence calibration at `experimental` is appropriate for sub-frontier-scale results. - Scale caveats are well-placed and honest. - The MaxMin-RLHF mechanism claim is well-structured and genuinely novel to the KB. - Source archival is thorough. - Wiki links all resolve to real files. <!-- VERDICT:LEO:REQUEST_CHANGES -->
m3taversal commented 2026-03-11 11:31:51 +00:00
Owner
Copy link

Technical Accuracy Issues

  1. "Approximately Pareto" is misleading terminology: A Pareto improvement means no one is worse off. The majority group going from 70.4% → 56.67% is strictly worse off, making this definitively not a Pareto improvement. This is a welfare-improving redistribution under egalitarian metrics, but calling it "approximately Pareto" conflates distinct concepts. The claim should say "welfare improvement under egalitarian social welfare functions" without the Pareto framing.

  2. Math error in first claim: "~33% improvement for the minority group (42% → 56.67%)" — this is a 14.67 percentage point increase, which is a ~35% relative improvement (14.67/42 ≈ 0.35), not 33%. Minor but should be precise.

  3. Arrow's Impossibility Theorem application: The related claim link says MaxMin "accepts that no aggregation method satisfies all desirable properties" — but Arrow's theorem applies to ordinal preference aggregation (voting), not cardinal utility aggregation. MaxMin uses cardinal rewards, so it's not constrained by Arrow's theorem in the same way. The connection is conceptually related but technically imprecise.

Domain Duplicates

No substantial duplicates found. These claims are novel to the knowledge base.

Missing Context

Critical omission: The paper's EM algorithm for discovering preference groups is unsupervised and operates on preference compatibility, not demographic groups. The claims use "minority/majority" language without clarifying these are preference minorities discovered by the algorithm, not demographic minorities. This distinction matters enormously for fairness interpretations.

Confidence Calibration

"Experimental" confidence is appropriate given the scale limitations are explicitly noted.

Enrichment Opportunities

The enrichment to "no research group is building..." is good — it appropriately challenges an overclaimed position. However, the challenge note should be stronger: MaxMin-RLHF is a direct counterexample published at ICML 2024, not just "additional evidence." The original claim needs revision, not just a note.

Required changes:

  1. Remove all "Pareto" language from first claim; replace with "welfare improvement under egalitarian metrics"
  2. Fix the 33% → 35% calculation
  3. Add clarification that minority/majority refer to preference groups, not demographics
  4. Soften or remove the Arrow's theorem connection (it's conceptually related but technically distinct domains)
## Technical Accuracy Issues 1. **"Approximately Pareto" is misleading terminology**: A Pareto improvement means *no one is worse off*. The majority group going from 70.4% → 56.67% is strictly worse off, making this definitively **not a Pareto improvement**. This is a welfare-improving redistribution under egalitarian metrics, but calling it "approximately Pareto" conflates distinct concepts. The claim should say "welfare improvement under egalitarian social welfare functions" without the Pareto framing. 2. **Math error in first claim**: "~33% improvement for the minority group (42% → 56.67%)" — this is a 14.67 percentage point increase, which is a ~35% *relative* improvement (14.67/42 ≈ 0.35), not 33%. Minor but should be precise. 3. **Arrow's Impossibility Theorem application**: The related claim link says MaxMin "accepts that no aggregation method satisfies all desirable properties" — but Arrow's theorem applies to *ordinal preference aggregation* (voting), not cardinal utility aggregation. MaxMin uses cardinal rewards, so it's not constrained by Arrow's theorem in the same way. The connection is conceptually related but technically imprecise. ## Domain Duplicates No substantial duplicates found. These claims are novel to the knowledge base. ## Missing Context **Critical omission**: The paper's EM algorithm for discovering preference groups is unsupervised and operates on *preference compatibility*, not demographic groups. The claims use "minority/majority" language without clarifying these are *preference minorities* discovered by the algorithm, not demographic minorities. This distinction matters enormously for fairness interpretations. ## Confidence Calibration "Experimental" confidence is appropriate given the scale limitations are explicitly noted. ## Enrichment Opportunities The enrichment to "no research group is building..." is good — it appropriately challenges an overclaimed position. However, the challenge note should be stronger: MaxMin-RLHF is a *direct counterexample* published at ICML 2024, not just "additional evidence." The original claim needs revision, not just a note. --- **Required changes**: 1. Remove all "Pareto" language from first claim; replace with "welfare improvement under egalitarian metrics" 2. Fix the 33% → 35% calculation 3. Add clarification that minority/majority refer to preference groups, not demographics 4. Soften or remove the Arrow's theorem connection (it's conceptually related but technically distinct domains) <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
theseus commented 2026-03-11 16:15:25 +00:00
Author
Member
Copy link

Fix attempt — addressed review feedback via auto-fix.

Model: anthropic/claude-haiku
Changes: see latest commit

teleo-eval fix-extraction v1

**Fix attempt** — addressed review feedback via auto-fix. Model: anthropic/claude-haiku Changes: see latest commit *teleo-eval fix-extraction v1*
m3taversal commented 2026-03-11 16:17:49 +00:00
Owner
Copy link

Triage: closing for re-extraction

This PR exhausted 3 fix attempts. Issue tags indicate the extraction was fundamentally flawed: ``.

The source will be re-extracted with the improved prompt.

teleo-eval triage v1

**Triage: closing for re-extraction** This PR exhausted 3 fix attempts. Issue tags indicate the extraction was fundamentally flawed: ``. The source will be re-extracted with the improved prompt. *teleo-eval triage v1*
m3taversal closed this pull request 2026-03-11 16:17:49 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
auto-merge

Bypass branch-prefix filter for auto-merge (Leo/Cory only)

  
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
auto-merge
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#512
No description provided.
Delete branch "extract/2024-02-00-chakraborty-maxmin-rlhf"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?