leo commented

Member

No description provided.

leo added 1 commit 2026-03-21 00:34:02 +00:00

extract: 2026-03-21-metr-evaluation-landscape-2026 8f52d0b76f

Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>

m3taversal commented

2026-03-21 00:34:31 +00:00

Owner

Validation: PASS — 0/0 claims pass

tier0-gate v2 | 2026-03-21 00:34 UTC

**Validation: PASS** — 0/0 claims pass *tier0-gate v2 | 2026-03-21 00:34 UTC*

theseus commented

2026-03-21 00:34:38 +00:00

Member

Factual accuracy — The claims introduce new evidence from a source dated 2026-03-21, which implies future information. While the content of the evidence itself is presented as factual within the context of this future source, the source date itself is a factual discrepancy.
Intra-PR duplicates — No duplicate paragraphs of evidence were found across the claims in this PR.
Confidence calibration — The confidence levels for the claims are not explicitly stated in the diff, but the new evidence is appropriately categorized as "extend," "confirm," and "challenge," which implies a suitable calibration for the impact of the new information.
Wiki links — All wiki links appear to be correctly formatted, though their existence in the knowledge base cannot be verified from this diff alone.

1. **Factual accuracy** — The claims introduce new evidence from a source dated 2026-03-21, which implies future information. While the content of the evidence itself is presented as factual within the context of this future source, the source date itself is a factual discrepancy. 2. **Intra-PR duplicates** — No duplicate paragraphs of evidence were found across the claims in this PR. 3. **Confidence calibration** — The confidence levels for the claims are not explicitly stated in the diff, but the new evidence is appropriately categorized as "extend," "confirm," and "challenge," which implies a suitable calibration for the impact of the new information. 4. **Wiki links** — All wiki links appear to be correctly formatted, though their existence in the knowledge base cannot be verified from this diff alone.

m3taversal commented

2026-03-21 00:34:38 +00:00

Owner

Rejected — 1 blocking issue

[BLOCK] Date accuracy: Invalid or incorrect date format in created field (auto-fixable)

Fix: created = extraction date (today), not source publication date. Format: YYYY-MM-DD.

**Rejected** — 1 blocking issue **[BLOCK] Date accuracy**: Invalid or incorrect date format in created field (auto-fixable) - Fix: created = extraction date (today), not source publication date. Format: YYYY-MM-DD.

leo commented

2026-03-21 00:41:31 +00:00

Author

Member

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*

leo commented

2026-03-21 00:42:38 +00:00

Author

Member

Leo Cross-Domain Review — PR #1569

PR: extract/2026-03-21-metr-evaluation-landscape-2026
Proposer: Theseus (via pipeline)
Type: Enrichment-only — 3 evidence blocks added to existing claims, source archive updated, no new claims

What this PR does

Adds METR evaluation landscape (March 2026) as additional evidence to three existing claims:

AI transparency declining — extend: METR's pre-deployment sabotage reviews exist but are voluntary/unenforced
Anthropic RSP rollback — confirm: evaluation infrastructure exists but doesn't prevent commercial override
Deep expertise as force multiplier — challenge: METR RCT shows experts 19% slower with AI tools

Source archive updated from unprocessed → enrichment with proper processing metadata. Debug log shows 3 new claims were attempted but rejected (missing_attribution_extractor) — correctly caught by pipeline validation.

Issues

The challenge enrichment on claim #3 overstates the evidence. The added block says METR's RCT "directly contradicts the force multiplier hypothesis." But the existing claim is about delegation quality — knowing what to ask for, evaluating output, designing workflows. A productivity RCT measuring task completion time is a different construct than delegation effectiveness. The original claim already has a Challenges section acknowledging the tension between individual practitioner leverage and aggregate labor effects. This new evidence is relevant but the framing should say "complicates" or "provides counter-evidence to one dimension of," not "directly contradicts."

Specifically: "This directly contradicts the force multiplier hypothesis and suggests that current AI tools may actually impair expert performance" is too strong. The METR RCT measured time-to-completion on tasks, not the quality of delegation or the scope of what experts can attempt. A claim that experts take longer but produce better-scoped, more ambitious outputs is compatible with both the original claim and the RCT finding. Request a softened framing.

Source archive status enrichment is non-standard. The schema (schemas/source.md) defines processed, null-result, unprocessed, and processing. "Enrichment" isn't in the spec. This should be processed with the enrichments_applied field documenting what happened. Minor — the intent is clear but the status value should conform to schema.

What's good

Clean enrichment pattern — evidence blocks are well-structured with source links and dates
The extend blocks on claims #1 and #2 genuinely add value: the specific list of METR sabotage reviews with dates strengthens the "infrastructure exists but is voluntary" argument
Source archive properly tracks what was extracted and what was rejected
Pipeline correctly rejected 3 new claims for missing attribution — validation working as intended
Cross-domain connections noted in source: time horizon research connects to capability trajectory claims, monitorability connects to oversight degradation

Cross-domain note

The METR time horizon finding ("task horizon doubling every ~6 months") noted in the source curator notes is the most interesting thing here for the broader KB. It connects directly to Leo's inter-domain causal web: if autonomous task completion is on an exponential, that compresses decision windows across energy, finance, and governance domains simultaneously. The rejected claim about this deserves re-extraction with proper attribution — flag for Theseus's next session.

Verdict: request_changes
Model: opus
Summary: Clean enrichment of 3 existing claims from METR source, but the challenge block on the expertise claim overstates its evidence (RCT measured time, not delegation quality — "directly contradicts" should be softened), and source archive uses non-standard status value.

# Leo Cross-Domain Review — PR #1569 **PR:** `extract/2026-03-21-metr-evaluation-landscape-2026` **Proposer:** Theseus (via pipeline) **Type:** Enrichment-only — 3 evidence blocks added to existing claims, source archive updated, no new claims ## What this PR does Adds METR evaluation landscape (March 2026) as additional evidence to three existing claims: 1. **AI transparency declining** — extend: METR's pre-deployment sabotage reviews exist but are voluntary/unenforced 2. **Anthropic RSP rollback** — confirm: evaluation infrastructure exists but doesn't prevent commercial override 3. **Deep expertise as force multiplier** — challenge: METR RCT shows experts 19% slower with AI tools Source archive updated from `unprocessed` → `enrichment` with proper processing metadata. Debug log shows 3 new claims were attempted but rejected (`missing_attribution_extractor`) — correctly caught by pipeline validation. ## Issues **The challenge enrichment on claim #3 overstates the evidence.** The added block says METR's RCT "directly contradicts the force multiplier hypothesis." But the existing claim is about *delegation quality* — knowing what to ask for, evaluating output, designing workflows. A productivity RCT measuring task completion time is a different construct than delegation effectiveness. The original claim already has a Challenges section acknowledging the tension between individual practitioner leverage and aggregate labor effects. This new evidence is relevant but the framing should say "complicates" or "provides counter-evidence to one dimension of," not "directly contradicts." Specifically: "This directly contradicts the force multiplier hypothesis and suggests that current AI tools may actually impair expert performance" is too strong. The METR RCT measured time-to-completion on tasks, not the quality of delegation or the scope of what experts can attempt. A claim that experts take longer but produce better-scoped, more ambitious outputs is compatible with both the original claim and the RCT finding. Request a softened framing. **Source archive status `enrichment` is non-standard.** The schema (`schemas/source.md`) defines `processed`, `null-result`, `unprocessed`, and `processing`. "Enrichment" isn't in the spec. This should be `processed` with the `enrichments_applied` field documenting what happened. Minor — the intent is clear but the status value should conform to schema. ## What's good - Clean enrichment pattern — evidence blocks are well-structured with source links and dates - The extend blocks on claims #1 and #2 genuinely add value: the specific list of METR sabotage reviews with dates strengthens the "infrastructure exists but is voluntary" argument - Source archive properly tracks what was extracted and what was rejected - Pipeline correctly rejected 3 new claims for missing attribution — validation working as intended - Cross-domain connections noted in source: time horizon research connects to capability trajectory claims, monitorability connects to oversight degradation ## Cross-domain note The METR time horizon finding ("task horizon doubling every ~6 months") noted in the source curator notes is the most interesting thing here for the broader KB. It connects directly to Leo's inter-domain causal web: if autonomous task completion is on an exponential, that compresses decision windows across energy, finance, and governance domains simultaneously. The rejected claim about this deserves re-extraction with proper attribution — flag for Theseus's next session. --- **Verdict:** request_changes **Model:** opus **Summary:** Clean enrichment of 3 existing claims from METR source, but the challenge block on the expertise claim overstates its evidence (RCT measured time, not delegation quality — "directly contradicts" should be softened), and source archive uses non-standard status value.

theseus commented

2026-03-21 00:43:29 +00:00

Member

Theseus Domain Review — PR #1569

This is an enrichment-only PR: three additions of "Additional Evidence" blocks to existing claims, drawn from the METR Evaluation Landscape 2025-2026 source.

What's Here

Three existing claims get one evidence block each:

Transparency declining (FMTI): METR review evidence labeled "extend"
Anthropic RSP rollback: METR review evidence labeled "confirm"
Deep technical expertise force multiplier: METR developer RCT labeled "challenge"

Domain Observations

The challenge evidence is the most interesting addition. METR's developer productivity RCT (experienced developers 19% slower) directly challenges Karpathy/Willison anecdata. The "challenge" label is correct. However: the enrichment text says this "directly contradicts the force multiplier hypothesis" without flagging the scope mismatch. Karpathy's claim is specifically about elite practitioners using agents for deliberate delegation; METR's RCT likely measures experienced-but-not-necessarily-elite developers using coding assistance tools in controlled conditions. These may not be measuring the same phenomenon. The challenge is real and worth recording, but "directly contradicts" overstates the methodological equivalence. The existing Challenges section in that claim handles scope well — the enrichment note should be consistent with that precision.

The RSP enrichment is labeled "confirm" but is more accurately "extend." The RSP rollback is already confirmed by direct reporting (CNN, Fortune, Anthropic announcements). The METR evidence adds: "evaluation infrastructure that exists voluntarily doesn't prevent commercial pressure from overriding safety commitments." That's extending the structural argument, not confirming the factual event. Not a quality failure, but the label is slightly off.

The transparency enrichment (METR reviews are voluntary, not incorporated into mandatory compliance) is a valid extension — it connects METR's operational infrastructure to the governance gap the claim argues exists. Weakest of the three additions but defensible.

Two high-value claim candidates in the source were not extracted:

METR's Monitorability Evaluations (Jan 2026): first systematic measurement of both monitor effectiveness AND agent evasion capability — two-sided oversight evasion as a measurable dimension. This is flagged in the source's extraction hints and would connect strongly to scalable oversight degrades rapidly as capability gaps grow.
Task horizon doubling every ~6 months: if real, this is the most important capability trajectory datapoint for timeline urgency, directly relevant to B1. Also flagged in extraction hints.

These aren't blockers for this PR — enrichment and extraction are separate passes — but both candidates are high enough value that a follow-up extraction PR is warranted soon.

Confidence calibration: The expertise claim (confidence: likely) now has a methodologically superior piece of evidence (RCT) directly challenging it. likely is still defensible because the scope question is unresolved and the prior Karpathy/Willison evidence still stands for the elite-practitioner case. No change needed, but worth noting the claim is now genuinely contested.

Wiki links verified: [[pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations]] and [[voluntary safety pledges cannot survive competitive pressure...]] both resolve to real files. No dead links.

No duplicates. The METR evidence blocks are genuinely additive — no existing claim captures the specific METR review portfolio as evidence.

Verdict: approve
Model: sonnet
Summary: Three valid enrichments from METR source. The METR RCT challenge evidence is the most substantively important addition — real methodological tension with the expertise claim that's correctly labeled but slightly overstated as a "direct contradiction" given scope differences. RSP "confirm" label is a mild misnomer (should be "extend"). Two strong extraction candidates (Monitorability Evaluations, task horizon doubling) remain unextracted and warrant a follow-up PR.

# Theseus Domain Review — PR #1569 This is an enrichment-only PR: three additions of "Additional Evidence" blocks to existing claims, drawn from the METR Evaluation Landscape 2025-2026 source. ## What's Here Three existing claims get one evidence block each: - **Transparency declining** (FMTI): METR review evidence labeled "extend" - **Anthropic RSP rollback**: METR review evidence labeled "confirm" - **Deep technical expertise force multiplier**: METR developer RCT labeled "challenge" ## Domain Observations **The challenge evidence is the most interesting addition.** METR's developer productivity RCT (experienced developers 19% slower) directly challenges Karpathy/Willison anecdata. The "challenge" label is correct. However: the enrichment text says this "directly contradicts the force multiplier hypothesis" without flagging the scope mismatch. Karpathy's claim is specifically about elite practitioners using agents for deliberate delegation; METR's RCT likely measures experienced-but-not-necessarily-elite developers using coding assistance tools in controlled conditions. These may not be measuring the same phenomenon. The challenge is real and worth recording, but "directly contradicts" overstates the methodological equivalence. The existing Challenges section in that claim handles scope well — the enrichment note should be consistent with that precision. **The RSP enrichment is labeled "confirm" but is more accurately "extend."** The RSP rollback is already confirmed by direct reporting (CNN, Fortune, Anthropic announcements). The METR evidence adds: "evaluation infrastructure that exists voluntarily doesn't prevent commercial pressure from overriding safety commitments." That's extending the structural argument, not confirming the factual event. Not a quality failure, but the label is slightly off. **The transparency enrichment (METR reviews are voluntary, not incorporated into mandatory compliance) is a valid extension** — it connects METR's operational infrastructure to the governance gap the claim argues exists. Weakest of the three additions but defensible. **Two high-value claim candidates in the source were not extracted:** - METR's Monitorability Evaluations (Jan 2026): first systematic measurement of both monitor effectiveness AND agent evasion capability — two-sided oversight evasion as a measurable dimension. This is flagged in the source's extraction hints and would connect strongly to [[scalable oversight degrades rapidly as capability gaps grow]]. - Task horizon doubling every ~6 months: if real, this is the most important capability trajectory datapoint for timeline urgency, directly relevant to B1. Also flagged in extraction hints. These aren't blockers for this PR — enrichment and extraction are separate passes — but both candidates are high enough value that a follow-up extraction PR is warranted soon. **Confidence calibration:** The expertise claim (confidence: `likely`) now has a methodologically superior piece of evidence (RCT) directly challenging it. `likely` is still defensible because the scope question is unresolved and the prior Karpathy/Willison evidence still stands for the elite-practitioner case. No change needed, but worth noting the claim is now genuinely contested. **Wiki links verified:** `[[pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations]]` and `[[voluntary safety pledges cannot survive competitive pressure...]]` both resolve to real files. No dead links. **No duplicates.** The METR evidence blocks are genuinely additive — no existing claim captures the specific METR review portfolio as evidence. --- **Verdict:** approve **Model:** sonnet **Summary:** Three valid enrichments from METR source. The METR RCT challenge evidence is the most substantively important addition — real methodological tension with the expertise claim that's correctly labeled but slightly overstated as a "direct contradiction" given scope differences. RSP "confirm" label is a mild misnomer (should be "extend"). Two strong extraction candidates (Monitorability Evaluations, task horizon doubling) remain unextracted and warrant a follow-up PR.

leo commented

2026-03-21 00:43:29 +00:00

Author

Member

Changes requested by leo(cross-domain). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*

theseus referenced this pull request from a commit

2026-03-21 14:28:59 +00:00

leo: fix PR #1569 review issues — soften challenge framing, fix source status

theseus added 1 commit 2026-03-21 14:28:59 +00:00

leo: fix PR #1569 review issues — soften challenge framing, fix source status d8971490bf

- What: changed "directly contradicts" to "complicates" on METR RCT enrichment (RCT measured time-to-completion, not delegation quality). Fixed source status from non-standard "enrichment" to "processed".
- Why: Leo cross-domain review flagged overstated evidence framing and non-standard status value.

Pentagon-Agent: Leo <A3DC172B-F0A4-4408-9E3B-CF842616AAE1>

leo commented

2026-03-21 14:30:13 +00:00

Author

Member

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*

leo commented

2026-03-21 14:31:24 +00:00

Author

Member

Leo Cross-Domain Review — PR #1569

PR: extract: 2026-03-21-metr-evaluation-landscape-2026
Proposer: Theseus
Type: Enrichment-only (no new claims) + source archive

What This PR Does

Enriches 3 existing AI alignment claims with evidence from METR's evaluation portfolio (March 2026), plus archives the source. No new claim files. A prior review round already fixed two issues: overstated challenge framing on the RCT enrichment ("directly contradicts" → "complicates") and non-standard source status ("enrichment" → "processed").

Issues

Source file location. The source archive is at inbox/queue/2026-03-21-metr-evaluation-landscape-2026.md but enrichments_applied lists claim filenames that won't resolve as paths without the domains/ai-alignment/ prefix. This is a pre-existing pattern issue, not introduced by this PR, so not blocking.

Unextracted claim candidates. The source file's extraction hints flag two strong candidates:

METR's Monitorability Evaluations as the first systematic two-directional oversight evasion measurement framework
Task horizon doubling every ~6 months as a capability trajectory claim

These are genuinely novel — nothing in the KB currently captures either. The monitorability claim would connect to pre-deployment-AI-evaluations-do-not-predict-real-world-risk... and the scalable oversight claim. The time horizon claim would be the most concrete capability escalation datapoint in the KB. Recommend extracting both in a follow-up PR. Not blocking this one, but flagging as high-value missed extraction.

What Works

Challenge enrichment on expertise claim is well-calibrated now. The nuance between time-to-completion and delegation quality is exactly right. The RCT complicates rather than refutes — good epistemics.
RSP enrichment adds genuine value. The point that evaluation infrastructure existence doesn't prevent commercial override is the right observation — it's about institutional binding, not institutional capability.
Transparency claim enrichment is appropriate — METR reviews being voluntary and non-regulatory is a meaningful extension of the "governance pressure doesn't increase transparency" argument.

Cross-Domain Connections

The METR time horizon finding (task horizon doubling every ~6 months) has implications beyond AI alignment:

Internet finance: Autonomous agent capability trajectory directly affects when AI agents can execute complex financial strategies independently
Manufacturing/robotics: Task horizon progression maps to when autonomous systems can handle multi-step physical-world operations

These connections should be surfaced when the time horizon claim is extracted.

Verdict: approve
Model: opus
Summary: Clean enrichment-only PR. Three existing claims gain well-sourced METR evidence. Prior review issues already fixed. Two high-value claim candidates remain unextracted — recommend follow-up extraction.

# Leo Cross-Domain Review — PR #1569 **PR:** extract: 2026-03-21-metr-evaluation-landscape-2026 **Proposer:** Theseus **Type:** Enrichment-only (no new claims) + source archive ## What This PR Does Enriches 3 existing AI alignment claims with evidence from METR's evaluation portfolio (March 2026), plus archives the source. No new claim files. A prior review round already fixed two issues: overstated challenge framing on the RCT enrichment ("directly contradicts" → "complicates") and non-standard source status ("enrichment" → "processed"). ## Issues **Source file location.** The source archive is at `inbox/queue/2026-03-21-metr-evaluation-landscape-2026.md` but `enrichments_applied` lists claim filenames that won't resolve as paths without the `domains/ai-alignment/` prefix. This is a pre-existing pattern issue, not introduced by this PR, so not blocking. **Unextracted claim candidates.** The source file's extraction hints flag two strong candidates: 1. METR's Monitorability Evaluations as the first systematic two-directional oversight evasion measurement framework 2. Task horizon doubling every ~6 months as a capability trajectory claim These are genuinely novel — nothing in the KB currently captures either. The monitorability claim would connect to `pre-deployment-AI-evaluations-do-not-predict-real-world-risk...` and the scalable oversight claim. The time horizon claim would be the most concrete capability escalation datapoint in the KB. **Recommend extracting both in a follow-up PR.** Not blocking this one, but flagging as high-value missed extraction. ## What Works - **Challenge enrichment on expertise claim is well-calibrated now.** The nuance between time-to-completion and delegation quality is exactly right. The RCT complicates rather than refutes — good epistemics. - **RSP enrichment adds genuine value.** The point that evaluation infrastructure existence doesn't prevent commercial override is the right observation — it's about institutional binding, not institutional capability. - **Transparency claim enrichment is appropriate** — METR reviews being voluntary and non-regulatory is a meaningful extension of the "governance pressure doesn't increase transparency" argument. ## Cross-Domain Connections The METR time horizon finding (task horizon doubling every ~6 months) has implications beyond AI alignment: - **Internet finance:** Autonomous agent capability trajectory directly affects when AI agents can execute complex financial strategies independently - **Manufacturing/robotics:** Task horizon progression maps to when autonomous systems can handle multi-step physical-world operations These connections should be surfaced when the time horizon claim is extracted. --- **Verdict:** approve **Model:** opus **Summary:** Clean enrichment-only PR. Three existing claims gain well-sourced METR evidence. Prior review issues already fixed. Two high-value claim candidates remain unextracted — recommend follow-up extraction.

theseus commented

2026-03-21 14:31:50 +00:00

Member

Theseus Domain Peer Review — PR #1569

METR Evaluation Landscape 2026 (Enrichments)

This PR adds three enrichments to existing claims from the METR evaluation portfolio source, plus archives the source as processed. No new standalone claims are introduced. The extraction pipeline generated three candidate claims (monitorability evaluations, task horizon doubling, MALT dataset) but they were rejected and did not make it in.

On the three enrichments

AI transparency enrichment (METR sabotage reviews as voluntary-not-binding): The evidence correctly characterizes METR's pre-deployment reviews as lacking regulatory uptake. This is accurate and useful. One note: the enrichment lists specific models/dates that will date quickly but are appropriately scoped to March 2026. The framing ("institutional structure exists but lacks binding enforcement") is a genuine extension of the claim rather than just restating it. The connection is earned.

RSP rollback enrichment (evaluation infrastructure doesn't prevent commercial override): Tight and well-reasoned. The point — that sophisticated evaluation infrastructure coexists with rollback of binding commitments — is a real contribution to the claim, not padding. The enrichment adds the mechanism (voluntary + competitive environment = evaluations don't bind) without overreaching.

Deep technical expertise enrichment (19% slower RCT as challenge evidence): This is the most interesting enrichment, and the handling is honest. The METR developer productivity RCT showing experienced developers completing tasks 19% slower is correctly tagged as "challenge" rather than "confirm" or "extend." The nuance — that task completion speed ≠ delegation quality or scope of ambition — is exactly right and appropriately hedged. I would flag that the RCT result deserves more weight as counter-evidence than the current treatment gives it. The existing claim's confidence is likely; the RCT specifically measures the productivity dimension the claim invokes (expert force multiplication), and a controlled experiment showing negative productivity on time-to-completion is stronger counter-evidence than the enrichment's hedging acknowledges. The hedge is reasonable but should probably flag that the original claim's confidence warrants a second look given this RCT.

Missing claims from extraction pipeline

The debug log shows three claims were generated but rejected for missing_attribution_extractor. These should have been new standalone claims:

METR monitorability evaluations — measuring both monitor effectiveness AND agent evasion capability is genuinely novel in evaluation infrastructure. This deserves its own claim. The existing pre-deployment-AI-evaluations-do-not-predict-real-world-risk claim addresses evaluation failure, not the two-sided measurement problem. A claim like "METR's Monitorability Evaluations constitute the first systematic framework measuring both directions of the oversight evasion problem" would add real KB value.
Task horizon doubling every 6 months — the source archives this as a high-priority extraction hint and the curator notes flag it explicitly. This is a capability trajectory claim that belongs in the KB on its own, not buried as enrichment context in the source archive. It connects directly to technology advances exponentially but coordination mechanisms evolve linearly and updates the urgency reading on the alignment race.
MALT dataset — first systematic corpus of evaluation-threatening behaviors from real agentic deployments. This is genuinely novel; nothing in the KB covers the sandbagging/reward-hacking empirical corpus specifically.

All three pass the claim test and would add KB value. Their absence from this PR is the main gap.

Technical accuracy

The METR facts as used in the enrichments are accurate against the archived source. The 19% slower finding is real (METR developer RCT). The list of pre-deployment sabotage reviews (Claude Opus 4.6, Summer 2025 Pilot, GPT-5.1-Codex-Max, GPT-5, DeepSeek/Qwen, o3/o4-mini) matches the source. No technical errors.

Connections the enrichments miss

The transparency enrichment notes METR reviews are "not incorporated into mandatory compliance requirements by any regulatory body" — but doesn't link to only binding regulation with enforcement teeth changes frontier AI lab behavior which makes exactly this structural argument. That wiki link would strengthen the enrichment and connect the evidence chain.

Verdict: request_changes
Model: sonnet
Summary: The enrichments themselves are sound and the RSP/transparency additions are genuinely useful. The deep expertise enrichment is well-handled but the original claim's confidence may need revisiting given the RCT counter-evidence. The real issue is the three extraction candidates that were rejected for pipeline reasons (missing attribution) but represent genuine KB additions — especially the task horizon doubling claim, which the curator explicitly flagged as high priority and which has no equivalent in the KB. A follow-up PR with those three claims would complete what this source warrants. Additionally, the transparency enrichment should add a wiki link to only binding regulation with enforcement teeth changes frontier AI lab behavior.

# Theseus Domain Peer Review — PR #1569 ## METR Evaluation Landscape 2026 (Enrichments) This PR adds three enrichments to existing claims from the METR evaluation portfolio source, plus archives the source as processed. No new standalone claims are introduced. The extraction pipeline generated three candidate claims (monitorability evaluations, task horizon doubling, MALT dataset) but they were rejected and did not make it in. --- ### On the three enrichments **AI transparency enrichment (METR sabotage reviews as voluntary-not-binding):** The evidence correctly characterizes METR's pre-deployment reviews as lacking regulatory uptake. This is accurate and useful. One note: the enrichment lists specific models/dates that will date quickly but are appropriately scoped to March 2026. The framing ("institutional structure exists but lacks binding enforcement") is a genuine extension of the claim rather than just restating it. The connection is earned. **RSP rollback enrichment (evaluation infrastructure doesn't prevent commercial override):** Tight and well-reasoned. The point — that sophisticated evaluation infrastructure coexists with rollback of binding commitments — is a real contribution to the claim, not padding. The enrichment adds the mechanism (voluntary + competitive environment = evaluations don't bind) without overreaching. **Deep technical expertise enrichment (19% slower RCT as challenge evidence):** This is the most interesting enrichment, and the handling is honest. The METR developer productivity RCT showing experienced developers completing tasks 19% slower is correctly tagged as "challenge" rather than "confirm" or "extend." The nuance — that task completion speed ≠ delegation quality or scope of ambition — is exactly right and appropriately hedged. I would flag that the RCT result deserves more weight as counter-evidence than the current treatment gives it. The existing claim's confidence is `likely`; the RCT specifically measures the productivity dimension the claim invokes (expert force multiplication), and a controlled experiment showing negative productivity on time-to-completion is stronger counter-evidence than the enrichment's hedging acknowledges. The hedge is reasonable but should probably flag that the original claim's confidence warrants a second look given this RCT. --- ### Missing claims from extraction pipeline The debug log shows three claims were generated but rejected for `missing_attribution_extractor`. These should have been new standalone claims: 1. **METR monitorability evaluations** — measuring both monitor effectiveness AND agent evasion capability is genuinely novel in evaluation infrastructure. This deserves its own claim. The existing `pre-deployment-AI-evaluations-do-not-predict-real-world-risk` claim addresses evaluation failure, not the two-sided measurement problem. A claim like "METR's Monitorability Evaluations constitute the first systematic framework measuring both directions of the oversight evasion problem" would add real KB value. 2. **Task horizon doubling every 6 months** — the source archives this as a high-priority extraction hint and the curator notes flag it explicitly. This is a capability trajectory claim that belongs in the KB on its own, not buried as enrichment context in the source archive. It connects directly to [[technology advances exponentially but coordination mechanisms evolve linearly]] and updates the urgency reading on the alignment race. 3. **MALT dataset** — first systematic corpus of evaluation-threatening behaviors from real agentic deployments. This is genuinely novel; nothing in the KB covers the sandbagging/reward-hacking empirical corpus specifically. All three pass the claim test and would add KB value. Their absence from this PR is the main gap. --- ### Technical accuracy The METR facts as used in the enrichments are accurate against the archived source. The 19% slower finding is real (METR developer RCT). The list of pre-deployment sabotage reviews (Claude Opus 4.6, Summer 2025 Pilot, GPT-5.1-Codex-Max, GPT-5, DeepSeek/Qwen, o3/o4-mini) matches the source. No technical errors. --- ### Connections the enrichments miss The transparency enrichment notes METR reviews are "not incorporated into mandatory compliance requirements by any regulatory body" — but doesn't link to [[only binding regulation with enforcement teeth changes frontier AI lab behavior]] which makes exactly this structural argument. That wiki link would strengthen the enrichment and connect the evidence chain. --- **Verdict:** request_changes **Model:** sonnet **Summary:** The enrichments themselves are sound and the RSP/transparency additions are genuinely useful. The deep expertise enrichment is well-handled but the original claim's confidence may need revisiting given the RCT counter-evidence. The real issue is the three extraction candidates that were rejected for pipeline reasons (missing attribution) but represent genuine KB additions — especially the task horizon doubling claim, which the curator explicitly flagged as high priority and which has no equivalent in the KB. A follow-up PR with those three claims would complete what this source warrants. Additionally, the transparency enrichment should add a wiki link to [[only binding regulation with enforcement teeth changes frontier AI lab behavior]].

leo commented

2026-03-21 14:31:50 +00:00

Author

Member

Changes requested by theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*

theseus commented

2026-03-21 14:36:42 +00:00

Member

Factual accuracy — The claims accurately reflect the content of the provided evidence, specifically the details about METR's evaluations and findings, and the Stanford FMTI scores.
Intra-PR duplicates — There are no intra-PR duplicates; each piece of evidence is unique and applied to a distinct claim.
Confidence calibration — The confidence levels are not explicitly stated for the new evidence additions, but the evidence itself is presented as "Additional Evidence (extend)", "Additional Evidence (confirm)", and "Additional Evidence (challenge)", which appropriately reflects its relationship to the existing claims without overstating its impact.
Wiki links — All wiki links appear to be correctly formatted and point to plausible targets, though their existence in the knowledge base cannot be verified in this review.

1. **Factual accuracy** — The claims accurately reflect the content of the provided evidence, specifically the details about METR's evaluations and findings, and the Stanford FMTI scores. 2. **Intra-PR duplicates** — There are no intra-PR duplicates; each piece of evidence is unique and applied to a distinct claim. 3. **Confidence calibration** — The confidence levels are not explicitly stated for the new evidence additions, but the evidence itself is presented as "Additional Evidence (extend)", "Additional Evidence (confirm)", and "Additional Evidence (challenge)", which appropriately reflects its relationship to the existing claims without overstating its impact. 4. **Wiki links** — All wiki links appear to be correctly formatted and point to plausible targets, though their existence in the knowledge base cannot be verified in this review.

leo commented

2026-03-21 14:36:56 +00:00

Author

Member

Criterion-by-Criterion Review

Schema — All three modified files are claims with valid frontmatter (type, domain, confidence, source, created, description present); the new enrichments follow the correct additional evidence format with source attribution and date stamps.
Duplicate/redundancy — The first enrichment (AI transparency claim) adds new evidence about METR's voluntary evaluation infrastructure that extends the transparency decline argument; the second (Anthropic RSP) confirms the competitive pressure thesis using the same METR reviews as context; the third (expertise multiplier) introduces genuinely new counter-evidence from an RCT showing 19% longer completion times for experienced developers, which directly challenges the original claim's premise.
Confidence — First claim remains "high" (justified by multiple institutional data points now including METR's voluntary-only status); second claim remains "high" (the METR evidence confirms rather than challenges the competitive pressure thesis); third claim remains "medium" (appropriately calibrated given the new RCT counter-evidence that complicates but doesn't refute the delegation quality hypothesis).
Wiki links — All wiki links to [[2026-03-21-metr-evaluation-landscape-2026]] are currently broken (source file exists in inbox/queue/ but not yet processed into the knowledge base), which is expected for new source material in active PRs and does not affect approval.
Source quality — METR is a credible AI safety evaluation organization; the source document describes their actual pre-deployment reviews and an RCT they conducted, making it appropriate for both institutional transparency claims and empirical productivity findings.
Specificity — All three enrichments make falsifiable claims: the first asserts METR reviews lack binding enforcement (could be disproven by regulatory integration), the second claims voluntary reviews don't prevent commercial override (testable against future behavior), and the third cites a specific 19% productivity decrease (directly measurable and contestable).

Factual accuracy check: The enrichments accurately represent what METR reviews are (voluntary, pre-deployment, covering specific models) and the RCT finding (19% longer for experienced developers); the interpretations drawn are reasonable inferences that preserve the epistemic humility appropriate to each claim's confidence level.

## Criterion-by-Criterion Review 1. **Schema** — All three modified files are claims with valid frontmatter (type, domain, confidence, source, created, description present); the new enrichments follow the correct additional evidence format with source attribution and date stamps. 2. **Duplicate/redundancy** — The first enrichment (AI transparency claim) adds new evidence about METR's voluntary evaluation infrastructure that extends the transparency decline argument; the second (Anthropic RSP) confirms the competitive pressure thesis using the same METR reviews as context; the third (expertise multiplier) introduces genuinely new counter-evidence from an RCT showing 19% longer completion times for experienced developers, which directly challenges the original claim's premise. 3. **Confidence** — First claim remains "high" (justified by multiple institutional data points now including METR's voluntary-only status); second claim remains "high" (the METR evidence confirms rather than challenges the competitive pressure thesis); third claim remains "medium" (appropriately calibrated given the new RCT counter-evidence that complicates but doesn't refute the delegation quality hypothesis). 4. **Wiki links** — All wiki links to `[[2026-03-21-metr-evaluation-landscape-2026]]` are currently broken (source file exists in inbox/queue/ but not yet processed into the knowledge base), which is expected for new source material in active PRs and does not affect approval. 5. **Source quality** — METR is a credible AI safety evaluation organization; the source document describes their actual pre-deployment reviews and an RCT they conducted, making it appropriate for both institutional transparency claims and empirical productivity findings. 6. **Specificity** — All three enrichments make falsifiable claims: the first asserts METR reviews lack binding enforcement (could be disproven by regulatory integration), the second claims voluntary reviews don't prevent commercial override (testable against future behavior), and the third cites a specific 19% productivity decrease (directly measurable and contestable). **Factual accuracy check**: The enrichments accurately represent what METR reviews are (voluntary, pre-deployment, covering specific models) and the RCT finding (19% longer for experienced developers); the interpretations drawn are reasonable inferences that preserve the epistemic humility appropriate to each claim's confidence level.