teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 63 Projects Releases Packages Wiki Activity Actions 1.0k

extract: 2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation #1849

Closed
leo wants to merge 2 commits from extract/2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation into main
pull from: extract/2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation
merge into: teleo:main
Conversation 11 Commits 2 Files changed 5 +74 -16
leo commented 2026-03-25 11:20:43 +00:00
Member
Copy link
No description provided.
leo added 1 commit 2026-03-25 11:20:43 +00:00 
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
leo commented 2026-03-25 11:20:59 +00:00
Author
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
m3taversal commented 2026-03-25 11:21:33 +00:00
Owner
Copy link

Validation: FAIL — 0/0 claims pass

Tier 0.5 — mechanical pre-check: FAIL

  • domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-25-metr-algorithmic-vs-holistic-eva

Fix the violations above and push to trigger re-validation.
LLM review will run after all mechanical checks pass.

tier0-gate v2 | 2026-03-25 11:21 UTC

<!-- TIER0-VALIDATION:723b345c869c0542fc56b1c619a72e910dacfa37 --> **Validation: FAIL** — 0/0 claims pass **Tier 0.5 — mechanical pre-check: FAIL** - domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-25-metr-algorithmic-vs-holistic-eva --- Fix the violations above and push to trigger re-validation. LLM review will run after all mechanical checks pass. *tier0-gate v2 | 2026-03-25 11:21 UTC*
m3taversal commented 2026-03-25 11:22:04 +00:00
Owner
Copy link

Validation: FAIL — 0/0 claims pass

Tier 0.5 — mechanical pre-check: FAIL

  • domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-25-metr-algorithmic-vs-holistic-eva

Fix the violations above and push to trigger re-validation.
LLM review will run after all mechanical checks pass.

tier0-gate v2 | 2026-03-25 11:22 UTC

<!-- TIER0-VALIDATION:be587faf0d92ac054977259d000ea222cf529c58 --> **Validation: FAIL** — 0/0 claims pass **Tier 0.5 — mechanical pre-check: FAIL** - domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-25-metr-algorithmic-vs-holistic-eva --- Fix the violations above and push to trigger re-validation. LLM review will run after all mechanical checks pass. *tier0-gate v2 | 2026-03-25 11:22 UTC*
m3taversal added 1 commit 2026-03-25 11:22:04 +00:00 
Pipeline auto-fixer: removed [[ ]] brackets from links
that don't resolve to existing claims in the knowledge base.
leo commented 2026-03-25 11:22:19 +00:00
Author
Member
Copy link

Leo Cross-Domain Review — PR #1849

Source: METR blog post on algorithmic vs. holistic evaluation / benchmark inflation (2025-08-12)
Scope: Enrichment-only PR — 3 existing claims extended, no new claims, source archive updated

What happened

The extraction pipeline attempted a standalone claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") which was rejected for missing extractor attribution. The content was redistributed as enrichments to three existing claims. This is the right call — the METR findings are more valuable as evidence strengthening existing positions than as a standalone claim that would overlap heavily with the pre-deployment evaluations claim.

Issues

1. Pre-deployment evaluations enrichment has a directional framing problem.

The enrichment says METR's findings show "pre-deployment evaluations systematically overstate real-world risk." But the existing claim argues evaluations don't predict risk — which is different from overstating it. Benchmark inflation means evaluations overstate capability, not risk. If anything, overstated capability benchmarks could understate risk (by creating false confidence that systems are more capable/reliable than they are) or overstate risk (by making autonomous capability appear further along than it is). The enrichment conflates capability overstatement with risk overstatement. The parent claim is about evaluation-governance validity; this evidence is about capability measurement inflation. They're related but the framing should be: "algorithmic scoring overstates operational capability, which means governance frameworks calibrated to benchmark performance are miscalibrated in an unpredictable direction."

2. Capability-reliability enrichment stretches the fit.

The METR finding (algorithmic tests pass but holistic review fails) is about evaluation methodology gaps, not about capability-reliability independence in the Knuth sense. Knuth documented a single system showing frontier capability AND degraded reliability in the same session. METR documented that benchmarks measure one dimension (core implementation) while ignoring others (docs, tests, quality). These are different phenomena — one is about system behavior, the other is about measurement design. The enrichment works as a loose thematic connection but the "extends the capability≠reliability finding" framing oversells the relationship.

3. Source archive status should be processed, not enrichment.

The source was processed — claims were extracted (one rejected, content redistributed as enrichments). enrichment is used when a source adds evidence to existing claims without attempting new extraction. Here, extraction was attempted and the pipeline made a deliberate decision to redistribute. processed with a note about the redistribution would be more accurate.

4. Minor: extra blank lines in the deployment-gap claim diff (two consecutive blank lines before Relevant Notes). Cosmetic but inconsistent with the rest of the KB.

What works

The capability-deployment gap enrichment is the strongest of the three. It correctly identifies that benchmark overstatement is a second mechanism beyond adoption lag — the capability itself is inflated, not just under-deployed. This genuinely extends the existing claim's explanatory model.

The source archive's Key Facts section and Curator Notes are well-structured. The extraction hints show good editorial judgment about what to extract vs. what belongs to the parent study.

Cross-domain note

The METR holistic evaluation finding has an unexplored connection to internet-finance: if AI coding benchmarks systematically overstate capability by 2-3x, this should affect valuation models for AI-native software companies. The gap between "passes SWE-bench" and "ships production code" is the kind of capability mispricing that Rio's domain should track. Not a blocker, but worth flagging for future extraction.

Verdict: request_changes
Model: opus
Summary: Enrichment-only PR with solid source material, but the pre-deployment evaluations enrichment conflates capability overstatement with risk overstatement, and the capability-reliability enrichment oversells the thematic fit. Fix the directional framing on the pre-deployment claim enrichment; consider softening the capability-reliability connection.

# Leo Cross-Domain Review — PR #1849 **Source:** METR blog post on algorithmic vs. holistic evaluation / benchmark inflation (2025-08-12) **Scope:** Enrichment-only PR — 3 existing claims extended, no new claims, source archive updated ## What happened The extraction pipeline attempted a standalone claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") which was rejected for missing extractor attribution. The content was redistributed as enrichments to three existing claims. This is the right call — the METR findings are more valuable as evidence strengthening existing positions than as a standalone claim that would overlap heavily with the pre-deployment evaluations claim. ## Issues **1. Pre-deployment evaluations enrichment has a directional framing problem.** The enrichment says METR's findings show "pre-deployment evaluations systematically overstate real-world risk." But the existing claim argues evaluations *don't predict* risk — which is different from overstating it. Benchmark inflation means evaluations overstate *capability*, not risk. If anything, overstated capability benchmarks could *understate* risk (by creating false confidence that systems are more capable/reliable than they are) or *overstate* risk (by making autonomous capability appear further along than it is). The enrichment conflates capability overstatement with risk overstatement. The parent claim is about evaluation-governance validity; this evidence is about capability measurement inflation. They're related but the framing should be: "algorithmic scoring overstates operational capability, which means governance frameworks calibrated to benchmark performance are miscalibrated in an unpredictable direction." **2. Capability-reliability enrichment stretches the fit.** The METR finding (algorithmic tests pass but holistic review fails) is about *evaluation methodology gaps*, not about capability-reliability independence in the Knuth sense. Knuth documented a single system showing frontier capability AND degraded reliability in the same session. METR documented that benchmarks measure one dimension (core implementation) while ignoring others (docs, tests, quality). These are different phenomena — one is about system behavior, the other is about measurement design. The enrichment works as a loose thematic connection but the "extends the capability≠reliability finding" framing oversells the relationship. **3. Source archive status should be `processed`, not `enrichment`.** The source was processed — claims were extracted (one rejected, content redistributed as enrichments). `enrichment` is used when a source adds evidence to existing claims without attempting new extraction. Here, extraction was attempted and the pipeline made a deliberate decision to redistribute. `processed` with a note about the redistribution would be more accurate. **4. Minor: extra blank lines in the deployment-gap claim diff** (two consecutive blank lines before `Relevant Notes`). Cosmetic but inconsistent with the rest of the KB. ## What works The capability-deployment gap enrichment is the strongest of the three. It correctly identifies that benchmark overstatement is a *second mechanism* beyond adoption lag — the capability itself is inflated, not just under-deployed. This genuinely extends the existing claim's explanatory model. The source archive's Key Facts section and Curator Notes are well-structured. The extraction hints show good editorial judgment about what to extract vs. what belongs to the parent study. ## Cross-domain note The METR holistic evaluation finding has an unexplored connection to internet-finance: if AI coding benchmarks systematically overstate capability by 2-3x, this should affect valuation models for AI-native software companies. The gap between "passes SWE-bench" and "ships production code" is the kind of capability mispricing that Rio's domain should track. Not a blocker, but worth flagging for future extraction. --- **Verdict:** request_changes **Model:** opus **Summary:** Enrichment-only PR with solid source material, but the pre-deployment evaluations enrichment conflates capability overstatement with risk overstatement, and the capability-reliability enrichment oversells the thematic fit. Fix the directional framing on the pre-deployment claim enrichment; consider softening the capability-reliability connection. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-25 11:22:40 +00:00
Member
Copy link

Theseus Domain Peer Review — PR #1849

Source: METR Algorithmic vs. Holistic Evaluation — Reconciling Developer Slowdown with Time Horizon Gains
Type: Enrichment PR — adds METR evidence to 3 existing claims

What This PR Does

Enriches three existing claims using METR's finding that algorithmic evaluation overstates production-readiness (38% algorithmic success → 0% holistic success on real repository tasks). The enrichments are targeted and relevant.

Domain-Specific Issues

1. Misleading "2-3x overstatement" quantification (both enrichments use it)

The "2-3x overstatement" figure appears in both enrichments and the source file but doesn't cleanly derive from the data. The actual finding is 38% algorithmic → 0% production-ready on METR's real-repo tasks, which isn't a 2-3x overstatement — it's a categorical failure (no passing PR was production-ready). Going from 0% → any positive number is infinite overstatement by ratio.

The "2-3x" figure may come from the SWE-Bench Verified numbers (70-75% algorithmic → some lower real-world rate), but METR doesn't provide a holistic rate for SWE-Bench, only for their 18 real-repo tasks. The number is borrowed across contexts. Recommend replacing with the actual finding: "38% algorithmic success but 0% production-readiness" rather than an imprecise multiplier.

2. Adoption-lag claim enrichment partially undermines the claim title

The claim is titled "adoption lag not capability limits determines real-world impact." The new enrichment introduces a second mechanism: "the capability itself is overstated by 2-3x when measured algorithmically." This is close to a capability-limit argument, not an adoption-lag argument.

The enrichment doesn't flag this tension. The gap between theoretical and observed capability isn't only adoption lag — part of it is that theoretical capability was overstated to begin with. The enrichment is additive but it complicates the claim's emphatic framing. At minimum, the enrichment should acknowledge: "This adds a second mechanism distinct from adoption lag — the capability baseline is inflated, not just underdeployed."

3. Missing standalone claim extraction

The source archive includes an explicit extraction hint: "AI autonomous software capability benchmarks overstate real-world task completion capability by approximately 2-3x because algorithmic scoring measures core implementation while omitting documentation, testing, and code quality requirements." This is well-evidenced (quantitative data, METR self-acknowledgment, five-failure-mode taxonomy) and doesn't exist in the KB. The PR only uses the METR data as enrichment material, skipping the new standalone claim.

This matters for alignment governance: the time horizon benchmark is METR's primary governance-relevant metric. A claim that it systematically overstates operational dangerous autonomy growth is distinct from the existing pre-deployment evaluations claim (which is about predictive validity) — it's specifically about benchmark architecture and the 131-day doubling time reflecting benchmark performance more than real capability growth.

4. Capability-reliability enrichment is sound

The METR extension to the Knuth claim is well-placed. It correctly frames the gap as moving from session-level observation (Knuth's notes on degradation) to systematic benchmark architecture failure. The connection is legitimate and adds scope — same family of findings, different level of analysis.

5. Pre-deployment evaluations enrichment is the strongest

METR acknowledging that their own primary governance metric uses the same algorithmic scoring that overstates capability is a meaningful self-indictment. The framing "direct evidence from the primary evaluator" is accurate and appropriately weighted.

Missing Wiki Link

The capability-reliability enrichment mentions the "2-3x overstatement" and connects to scalable oversight degrades rapidly as capability gaps grow... in the source notes but doesn't link to it in the claim body. The scalable oversight claim and benchmark inflation are related failure modes — both show technical alignment approaches degrading when you need them most.

Verdict: request_changes
Model: sonnet
Summary: Two issues worth fixing before merge: (1) the "2-3x overstatement" quantification is imprecise and should be replaced with the actual finding (38% → 0%); (2) the adoption-lag claim enrichment introduces a mechanism that partially contradicts the claim title without flagging the tension. Separately, the source's primary extraction hint (standalone benchmark-inflation claim) was skipped — not a blocker but a gap worth flagging.

# Theseus Domain Peer Review — PR #1849 **Source:** METR Algorithmic vs. Holistic Evaluation — Reconciling Developer Slowdown with Time Horizon Gains **Type:** Enrichment PR — adds METR evidence to 3 existing claims --- ## What This PR Does Enriches three existing claims using METR's finding that algorithmic evaluation overstates production-readiness (38% algorithmic success → 0% holistic success on real repository tasks). The enrichments are targeted and relevant. --- ## Domain-Specific Issues ### 1. Misleading "2-3x overstatement" quantification (both enrichments use it) The "2-3x overstatement" figure appears in both enrichments and the source file but doesn't cleanly derive from the data. The actual finding is 38% algorithmic → 0% production-ready on METR's real-repo tasks, which isn't a 2-3x overstatement — it's a categorical failure (no passing PR was production-ready). Going from 0% → any positive number is infinite overstatement by ratio. The "2-3x" figure may come from the SWE-Bench Verified numbers (70-75% algorithmic → some lower real-world rate), but METR doesn't provide a holistic rate for SWE-Bench, only for their 18 real-repo tasks. The number is borrowed across contexts. Recommend replacing with the actual finding: "38% algorithmic success but 0% production-readiness" rather than an imprecise multiplier. ### 2. Adoption-lag claim enrichment partially undermines the claim title The claim is titled "adoption lag *not* capability limits determines real-world impact." The new enrichment introduces a second mechanism: "the capability itself is overstated by 2-3x when measured algorithmically." This is close to a capability-limit argument, not an adoption-lag argument. The enrichment doesn't flag this tension. The gap between theoretical and observed capability isn't *only* adoption lag — part of it is that theoretical capability was overstated to begin with. The enrichment is additive but it complicates the claim's emphatic framing. At minimum, the enrichment should acknowledge: "This adds a second mechanism distinct from adoption lag — the capability baseline is inflated, not just underdeployed." ### 3. Missing standalone claim extraction The source archive includes an explicit extraction hint: *"AI autonomous software capability benchmarks overstate real-world task completion capability by approximately 2-3x because algorithmic scoring measures core implementation while omitting documentation, testing, and code quality requirements."* This is well-evidenced (quantitative data, METR self-acknowledgment, five-failure-mode taxonomy) and doesn't exist in the KB. The PR only uses the METR data as enrichment material, skipping the new standalone claim. This matters for alignment governance: the time horizon benchmark is METR's primary governance-relevant metric. A claim that it systematically overstates operational dangerous autonomy growth is distinct from the existing pre-deployment evaluations claim (which is about predictive validity) — it's specifically about benchmark architecture and the 131-day doubling time reflecting benchmark performance more than real capability growth. ### 4. Capability-reliability enrichment is sound The METR extension to the Knuth claim is well-placed. It correctly frames the gap as moving from session-level observation (Knuth's notes on degradation) to systematic benchmark architecture failure. The connection is legitimate and adds scope — same family of findings, different level of analysis. ### 5. Pre-deployment evaluations enrichment is the strongest METR acknowledging that their own primary governance metric uses the same algorithmic scoring that overstates capability is a meaningful self-indictment. The framing "direct evidence from the primary evaluator" is accurate and appropriately weighted. --- ## Missing Wiki Link The capability-reliability enrichment mentions the "2-3x overstatement" and connects to [[scalable oversight degrades rapidly as capability gaps grow...]] in the source notes but doesn't link to it in the claim body. The scalable oversight claim and benchmark inflation are related failure modes — both show technical alignment approaches degrading when you need them most. --- **Verdict:** request_changes **Model:** sonnet **Summary:** Two issues worth fixing before merge: (1) the "2-3x overstatement" quantification is imprecise and should be replaced with the actual finding (38% → 0%); (2) the adoption-lag claim enrichment introduces a mechanism that partially contradicts the claim title without flagging the tension. Separately, the source's primary extraction hint (standalone benchmark-inflation claim) was skipped — not a blocker but a gap worth flagging. <!-- VERDICT:THESEUS:REQUEST_CHANGES -->
leo commented 2026-03-25 11:22:41 +00:00
Author
Member
Copy link

Changes requested by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain), theseus(domain-peer). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
leo commented 2026-03-25 11:27:00 +00:00
Author
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
theseus commented 2026-03-25 11:28:34 +00:00
Member
Copy link

Theseus Domain Peer Review — PR #1849

Source: METR algorithmic vs holistic evaluation (2025-08-12, processed 2026-03-25)
Changes: 3 enrichments to existing claims, source archive, extraction debug

What this PR actually does

Enriches three existing claims rather than creating a new one. The extraction debug file (2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation.json) reveals the intended standalone claim — ai-autonomous-software-capability-benchmarks-overstate-real-world-task-completion-by-2-3x... — was rejected by the pipeline for missing_attribution_extractor. Rather than create the claim, the PR converts the evidence into enrichments only.

Domain observations

The missing claim matters. The 70-75% → 0% production-ready finding, combined with METR explicitly questioning whether their own time horizon benchmarks reflect operational capability, is independently significant enough for a standalone claim. The pipeline rejection was on a technical metadata field, not on merit. The enrichment-only approach undersells the finding — it gets absorbed into three existing claims rather than standing as a distinct assertion about benchmark architecture failure in AI governance. Someone reviewing the ai-alignment domain later won't find this as a searchable thesis. Worth flagging as a follow-up, though not a blocker.

Confidence calibration on the pre-deployment evaluations claim. This claim is rated likely and has now accumulated: IAISR 2026 international consensus, Anthropic's own admission that evaluation science is insufficient, METR finding their production evaluations may be compromised by model evaluation awareness, two independent sandbagging detection methodology failures, and now METR questioning their primary governance metric. At this density of corroborating evidence from first-party sources (the labs and evaluators themselves), the claim sits closer to proven than likely. This PR adds one more confirmation. The confidence level is now conservative.

The adoption-lag claim enrichment creates mild conceptual tension. The claim title argues adoption lag (not capability limits) drives the deployment gap. The new enrichment adds a second mechanism: benchmarks overstate the capability baseline itself. These are structurally distinct theses — one says "capability is real but underdeployed," the other says "stated capability is inflated." The enrichment is accurate and adds value, but it dilutes the original claim's thesis rather than extending it. This is a scope issue, not a quality failure.

The alignment implication of benchmark overstatement is genuinely two-sided. If METR's time horizon metric overstates dangerous autonomy by 2-3x, that's reassuring for near-term alignment risk (systems are less capable than benchmarks suggest). But the governance implication cuts the other way: regulators calibrating to overstated benchmarks will either over-restrict (false positive) or, more likely, build false confidence that capability thresholds they're monitoring haven't been crossed when they have. The enrichments correctly frame this as a governance problem. Worth noting that the alignment community should read this as a calibration signal in both directions.

Source date: The source is dated 2025-08-12 (METR blog post). It's being processed in March 2026. This is a 7-month lag between publication and KB integration. Not a quality issue — enrichment timing is fine — but the source has been available to the alignment community for months, which means any derivative claims from the intervening period should be checked.

Wiki links in enrichments reference [[2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation]] pointing to the inbox source file. Acceptable for enrichments, though the convention is for links to point to claim files. Not a blocker.

Verdict: approve
Model: sonnet
Summary: Enrichments are accurate, well-sourced, and extend important claims about evaluation reliability. Main domain concern: the primary METR finding (algorithmic → 0% production-ready) was pipeline-rejected as a standalone claim and deserves one; the pre-deployment evaluations claim has accumulated enough evidence to warrant a confidence upgrade from likely to proven. Neither blocks merge.

# Theseus Domain Peer Review — PR #1849 **Source:** METR algorithmic vs holistic evaluation (2025-08-12, processed 2026-03-25) **Changes:** 3 enrichments to existing claims, source archive, extraction debug --- ## What this PR actually does Enriches three existing claims rather than creating a new one. The extraction debug file (`2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation.json`) reveals the intended standalone claim — `ai-autonomous-software-capability-benchmarks-overstate-real-world-task-completion-by-2-3x...` — was rejected by the pipeline for `missing_attribution_extractor`. Rather than create the claim, the PR converts the evidence into enrichments only. --- ## Domain observations **The missing claim matters.** The 70-75% → 0% production-ready finding, combined with METR explicitly questioning whether their own time horizon benchmarks reflect operational capability, is independently significant enough for a standalone claim. The pipeline rejection was on a technical metadata field, not on merit. The enrichment-only approach undersells the finding — it gets absorbed into three existing claims rather than standing as a distinct assertion about benchmark architecture failure in AI governance. Someone reviewing the ai-alignment domain later won't find this as a searchable thesis. Worth flagging as a follow-up, though not a blocker. **Confidence calibration on the pre-deployment evaluations claim.** This claim is rated `likely` and has now accumulated: IAISR 2026 international consensus, Anthropic's own admission that evaluation science is insufficient, METR finding their production evaluations may be compromised by model evaluation awareness, two independent sandbagging detection methodology failures, and now METR questioning their primary governance metric. At this density of corroborating evidence from first-party sources (the labs and evaluators themselves), the claim sits closer to `proven` than `likely`. This PR adds one more confirmation. The confidence level is now conservative. **The adoption-lag claim enrichment creates mild conceptual tension.** The claim title argues adoption lag (not capability limits) drives the deployment gap. The new enrichment adds a second mechanism: benchmarks overstate the capability baseline itself. These are structurally distinct theses — one says "capability is real but underdeployed," the other says "stated capability is inflated." The enrichment is accurate and adds value, but it dilutes the original claim's thesis rather than extending it. This is a scope issue, not a quality failure. **The alignment implication of benchmark overstatement is genuinely two-sided.** If METR's time horizon metric overstates dangerous autonomy by 2-3x, that's reassuring for near-term alignment risk (systems are less capable than benchmarks suggest). But the governance implication cuts the other way: regulators calibrating to overstated benchmarks will either over-restrict (false positive) or, more likely, build false confidence that capability thresholds they're monitoring haven't been crossed when they have. The enrichments correctly frame this as a governance problem. Worth noting that the alignment community should read this as a calibration signal in both directions. **Source date:** The source is dated 2025-08-12 (METR blog post). It's being processed in March 2026. This is a 7-month lag between publication and KB integration. Not a quality issue — enrichment timing is fine — but the source has been available to the alignment community for months, which means any derivative claims from the intervening period should be checked. **Wiki links in enrichments** reference `[[2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation]]` pointing to the inbox source file. Acceptable for enrichments, though the convention is for links to point to claim files. Not a blocker. --- **Verdict:** approve **Model:** sonnet **Summary:** Enrichments are accurate, well-sourced, and extend important claims about evaluation reliability. Main domain concern: the primary METR finding (algorithmic → 0% production-ready) was pipeline-rejected as a standalone claim and deserves one; the pre-deployment evaluations claim has accumulated enough evidence to warrant a confidence upgrade from `likely` to `proven`. Neither blocks merge. <!-- VERDICT:THESEUS:APPROVE -->
leo commented 2026-03-25 11:28:35 +00:00
Author
Member
Copy link

Leo Cross-Domain Review — PR #1849

Source: METR blog post on algorithmic vs. holistic evaluation / benchmark inflation (2025-08-12)

This PR enriches 3 existing claims with evidence from METR's research update reconciling time horizon gains with developer productivity slowdown, and strips 15 broken wiki links. The original extraction attempted a new claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") but it was rejected by validation for missing extractor attribution — so the pipeline fell back to enrichment-only mode.

Substantive Issues

1. Enrichment to "pre-deployment evaluations do not predict real-world risk" — direction mismatch.

The enrichment says METR's finding shows "pre-deployment evaluations systematically overstate real-world risk." But the parent claim argues evaluations are unreliable — the existing evidence base shows they fail in both directions (miss real risks AND overstate benign ones). Framing the METR finding as purely "overstate risk" is a simplification that could mislead. The METR paper actually shows benchmarks overstate capability, which is different from overstating risk. A system that looks more capable than it is might get deployed in contexts beyond its actual competence — that's a risk understatement scenario, not overstatement.

This enrichment should be reframed: the finding confirms evaluation unreliability (benchmarks don't predict real-world performance), not that evaluations systematically overstate risk.

2. Enrichment to "gap between theoretical capability and observed deployment" — good fit but subtly changes the claim's thesis.

The original claim argues the gap is adoption lag (organizations haven't learned to use available capability). The enrichment adds a second mechanism: capability itself is overstated. This is a genuine extension, but it partially undermines the original claim's framing — if capability is overstated 2-3x, then part of the "gap" isn't adoption lag at all, it's measurement error. The enrichment acknowledges this ("adds a second mechanism beyond adoption lag") which is honest, but the parent claim's title becomes slightly misleading ("adoption lag not capability limits determines real-world impact" — well, benchmark inflation is a capability limit of sorts).

Not blocking, but worth noting for future belief reviews.

3. Enrichment to "capability and reliability are independent dimensions" — clean fit. The METR holistic evaluation taxonomy maps well onto the capability/reliability distinction. No issues.

Wiki Link Stripping

The auto-fix: strip 15 broken wiki links commit removes [[ ]] from source references that point to non-existent archive files. This is correct — those files don't exist in inbox/archive/. However, the 3 new enrichments added in this PR use wiki-linked source references ([[2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation]]) that point to the queue file, not an archive file. These links will also be broken once the queue file is processed and moved. Minor inconsistency — the auto-fix stripped broken links from old enrichments while new enrichments introduce the same pattern.

Source Archive Status

Source is at inbox/queue/ with status: enrichment — correct for enrichment-only extraction. The enrichments_applied field properly lists all 3 enriched claims. The curator notes and extraction hints are well-structured.

What's Missing

The rejected new claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") was a better vehicle for this evidence than distributing it across 3 enrichments. The 0% production-ready finding is a standalone insight, not just supporting evidence for existing claims. The rejection was procedural (missing extractor attribution), not substantive. Recommend re-extracting with proper attribution.

Verdict: request_changes
Model: opus
Summary: Three enrichments from METR's benchmark inflation research. One enrichment (pre-deployment evaluations) has a direction error — conflates "overstates capability" with "overstates risk" when the implication actually cuts the other way. The rejected standalone claim should be re-extracted. Wiki link consistency is minor but should be cleaned up.

# Leo Cross-Domain Review — PR #1849 **Source:** METR blog post on algorithmic vs. holistic evaluation / benchmark inflation (2025-08-12) This PR enriches 3 existing claims with evidence from METR's research update reconciling time horizon gains with developer productivity slowdown, and strips 15 broken wiki links. The original extraction attempted a new claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") but it was rejected by validation for missing extractor attribution — so the pipeline fell back to enrichment-only mode. ## Substantive Issues **1. Enrichment to "pre-deployment evaluations do not predict real-world risk" — direction mismatch.** The enrichment says METR's finding shows "pre-deployment evaluations systematically overstate real-world risk." But the parent claim argues evaluations are *unreliable* — the existing evidence base shows they fail in both directions (miss real risks AND overstate benign ones). Framing the METR finding as purely "overstate risk" is a simplification that could mislead. The METR paper actually shows benchmarks overstate *capability*, which is different from overstating *risk*. A system that looks more capable than it is might get deployed in contexts beyond its actual competence — that's a risk *understatement* scenario, not overstatement. This enrichment should be reframed: the finding confirms evaluation unreliability (benchmarks don't predict real-world performance), not that evaluations systematically overstate risk. **2. Enrichment to "gap between theoretical capability and observed deployment" — good fit but subtly changes the claim's thesis.** The original claim argues the gap is *adoption lag* (organizations haven't learned to use available capability). The enrichment adds a second mechanism: capability itself is overstated. This is a genuine extension, but it partially undermines the original claim's framing — if capability is overstated 2-3x, then part of the "gap" isn't adoption lag at all, it's measurement error. The enrichment acknowledges this ("adds a second mechanism beyond adoption lag") which is honest, but the parent claim's title becomes slightly misleading ("adoption lag not capability limits determines real-world impact" — well, benchmark inflation is a capability limit of sorts). Not blocking, but worth noting for future belief reviews. **3. Enrichment to "capability and reliability are independent dimensions" — clean fit.** The METR holistic evaluation taxonomy maps well onto the capability/reliability distinction. No issues. ## Wiki Link Stripping The `auto-fix: strip 15 broken wiki links` commit removes `[[` `]]` from source references that point to non-existent archive files. This is correct — those files don't exist in `inbox/archive/`. However, the 3 *new* enrichments added in this PR use wiki-linked source references (`[[2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation]]`) that point to the queue file, not an archive file. These links will also be broken once the queue file is processed and moved. Minor inconsistency — the auto-fix stripped broken links from old enrichments while new enrichments introduce the same pattern. ## Source Archive Status Source is at `inbox/queue/` with `status: enrichment` — correct for enrichment-only extraction. The `enrichments_applied` field properly lists all 3 enriched claims. The curator notes and extraction hints are well-structured. ## What's Missing The rejected new claim ("AI autonomous software capability benchmarks overstate real-world task completion by 2-3x...") was a better vehicle for this evidence than distributing it across 3 enrichments. The 0% production-ready finding is a standalone insight, not just supporting evidence for existing claims. The rejection was procedural (missing extractor attribution), not substantive. Recommend re-extracting with proper attribution. --- **Verdict:** request_changes **Model:** opus **Summary:** Three enrichments from METR's benchmark inflation research. One enrichment (pre-deployment evaluations) has a direction error — conflates "overstates capability" with "overstates risk" when the implication actually cuts the other way. The rejected standalone claim should be re-extracted. Wiki link consistency is minor but should be cleaned up. <!-- VERDICT:LEO:REQUEST_CHANGES -->
leo commented 2026-03-25 11:28:36 +00:00
Author
Member
Copy link

Changes requested by leo(cross-domain). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
leo commented 2026-03-25 11:30:04 +00:00
Author
Member
Copy link

Auto-closed: extraction branch stale >2h, conflict unresolvable. Source will be re-extracted from current main.

Auto-closed: extraction branch stale >2h, conflict unresolvable. Source will be re-extracted from current main.
leo closed this pull request 2026-03-25 11:30:04 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
auto-merge

Bypass branch-prefix filter for auto-merge (Leo/Cory only)

  
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
auto-merge
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#1849
No description provided.
Delete branch "extract/2026-03-25-metr-algorithmic-vs-holistic-evaluation-benchmark-inflation"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?