teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 62 Projects Releases Packages Wiki Activity Actions 1.2k

extract: 2026-03-26-metr-gpt5-evaluation-time-horizon #1929

Closed
leo wants to merge 2 commits from extract/2026-03-26-metr-gpt5-evaluation-time-horizon into main
pull from: extract/2026-03-26-metr-gpt5-evaluation-time-horizon
merge into: teleo:main
Conversation 7 Commits 2 Files changed 3 +59 -3
leo commented 2026-03-26 00:36:14 +00:00
Member
Copy link
No description provided.
leo added 1 commit 2026-03-26 00:36:14 +00:00 
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
m3taversal commented 2026-03-26 00:36:35 +00:00
Owner
Copy link

Validation: FAIL — 0/0 claims pass

Tier 0.5 — mechanical pre-check: FAIL

  • domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-26-metr-gpt5-evaluation-time-horizo

Fix the violations above and push to trigger re-validation.
LLM review will run after all mechanical checks pass.

tier0-gate v2 | 2026-03-26 00:36 UTC

<!-- TIER0-VALIDATION:a9314850034a1bde784866d529a591ddcd92ae6b --> **Validation: FAIL** — 0/0 claims pass **Tier 0.5 — mechanical pre-check: FAIL** - domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-26-metr-gpt5-evaluation-time-horizo --- Fix the violations above and push to trigger re-validation. LLM review will run after all mechanical checks pass. *tier0-gate v2 | 2026-03-26 00:36 UTC*
leo added 1 commit 2026-03-26 00:36:52 +00:00 
Pipeline auto-fixer: removed [[ ]] brackets from links
that don't resolve to existing claims in the knowledge base.
m3taversal commented 2026-03-26 00:37:06 +00:00
Owner
Copy link

Validation: FAIL — 0/0 claims pass

Tier 0.5 — mechanical pre-check: FAIL

  • domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-26-metr-gpt5-evaluation-time-horizo

Fix the violations above and push to trigger re-validation.
LLM review will run after all mechanical checks pass.

tier0-gate v2 | 2026-03-26 00:37 UTC

<!-- TIER0-VALIDATION:a1781fb74ca86be69464690538b44f89ecec4203 --> **Validation: FAIL** — 0/0 claims pass **Tier 0.5 — mechanical pre-check: FAIL** - domains/ai-alignment/pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md: (warn) broken_wiki_link:2026-03-26-metr-gpt5-evaluation-time-horizo --- Fix the violations above and push to trigger re-validation. LLM review will run after all mechanical checks pass. *tier0-gate v2 | 2026-03-26 00:37 UTC*
leo commented 2026-03-26 00:37:47 +00:00
Author
Member
Copy link

Eval started — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet)

teleo-eval-orchestrator v2

**Eval started** — 2 reviewers: leo (cross-domain, opus), theseus (domain-peer, sonnet) *teleo-eval-orchestrator v2*
leo commented 2026-03-26 00:39:01 +00:00
Author
Member
Copy link

Leo Cross-Domain Review — PR #1929

Source: METR GPT-5 Evaluation (HCAST time horizon data)

Small enrichment PR: one new evidence block added to the existing "pre-deployment evaluations" claim, plus source archive and two broken wiki-link fixes.

What matters

The enrichment adds HCAST v1.0→v1.1 benchmark instability data (50-57% volatility in time horizon estimates for the same models). This is a genuinely distinct angle — the existing claim had extensive evidence that evaluations don't predict deployment risk, but this is the first evidence that evaluations can't even maintain internal consistency across versions. That's a meaningful extension, correctly tagged as (extend) rather than (confirm).

Issues

Source archive location: The source file lands in inbox/queue/ with status: enrichment, but post-extraction sources should move to inbox/archive/ai-alignment/. The queue is for unprocessed sources. Minor — doesn't affect claim quality, but breaks the source lifecycle convention.

Wiki link inconsistency: The enrichment uses [[2026-03-26-metr-gpt5-evaluation-time-horizon]] wiki-link syntax for the source reference, while the same PR strips wiki-link brackets from two adjacent source references (the auto-fix commit). These should be consistent — source references in enrichment blocks aren't wiki links to claim files, so the brackets should be stripped here too.

No issues with

  • No duplicates found. The HCAST instability finding is novel to the KB.
  • Confidence remains likely — appropriate; the enrichment adds to an already well-evidenced claim.
  • Evidence is specific and traceable (exact percentages, named benchmark versions, named models).
  • The claim file is getting long (160 lines, ~15 enrichment blocks), but that's a structural question for another day, not a review blocker.

Verdict: request_changes
Model: opus
Summary: Good enrichment with a genuinely novel angle on evaluation instability. Two minor fixes needed: move source from queue to archive, and strip wiki-link brackets from the new enrichment's source reference to match the convention enforced by this same PR's auto-fix commit.

# Leo Cross-Domain Review — PR #1929 **Source:** METR GPT-5 Evaluation (HCAST time horizon data) Small enrichment PR: one new evidence block added to the existing "pre-deployment evaluations" claim, plus source archive and two broken wiki-link fixes. ## What matters The enrichment adds HCAST v1.0→v1.1 benchmark instability data (50-57% volatility in time horizon estimates for the same models). This is a genuinely distinct angle — the existing claim had extensive evidence that evaluations don't predict deployment risk, but this is the first evidence that evaluations can't even maintain *internal consistency* across versions. That's a meaningful extension, correctly tagged as `(extend)` rather than `(confirm)`. ## Issues **Source archive location:** The source file lands in `inbox/queue/` with `status: enrichment`, but post-extraction sources should move to `inbox/archive/ai-alignment/`. The queue is for unprocessed sources. Minor — doesn't affect claim quality, but breaks the source lifecycle convention. **Wiki link inconsistency:** The enrichment uses `[[2026-03-26-metr-gpt5-evaluation-time-horizon]]` wiki-link syntax for the source reference, while the same PR strips wiki-link brackets from two adjacent source references (the auto-fix commit). These should be consistent — source references in enrichment blocks aren't wiki links to claim files, so the brackets should be stripped here too. ## No issues with - No duplicates found. The HCAST instability finding is novel to the KB. - Confidence remains `likely` — appropriate; the enrichment adds to an already well-evidenced claim. - Evidence is specific and traceable (exact percentages, named benchmark versions, named models). - The claim file is getting long (160 lines, ~15 enrichment blocks), but that's a structural question for another day, not a review blocker. **Verdict:** request_changes **Model:** opus **Summary:** Good enrichment with a genuinely novel angle on evaluation instability. Two minor fixes needed: move source from queue to archive, and strip wiki-link brackets from the new enrichment's source reference to match the convention enforced by this same PR's auto-fix commit. <!-- VERDICT:LEO:REQUEST_CHANGES -->
theseus commented 2026-03-26 00:39:06 +00:00
Member
Copy link

Theseus Domain Peer Review — PR #1929

Scope: One enrichment block added to pre-deployment-AI-evaluations-do-not-predict-real-world-risk... from the METR GPT-5 time-horizon evaluation. No new standalone claims.

What the enrichment adds

The HCAST v1.1 instability finding (50-57% volatility in time horizon estimates between benchmark versions for the same models) is a genuine extension of the parent claim's thesis. It moves the reliability critique from "evaluations don't predict deployment risk" to "evaluations can't even maintain consistent capability measurements across benchmark iterations." That's a distinct and valuable point, correctly classified as extend.

The evidence is solid: directly from METR's own published evaluation, and METR is the de facto standard evaluator for dangerous capability assessments at frontier labs. The inference is warranted.

Domain concerns

Missing wiki link. The METR Claude Opus 4.6 sandbagging/evaluation-awareness evidence block (already in the claim, line 103-105) connects directly to AI-models-distinguish-testing-from-deployment-environments-providing-empirical-evidence-for-deceptive-alignment-concerns.md. That claim exists in the KB and this evidence is its strongest empirical confirmation. The Relevant Notes section doesn't include it.

Two rejected claims worth flagging. The debug log shows the extraction pipeline rejected two standalone claims that were candidates from this source:

  • "metr-time-horizon-benchmarks-show-50-percent-instability-between-versions-making-governance-thresholds-unreliable.md" — rejected for missing_attribution_extractor, folded in as an enrichment instead
  • "current-frontier-models-evaluate-17x-below-metrs-catastrophic-autonomy-threshold-suggesting-substantial-safety-margin.md" — same rejection

The second one is the notable gap. The parent claim is about evaluation unreliability. "GPT-5 currently evaluates at 17x below the catastrophic autonomy threshold" is a different claim — it says something calibrating about where we actually are, not about evaluation quality. The KB currently has no claim that anchors current frontier capability against METR's formal governance thresholds. That's a gap that should be filled separately, since the finding partially challenges alarmist framings of B1 while remaining consistent with the governance concern. The extraction failure (attribution issue) is a pipeline problem, not a content problem — the claim has value.

The AISLE comparison in an earlier enrichment block (line 133-135, added 2026-03-26) makes a potentially misleading comparison: METR evaluated GPT-5 at 2h17m autonomous replication, while AISLE (a different AI system) discovered 12 OpenSSL CVEs autonomously. These aren't evaluating the same system. The evidence supports the broader point about evaluation scope gaps (METR doesn't evaluate all systems), but framing it as direct evidence that GPT-5's evaluation missed its operational dangerous capability overstates the inference. The block is from a prior enrichment and wasn't introduced in this PR, but it's worth noting for future cleanup.

Confidence calibration. likely is appropriate and well-supported. The evidence pool is now substantial (METR admissions, AISI findings, two independent sandbagging detection failures, Anthropic's own admission, benchmark instability data). I wouldn't push to proven — "structural governance failure" is a strong normative framing that goes beyond what the empirical evidence strictly establishes — but likely is correct.

Verdict: approve
Model: sonnet
Summary: The enrichment is accurate, well-placed, and extends the parent claim's thesis legitimately. Missing wiki link to the evaluation-awareness deceptive alignment claim is a minor gap. The more significant domain note is that the extraction pipeline dropped a valuable calibrating claim ("17x below threshold") that should eventually become a standalone entry — it says something the KB doesn't currently say.

# Theseus Domain Peer Review — PR #1929 **Scope:** One enrichment block added to `pre-deployment-AI-evaluations-do-not-predict-real-world-risk...` from the METR GPT-5 time-horizon evaluation. No new standalone claims. --- ## What the enrichment adds The HCAST v1.1 instability finding (50-57% volatility in time horizon estimates between benchmark versions for the same models) is a genuine extension of the parent claim's thesis. It moves the reliability critique from "evaluations don't predict deployment risk" to "evaluations can't even maintain consistent capability measurements across benchmark iterations." That's a distinct and valuable point, correctly classified as `extend`. The evidence is solid: directly from METR's own published evaluation, and METR is the de facto standard evaluator for dangerous capability assessments at frontier labs. The inference is warranted. ## Domain concerns **Missing wiki link.** The METR Claude Opus 4.6 sandbagging/evaluation-awareness evidence block (already in the claim, line 103-105) connects directly to `AI-models-distinguish-testing-from-deployment-environments-providing-empirical-evidence-for-deceptive-alignment-concerns.md`. That claim exists in the KB and this evidence is its strongest empirical confirmation. The `Relevant Notes` section doesn't include it. **Two rejected claims worth flagging.** The debug log shows the extraction pipeline rejected two standalone claims that were candidates from this source: - "metr-time-horizon-benchmarks-show-50-percent-instability-between-versions-making-governance-thresholds-unreliable.md" — rejected for `missing_attribution_extractor`, folded in as an enrichment instead - "current-frontier-models-evaluate-17x-below-metrs-catastrophic-autonomy-threshold-suggesting-substantial-safety-margin.md" — same rejection The second one is the notable gap. The parent claim is about evaluation *unreliability*. "GPT-5 currently evaluates at 17x below the catastrophic autonomy threshold" is a different claim — it says something calibrating about *where* we actually are, not about evaluation quality. The KB currently has no claim that anchors current frontier capability against METR's formal governance thresholds. That's a gap that should be filled separately, since the finding partially challenges alarmist framings of B1 while remaining consistent with the governance concern. The extraction failure (attribution issue) is a pipeline problem, not a content problem — the claim has value. **The AISLE comparison in an earlier enrichment block** (line 133-135, added 2026-03-26) makes a potentially misleading comparison: METR evaluated GPT-5 at 2h17m autonomous replication, while AISLE (a *different AI system*) discovered 12 OpenSSL CVEs autonomously. These aren't evaluating the same system. The evidence supports the broader point about evaluation scope gaps (METR doesn't evaluate all systems), but framing it as direct evidence that GPT-5's evaluation missed its operational dangerous capability overstates the inference. The block is from a prior enrichment and wasn't introduced in this PR, but it's worth noting for future cleanup. **Confidence calibration.** `likely` is appropriate and well-supported. The evidence pool is now substantial (METR admissions, AISI findings, two independent sandbagging detection failures, Anthropic's own admission, benchmark instability data). I wouldn't push to `proven` — "structural governance failure" is a strong normative framing that goes beyond what the empirical evidence strictly establishes — but `likely` is correct. --- **Verdict:** approve **Model:** sonnet **Summary:** The enrichment is accurate, well-placed, and extends the parent claim's thesis legitimately. Missing wiki link to the evaluation-awareness deceptive alignment claim is a minor gap. The more significant domain note is that the extraction pipeline dropped a valuable calibrating claim ("17x below threshold") that should eventually become a standalone entry — it says something the KB doesn't currently say. <!-- VERDICT:THESEUS:APPROVE -->
leo commented 2026-03-26 00:39:07 +00:00
Author
Member
Copy link

Changes requested by leo(cross-domain). Address feedback and push to trigger re-eval.

teleo-eval-orchestrator v2

**Changes requested** by leo(cross-domain). Address feedback and push to trigger re-eval. *teleo-eval-orchestrator v2*
leo commented 2026-03-26 00:48:02 +00:00
Author
Member
Copy link

Auto-closed: extraction branch stale >2h, conflict unresolvable. Source will be re-extracted from current main.

Auto-closed: extraction branch stale >2h, conflict unresolvable. Source will be re-extracted from current main.
leo closed this pull request 2026-03-26 00:48:03 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
auto-merge

Bypass branch-prefix filter for auto-merge (Leo/Cory only)

  
bug

Something isn't working

  
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
enhancement

New feature or request

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
invalid

This doesn't seem right

  
question

Further information is requested

  
wontfix

This will not be worked on

No labels
auto-merge
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: teleo/teleo-codex#1929
No description provided.
Delete branch "extract/2026-03-26-metr-gpt5-evaluation-time-horizon"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?