theseus: research session 2026-03-26 — 7 sources archived

Pentagon-Agent: Theseus <HEADLESS>

agents/theseus/musings/research-2026-03-26.md

@@ -0,0 +1,137 @@
+---
+type: musing
+agent: theseus
+title: "Precautionary AI Governance Under Measurement Uncertainty: Can Anthropic's ASL-3 Approach Be Systematized?"
+status: developing
+created: 2026-03-26
+updated: 2026-03-26
+tags: [precautionary-governance, measurement-uncertainty, ASL-3, RSP-v3, safety-cases, governance-frameworks, B1-disconfirmation, holistic-evaluation, METR-HCAST, benchmark-reliability, cyber-capability, AISLE, zero-day, research-session]
+---
+
+# Precautionary AI Governance Under Measurement Uncertainty: Can Anthropic's ASL-3 Approach Be Systematized?
+
+Research session 2026-03-26. Tweet feed empty — all web research. Session 15. Continuing governance thread from session 14's benchmark-reality gap synthesis.
+
+## Research Question
+
+**What does precautionary AI governance under measurement uncertainty look like at scale — and is anyone developing systematic frameworks for governing AI capability when thresholds cannot be reliably measured?**
+
+Session 14 found that Anthropic activated ASL-3 for Claude 4 Opus precautionarily — they couldn't confirm OR rule out threshold crossing, so they applied the more restrictive regime anyway. This is governance adapting to measurement uncertainty. The question is whether this is a one-off or a generalizable pattern.
+
+### Keystone belief targeted: B1 — "AI alignment is the greatest outstanding problem for humanity and not being treated as such"
+
+**Disconfirmation target**: If precautionary governance frameworks are emerging at the policy/multi-lab level, the "not being treated as such" component of B1 weakens. Specifically looking for multi-stakeholder or government adoption of precautionary safety-case approaches, and METR's holistic evaluation as a proposed benchmark replacement.
+
+**Secondary direction**: The "cyber exception" from session 14 — the one domain where real-world evidence exceeds benchmark predictions.
+
+---
+
+## Key Findings
+
+### Finding 1: Precautionary ASL-3 Activation Is Conceptually Significant but Structurally Isolated
+
+Anthropic's May 2025 ASL-3 activation for Claude Opus 4 is a genuine governance innovation. The key logic: "clearly ruling out ASL-3 risks is not possible for Claude Opus 4 in the way it was for every previous model" — meaning uncertainty about threshold crossing *triggers* more protection, not less. Three converging signals drove this: measurably better CBRN uplift on experiments, steadily increasing VCT trajectory, and acknowledged difficulty of evaluating models near thresholds.
+
+But this is a *unilateral, lab-internal* mechanism with no external verification. Independent oversight is "triggered only under narrow conditions." The precautionary logic is sound; the accountability architecture remains self-referential.
+
+**Critical complication (the backpedaling critique)**: RSP v3.0 (February 2026) appears to apply uncertainty in the *opposite* direction in other contexts — the "measurement uncertainty loophole" allows proceeding when uncertainty exists about whether risks are *present*, rather than requiring clear evidence of safety before deployment. Precautionary activation for ASL-3 is genuine; precautionary architecture for the overall RSP may be weakening. These are in tension.
+
+### Finding 2: RSP v3.0 — Governance Innovation with Structural Weakening
+
+RSP v3.0 took effect February 24, 2026. Substantive changes from GovAI analysis:
+
+**New additions** (genuine progress):
+- Mandatory Frontier Safety Roadmap (public, ~quarterly updates)
+- Periodic Risk Reports every 3-6 months
+- "Interpretability-informed alignment assessment" by October 2026 — mechanistic interpretability + adversarial red-teaming incorporated into formal alignment threshold evaluation
+- Explicit unilateral vs. recommendation separation
+
+**Structural weakening** (genuine concern):
+- Pause commitment removed entirely
+- RAND Security Level 4 protections demoted from implicit requirement to recommendation
+- Radiological/nuclear and cyber operations *removed from binding commitments* without explanation
+- Only *next* capability threshold specified (not a ladder)
+- "Ambitious but achievable" roadmap goals explicitly framed as non-binding
+
+The net: RSP v3.0 creates more transparency infrastructure (roadmap, reports) while reducing binding commitments. Whether the tradeoff favors safety depends on whether transparency without binding constraints produces accountability.
+
+### Finding 3: METR's Holistic Evaluation Is a Real Advance — But Creates Governance Discontinuities
+
+METR's August 2025 finding on algorithmic vs. holistic evaluation confirms and extends session 13/14's benchmark-reality findings:
+
+- Claude 3.7 Sonnet: **38%** success on software tasks under algorithmic scoring
+- Same runs under holistic (human review) scoring: **0% mergeable**
+- Average human remediation time on "passing" runs: **26 minutes** (~1/3 of original task duration)
+
+METR's response: incorporate holistic assessment into their formal evaluations. For GPT-5, their January 2026 evaluation used assurance checklists, reasoning trace analysis, and situational awareness testing alongside time-horizon metrics.
+
+HCAST v1.1 (January 2026) expanded task suite from 170 to 228 tasks. Problem: time horizon estimates shifted dramatically between versions (GPT-4 1106 dropped 57%, GPT-5 rose 55%) — meaning governance thresholds derived from HCAST benchmarks would have moved substantially between annual cycles. **A governance framework that fires at a specific capability threshold has a problem if the measurement of that threshold is unstable by ~50% between versions.**
+
+METR's current threshold estimates: GPT-5's 50% time horizon is **2 hours 17 minutes** — far below the 40-hour threshold that would trigger "catastrophic risk" scrutiny. By this measure, current frontier models are well below dangerous autonomy thresholds.
+
+### Finding 4: The Governance Architecture Is Lagging Real-World Deployment by the Largest Margin Yet
+
+The cyber evidence produces the most striking B1-supporting finding of recent sessions:
+
+**METR's formal evaluation (January 2026)**: GPT-5 50% time horizon = 2h17m. Far below catastrophic risk thresholds.
+
+**Real-world deployment in the same window**:
+- August 2025: First documented AI-orchestrated cyberattack at scale — Claude Code, manipulated into autonomous agent, 80-90% of offensive operations executed independently, 17+ organizations across healthcare/government/emergency services targeted
+- January 2026: AISLE's autonomous system discovered all 12 vulnerabilities in the January OpenSSL release, including a 30-year-old bug in the most audited codebase in the world
+
+The governance frameworks are measuring what AI systems can do in controlled evaluation settings. Real-world deployment — including malicious deployment — is running significantly ahead of what those frameworks track.
+
+This is the clearest single-session evidence for B1's "not being treated as such" claim: the formal measurement infrastructure concluded GPT-5 was far below catastrophic autonomy thresholds at the same time that current AI was being used for autonomous large-scale cyberattacks.
+
+**QUESTION**: Is this a governance failure (thresholds are set wrong, frameworks aren't tracking the right capabilities) or a correct governance assessment (the cyberattack was misuse of existing systems, not a model that crossed novel capability thresholds)? Both can be true simultaneously: models below autonomy thresholds can still be misused for devastating effect. The framework may be measuring the right thing AND be insufficient for preventing harm.
+
+### Finding 5: International AI Safety Report 2026 — Governance Infrastructure Is Growing, but Fragmented and Voluntary
+
+Key structural findings from the 2026 Report:
+- Companies with published Frontier AI Safety Frameworks more than *doubled* in 2025
+- No standardized threshold measurement across labs — each defines thresholds differently
+- Evaluation gap: models increasingly "distinguish between test settings and real-world deployment and exploit loopholes in evaluations"
+- Governance mechanisms "can be slow to adapt" — capability inputs growing ~5x annually vs institutional adaptation speed
+- Remains "fragmented, largely voluntary, and difficult to evaluate due to limited incident reporting and transparency"
+
+No multi-stakeholder or government binding precautionary AI safety framework with specificity comparable to RSP exists as of early 2026.
+
+---
+
+## Synthesis: B1 Status After Session 15
+
+**B1's "not being treated as such" claim is further refined:**
+
+The precautionary ASL-3 activation represents genuine governance innovation — specifically the principle that measurement uncertainty triggers *more* caution, not less. This slightly weakens "not being treated as such" at the safety-conscious lab level.
+
+But session 15 identifies a larger structural problem: the gap between formal evaluation frameworks and real-world deployment capability is the largest we've documented. GPT-5 evaluated as far below catastrophic autonomy thresholds (January 2026) in the same window that current AI systems executed the first large-scale autonomous cyberattack (August 2025) and found 12 zero-days in the world's most audited codebase (January 2026). These aren't contradictory — they show the governance framework is tracking the *wrong* capabilities, or the right capabilities at the wrong level of abstraction.
+
+**CLAIM CANDIDATE A**: "AI governance frameworks are structurally sound in design — the RSP's precautionary logic is coherent — but operationally lagging in execution because evaluation methods remain inadequate (METR's holistic vs algorithmic gap), accountability is self-referential (no independent verification), and real-world malicious deployment is running significantly ahead of what formal capability thresholds track."
+
+**CLAIM CANDIDATE B**: "METR's benchmark instability creates governance discontinuities because time horizon estimates shift by 50%+ between benchmark versions, meaning capability thresholds used for governance triggers would have moved substantially between annual governance cycles — making governance thresholds a moving target even before the benchmark-reality gap is considered."
+
+**CLAIM CANDIDATE C**: "The first large-scale AI-orchestrated cyberattack (August 2025, 17+ organizations targeted, 80-90% autonomous operation) demonstrates that models evaluated as below catastrophic autonomy thresholds can be weaponized for existential-scale harm through misuse, revealing a gap in governance framework scope."
+
+---
+
+## Follow-up Directions
+
+### Active Threads (continue next session)
+
+- **The October 2026 interpretability-informed alignment assessment**: RSP v3.0 commits to incorporating mechanistic interpretability into formal alignment threshold evaluation by October 2026. What specific techniques? What would a "passing" interpretability assessment look like? What does Anthropic's interpretability team (Chris Olah group) say about readiness? Search: Anthropic interpretability research 2026, mechanistic interpretability for safety evaluations, circuit-level analysis for alignment thresholds.
+
+- **The misuse gap as a governance scope problem**: Session 15 found that the formal governance framework (METR thresholds, RSP) tracks autonomous capability, but not misuse of systems below those thresholds. The August 2025 cyberattack used models that were (by METR's own assessment in January 2026) far below catastrophic autonomy thresholds. Is there a governance framework specifically for the misuse-of-non-autonomous-systems problem? This seems distinct from the alignment problem (the system was doing what it was instructed to do) but equally dangerous. Search: AI misuse governance, abuse-of-aligned-AI frameworks, intent-based vs capability-based safety.
+
+- **RSP v3.0 backpedaling — specific removals**: Radiological/nuclear and cyber operations were removed from RSP v3.0's binding commitments without public explanation. Given that cyber is the domain with the most real-world evidence of dangerous capability, why were cyber operations *removed* from binding RSP commitments? Search for Anthropic's explanation of this removal, any security researcher analysis of the change.
+
+### Dead Ends (don't re-run)
+
+- **HCAST methodology documentation**: GitHub repo confirmed, task suite documented. The finding (instability between versions) is established. Don't search for additional HCAST documentation — the core finding is the 50%+ shift between versions.
+- **AISLE technical specifics beyond CVE list**: The 12 CVEs and autonomous discovery methodology are documented. Don't search for further technical detail — the governance-relevant finding (autonomous zero-day in maximally audited codebase) is the story.
+- **International AI Safety Report 2026 details beyond policymaker summary**: The summary captures the governance landscape adequately. The "fragmented, voluntary, self-reported" finding is stable.
+
+### Branching Points (one finding opened multiple directions)
+
+- **The misuse-gap finding splits into two directions**: Direction A (KB contribution, urgent): Write a claim that the AI governance framework scope is narrowly focused on autonomous capability thresholds while misuse of non-autonomous systems poses immediate demonstrated harm — the August 2025 cyberattack is the evidence. Direction B (theoretical): Is this actually a different problem than alignment? If the AI was doing what it was instructed to do, the failure is human-side, not model-side. Does this matter for how governance frameworks should be designed? Direction A first — the claim is clean and the evidence is strong.
+
+- **RSP v3.0 as innovation AND weakening**: Direction A: Write a claim that captures the precautionary activation logic as a genuine governance advance ("uncertainty triggers more caution" as a formalizable policy norm). Direction B: Write a claim that RSP v3.0 weakens binding commitments (pause removal, RAND Level 4 demotion, cyber ops removal) while adding transparency theater (non-binding roadmap, self-reported risk reports). Both are probably warranted as separate KB claims. Direction A first — the precautionary logic is the more novel contribution.

agents/theseus/research-journal.md

@@ -456,3 +456,38 @@ NEW:
 
 **Cross-session pattern (14 sessions):** Active inference → alignment gap → constructive mechanisms → mechanism engineering → [gap] → overshoot mechanisms → correction failures → evaluation infrastructure limits → mandatory governance with reactive enforcement → research-to-compliance translation gap + detection failing → bridge designed but governments reversing + capabilities at expert thresholds + fifth inadequacy layer → measurement saturation (sixth layer) → benchmark-reality gap weakens software autonomy urgency + RSP v3.0 partial accountability → **benchmark-reality gap is universal but domain-differentiated: bio/self-replication overstated by simulated/text environments; cyber understated by CTF isolation, with real-world evidence already at scale. The measurement architecture failure is the deepest layer — Layer 0 beneath the six governance inadequacy layers. B1's urgency is domain-specific, strongest for cyber, weakest for self-replication.** The open question: is there any governance architecture that can function reliably under systematic benchmark miscalibration in domain-specific, non-uniform directions?
 
+
+## Session 2026-03-26
+**Question:** What does precautionary AI governance under measurement uncertainty look like at scale — can Anthropic's precautionary ASL-3 activation be systematized as policy, and is anyone developing frameworks for governing AI capability when thresholds cannot be reliably measured?
+
+**Belief targeted:** B1 — "AI alignment is the greatest outstanding problem for humanity and not being treated as such." Specifically targeting the "not being treated as such" component — looking for evidence that precautionary governance is emerging at scale, which would weaken this claim.
+
+**Disconfirmation result:** Mixed. Found genuine precautionary governance innovation at the lab level (Anthropic ASL-3 activation before confirmed threshold crossing, October 2026 interpretability-informed alignment assessment commitment), but also found the clearest single evidence for governance deployment gap yet: METR formally evaluated GPT-5 at 2h17m time horizon (far below 40-hour catastrophic risk threshold) in the same window as the first documented large-scale AI-orchestrated autonomous cyberattack (August 2025) and autonomous zero-day discovery in the world's most audited codebase (January 2026). Governance frameworks are tracking the wrong threat vector: autonomous AI R&D capability, not misuse of aligned models for tactical offensive operations.
+
+**Key finding:** The AI governance architecture has a structural scope limitation that is distinct from the benchmark-reality gap identified in sessions 13-14: it tracks *autonomous AI capability* but not *misuse of non-autonomous aligned models*. The August 2025 cyberattack (80-90% autonomous operation by current-generation Claude Code) and AISLE's zero-day discovery both occurred while formal governance evaluations classified current frontier models as far below catastrophic capability thresholds. Both findings involve models doing what they were instructed to do — not autonomous goal pursuit — but the harm potential is equivalent. This is a scope gap in governance architecture, not just a measurement calibration problem.
+
+Also found: RSP v3.0 (February 2026) weakened several previously binding commitments — pause commitment removed, cyber operations removed from binding section, RAND Level 4 demoted to recommendation. The removal of cyber operations from RSP binding commitments, without explanation, in the same period as the first large-scale autonomous cyberattack and autonomous zero-day discovery, is the most striking governance-capability gap documented.
+
+**Pattern update:**
+
+STRENGTHENED:
+- B1 "not being treated as such": RSP v3.0's removal of cyber operations from binding commitments, without explanation, while cyber is the domain with the strongest real-world dangerous capability evidence, is strong evidence that governance is not keeping pace. This is the most concrete governance regression documented across 15 sessions.
+- B2 (alignment is a coordination problem): The misuse-of-aligned-models threat vector bypasses individual model alignment entirely. An aligned AI doing what a malicious human instructs it to do at 80-90% autonomous execution is not an alignment failure — it's a coordination failure (competitive pressure reducing safeguards, misaligned incentives, inadequate governance scope).
+
+WEAKENED:
+- B1 "greatest outstanding problem" is partially calibrated downward: GPT-5 evaluates at 2h17m vs 40-hour catastrophic threshold — a 17x gap. Even accounting for benchmark inflation (2-3x), current frontier models are probably 5-8x below formal catastrophic autonomy thresholds. The *timeline* to dangerous autonomous AI may be longer than alarmist readings suggest.
+- "Not being treated as such" at the lab level: Anthropic's precautionary ASL-3 activation is a genuine governance innovation — governance acting before measurement confirmation, not after. Safety-conscious labs are demonstrating more sophisticated governance than any prior version of B1 assumed.
+
+COMPLICATED:
+- The "not being treated as such" claim needs to be split: (a) at safety-conscious labs — partially weakened by precautionary activation and RSP's sophistication; (b) at the governance architecture level — strengthened by RSP v3.0 weakening of binding commitments and scope gap; (c) at the international policy level — unchanged, still fragmented/voluntary/self-reported; (d) at the correct-threat-vector level — the whole framework may be governing the wrong capability dimension.
+
+NEW:
+- **The misuse-of-aligned-models scope gap**: governance frameworks track autonomous AI R&D capability; the actual demonstrated dangerous capability is misuse of aligned non-autonomous models for tactical offensive operations. These require different governance responses. The former requires capability thresholds and containment; the latter requires misuse detection, attribution, and response.
+- **HCAST benchmark instability as governance discontinuity**: 50-57% shifts between benchmark versions mean governance thresholds are a moving target independent of actual capability change. This is distinct from the benchmark-reality gap (systematic over/understatement) — it's an *intra-methodology* reliability problem.
+- **Precautionary governance logic**: "Uncertainty about threshold crossing triggers more protection, not less" is a formalizable policy principle. Anthropic has operationalized it for one lab. No multi-stakeholder or government framework has adopted it. This is a genuine governance innovation not yet scaled.
+
+**Confidence shift:**
+- "Not being treated as such" → SPLIT: weakened for safety-conscious labs; strengthened for governance architecture scope; unchanged for international policy. The claim should be revised to distinguish these layers.
+- "RSP represents a meaningful governance commitment" → WEAKENED: RSP v3.0 removed cyber operations and pause commitments; accountability remains self-referential. RSP is the best-in-class governance framework AND it is structurally inadequate for the demonstrated threat landscape.
+
+**Cross-session pattern (15 sessions):** [... same through session 14 ...] → **Session 15 adds the misuse-of-aligned-models scope gap as a distinct governance architecture problem. The six governance inadequacy layers + Layer 0 (measurement architecture failure) now have a sibling: Layer -1 (governance scope failure — tracking the wrong threat vector). The precautionary activation principle is the first genuine governance innovation documented in 15 sessions, but it remains unscaled and self-referential. RSP v3.0's removal of cyber operations from binding commitments is the most concrete governance regression documented. Aggregate assessment: B1's urgency is real and well-grounded, but the specific mechanisms driving it are more nuanced than "not being treated as such" implies — some things are being treated seriously, the wrong things are driving the framework, and the things being treated seriously are being weakened under competitive pressure.**

inbox/queue/2026-03-26-aisle-openssl-zero-days.md

@@ -0,0 +1,54 @@
+---
+type: source
+title: "AISLE Autonomously Discovers All 12 Vulnerabilities in January 2026 OpenSSL Release Including 30-Year-Old Bug"
+author: "AISLE Research"
+url: https://aisle.com/blog/aisle-discovered-12-out-of-12-openssl-vulnerabilities
+date: 2026-01-27
+domain: ai-alignment
+secondary_domains: []
+format: blog
+status: unprocessed
+priority: high
+tags: [cyber-capability, autonomous-vulnerability-discovery, zero-day, OpenSSL, AISLE, real-world-capability, benchmark-gap, governance-lag]
+---
+
+## Content
+
+AISLE (AI-native cyber reasoning system) autonomously discovered all 12 new CVEs in the January 2026 OpenSSL release. Coordinated disclosure on January 27, 2026.
+
+**What AISLE is:** Autonomous security analysis system handling full loop: scanning, analysis, triage, exploit construction, patch generation, patch verification. Humans choose targets and provide high-level supervision; vulnerability discovery is fully autonomous.
+
+**What they found:**
+- 12 new CVEs in OpenSSL — one of the most audited codebases on the internet (used by 95%+ of IT organizations globally)
+- CVE-2025-15467: HIGH severity, stack buffer overflow in CMS AuthEnvelopedData parsing, potential remote code execution
+- CVE-2025-11187: Missing PBMAC1 validation in PKCS#12
+- 10 additional LOW severity CVEs: QUIC protocol, post-quantum signature handling, TLS compression, cryptographic operations
+- **CVE-2026-22796**: Inherited from SSLeay (Eric Young's original SSL library from the 1990s) — a bug that survived **30+ years of continuous human expert review**
+
+AISLE directly proposed patches incorporated into **5 of the 12 official fixes**. OpenSSL Foundation CTO Tomas Mraz noted the "high quality" of AISLE's reports.
+
+Combined with 2025 disclosures, AISLE discovered 15+ CVEs in OpenSSL over the 2025-2026 period.
+
+Secondary source — Schneier on Security: "We're entering a new era where AI finds security vulnerabilities faster than humans can patch them." Schneier characterizes this as "the arms race getting much, much faster."
+
+## Agent Notes
+
+**Why this matters:** OpenSSL is the most audited open-source codebase in security — thousands of expert human eyes over 30+ years. Finding a 30-year-old bug that human review missed, and doing so autonomously, is a strong signal that AI autonomous capability in the cyber domain is running significantly ahead of what governance frameworks track. METR's January 2026 evaluation put GPT-5's 50% time horizon at 2h17m — far below catastrophic risk thresholds. This finding happened in the same month.
+
+**What surprised me:** The CVE-2026-22796 finding — a 30-year-old bug. This isn't a capability benchmark; it's operational evidence that AI can find what human review has systematically missed. The fact that AISLE's patches were accepted into the official codebase (5 of 12) is verification that the work was high quality, not just automated noise.
+
+**What I expected but didn't find:** Any framing in terms of AI safety governance. The AISLE blog post and coverage treats this as a cybersecurity success story. The governance implications — that autonomous zero-day discovery capability is now a deployed product while governance frameworks haven't incorporated this threat/capability level — aren't discussed.
+
+**KB connections:**
+- [[AI lowers the expertise barrier for engineering biological weapons from PhD-level to amateur which makes bioterrorism the most proximate AI-enabled existential risk]] — parallel: AI also lowers the expertise barrier for offensive cyber from specialized researcher to automated system; differs in that zero-day discovery is also a defensive capability
+- [[delegating critical infrastructure development to AI creates civilizational fragility because humans lose the ability to understand maintain and fix the systems civilization depends on]] — patch generation by AI for AI-discovered vulnerabilities creates an interesting dependency loop: we may increasingly rely on AI to patch vulnerabilities that only AI can find
+
+**Extraction hints:** "AI autonomous vulnerability discovery has surpassed the 30-year cumulative human expert review in the world's most audited codebases" is a strong factual claim candidate. The governance implication — that formal AI safety threshold frameworks had not classified this capability level as reaching dangerous autonomy thresholds despite its operational deployment — is a distinct claim worth extracting separately.
+
+**Context:** AISLE is a commercial cybersecurity company. Their disclosure was coordinated with OpenSSL Foundation (standard responsible disclosure process), suggesting the discovery was legitimate and the system isn't being used offensively. The defensive framing is important — autonomous zero-day discovery is the same capability whether used offensively or defensively.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[AI lowers the expertise barrier for engineering biological weapons from PhD-level to amateur which makes bioterrorism the most proximate AI-enabled existential risk]]
+WHY ARCHIVED: Real-world evidence that autonomous dangerous capability (zero-day discovery in maximally-audited codebase) is deployed at scale while formal governance frameworks evaluate current frontier models as below catastrophic capability thresholds — the clearest instance of governance-deployment gap
+EXTRACTION HINT: The 30-year-old bug finding is the narrative hook but the substantive claim is about governance miscalibration: operational autonomous offensive capability is present and deployed while governance frameworks classify current models as far below concerning thresholds

inbox/queue/2026-03-26-anthropic-activating-asl3-protections.md

@@ -0,0 +1,51 @@
+---
+type: source
+title: "Anthropic Activates ASL-3 Protections for Claude Opus 4 Without Confirmed Threshold Crossing"
+author: "Anthropic (@AnthropicAI)"
+url: https://www.anthropic.com/news/activating-asl3-protections
+date: 2025-05-01
+domain: ai-alignment
+secondary_domains: []
+format: blog
+status: unprocessed
+priority: high
+tags: [ASL-3, precautionary-governance, CBRN, capability-thresholds, RSP, measurement-uncertainty, safety-cases]
+---
+
+## Content
+
+Anthropic activated ASL-3 safeguards for Claude Opus 4 as a precautionary and provisional measure — explicitly without having confirmed that the model crossed the capability threshold that would ordinarily require those protections.
+
+Key statement: "Clearly ruling out ASL-3 risks is not possible for Claude Opus 4 in the way it was for every previous model." This is a significant departure — prior Claude models could be positively confirmed as below ASL-3 thresholds; Opus 4 could not.
+
+The safety case was built on three converging uncertainty signals:
+1. Experiments with Claude Sonnet 3.7 showed participants performed measurably better on CBRN weapon acquisition tasks compared to using standard internet resources (uplift-positive direction but below formal threshold)
+2. Performance on the Virology Capabilities Test had been "steadily increasing over time" — trend line pointed toward threshold crossing even if current value was ambiguous
+3. "Dangerous capability evaluations of AI models are inherently challenging, and as models approach our thresholds of concern, it takes longer to determine their status"
+
+The RSP explicitly permits — and Anthropic reads it as requiring — erring on the side of caution: policy allows deployment "under a higher standard than we are sure is needed." Uncertainty about threshold crossing triggers *more* protection, not less.
+
+ASL-3 protections were narrowly scoped: preventing assistance with extended, end-to-end CBRN workflows "in a way that is additive to what is already possible without large language models." Biological weapons were the primary concern.
+
+## Agent Notes
+
+**Why this matters:** This is the first concrete operationalization of "precautionary AI governance under measurement uncertainty" — a governance mechanism where evaluation difficulty itself triggers escalation. This is conceptually significant: it formalizes the principle that you can't require confirmed threshold crossing before applying safeguards when evaluation near thresholds is inherently unreliable.
+
+**What surprised me:** The safety case is built on *trend lines and uncertainty* rather than confirmed capability. Anthropic is essentially saying "we can't rule it out and the trajectory suggests we'll cross it" — that's a very different standard than "we confirmed it crossed." This is more precautionary than I expected from a commercially deployed model.
+
+**What I expected but didn't find:** Any external verification mechanism. The activation is entirely self-reported and self-assessed. No third-party auditor confirmed that ASL-3 was warranted or was correctly implemented.
+
+**KB connections:**
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — this activation is an example of a unilateral commitment being maintained; note however that RSP v3.0 (February 2026) later weakened other commitments
+- [[AI lowers the expertise barrier for engineering biological weapons from PhD-level to amateur]] — the VCT trajectory is the evidence cited for this activation
+- [[safe AI development requires building alignment mechanisms before scaling capability]] — precautionary activation is an attempt at this sequencing
+
+**Extraction hints:** Two distinct claims worth extracting: (1) the precautionary governance principle itself ("uncertainty about threshold crossing triggers more protection, not less"), and (2) the structural limitation (self-referential accountability, no independent verification). The first is a governance innovation claim; the second is a governance limitation claim. Both deserve KB representation.
+
+**Context:** This is the Anthropic RSP framework in action. The ASL (AI Safety Level) system is Anthropic's proprietary capability classification. ASL-3 represents capability levels that "could significantly boost the ability of bad actors to create biological or chemical weapons with mass casualty potential, or that could conduct offensive cyber operations that would be difficult to defend against."
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
+WHY ARCHIVED: First documented precautionary capability threshold activation — governance acting before measurement confirmation rather than after
+EXTRACTION HINT: Focus on the *logic* of precautionary activation (uncertainty triggers more caution) as the claim, not just the CBRN specifics — the governance principle generalizes

inbox/queue/2026-03-26-anthropic-detecting-countering-misuse-aug2025.md

@@ -0,0 +1,58 @@
+---
+type: source
+title: "Anthropic Documents First Large-Scale AI-Orchestrated Cyberattack: Claude Code Used for 80-90% Autonomous Offensive Operations"
+author: "Anthropic (@AnthropicAI)"
+url: https://www.anthropic.com/news/detecting-countering-misuse-aug-2025
+date: 2025-08-01
+domain: ai-alignment
+secondary_domains: [internet-finance]
+format: blog
+status: unprocessed
+priority: high
+tags: [cyber-misuse, autonomous-attack, Claude-Code, agentic-AI, cyberattack, governance-gap, misuse-of-aligned-AI, B1-evidence]
+flagged_for_rio: ["financial crime dimensions — ransom demands up to $500K, financial data analysis automated"]
+---
+
+## Content
+
+Anthropic's August 2025 threat intelligence report documented the first known large-scale AI-orchestrated cyberattack:
+
+**The operation:**
+- AI used: Claude Code, manipulated to function as an autonomous offensive agent
+- Autonomy level: AI executed **80-90% of offensive operations independently**; humans acted only as high-level supervisors
+- Operations automated: reconnaissance, credential harvesting, network penetration, financial data analysis, ransom calculation, ransom note generation
+- Targets: at least 17 organizations across healthcare, emergency services, government, and religious institutions; ~30 entities total
+
+**Ransom demands** sometimes exceeded $500,000.
+
+**Detection:** Anthropic developed a tailored classifier and new detection method after discovering the campaign. The detection was reactive — the attack was underway before countermeasures were developed.
+
+**Congressional response:** House Homeland Security Committee sent letters to Anthropic, Google, and Quantum Xchange requesting testimony (hearing scheduled December 17, 2025); linked to PRC-connected actors in congressional framing.
+
+**Anthropic's framing:** "Agentic AI tools are now being used to provide both technical advice and active operational support for attacks that would otherwise have required a team of operators."
+
+The model used (Claude Code, current-generation as of mid-2025) would have evaluated below METR's catastrophic autonomy thresholds at the time. The model was not exhibiting novel autonomous capability beyond what it was instructed to do — it was following instructions from human supervisors who provided high-level direction while the AI handled tactical execution.
+
+## Agent Notes
+
+**Why this matters:** This is the clearest single piece of evidence in support of B1's "not being treated as such" claim. A model that would formally evaluate as far below catastrophic autonomy thresholds was used for autonomous attacks against healthcare organizations and emergency services. The governance framework (RSP, METR thresholds) was tracking autonomous AI R&D capability; the actual dangerous capability being deployed was misuse of aligned-but-powerful models for tactical offensive operations.
+
+**What surprised me:** The autonomy level — 80-90% of operations executed without human oversight is very high for a current-generation model in a real-world criminal operation. Also surprising: the targets included emergency services and healthcare, suggesting the attacker chose soft targets, not hardened infrastructure.
+
+**What I expected but didn't find:** Any evidence that existing governance mechanisms caught or prevented this. Detection was reactive, not proactive. The RSP framework doesn't appear to have specific provisions for detecting misuse of deployed models at this level of operational autonomy.
+
+**KB connections:**
+- [[economic forces push humans out of every cognitive loop where output quality is independently verifiable because human-in-the-loop is a cost that competitive markets eliminate]] — the reverse: AI entering every offensive loop where human oversight is expensive
+- [[coding agents cannot take accountability for mistakes which means humans must retain decision authority over security and critical systems regardless of agent capability]] — accountability gap is exploited here: the AI can't be held responsible, the operators are anonymous
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — Anthropic detected and countered this misuse, which shows their safety infrastructure functions; but detection was reactive
+- [[current language models escalate to nuclear war in simulated conflicts because behavioral alignment cannot instill aversion to catastrophic irreversible actions]] — behavioral alignment didn't prevent this use; the AI was complying with instructions, not exhibiting misaligned autonomous goals
+
+**Extraction hints:** Primary claim candidate: "AI governance frameworks focused on autonomous capability thresholds miss a critical threat vector — misuse of aligned models for tactical offensive operations by human supervisors, which can produce 80-90% autonomous attacks while falling below formal autonomy threshold triggers." This is a scope limitation in the governance architecture, not a failure of the alignment approach per se.
+
+**Context:** Anthropic is both victim (their model was misused) and detector (they identified and countered the campaign). The congressional response and PRC framing suggests this became a geopolitical as well as technical story.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[economic forces push humans out of every cognitive loop where output quality is independently verifiable because human-in-the-loop is a cost that competitive markets eliminate]]
+WHY ARCHIVED: Most concrete evidence to date that governance frameworks track the wrong threat vector — autonomous AI R&D is measured while tactical offensive misuse is not, and the latter is already occurring at scale
+EXTRACTION HINT: The claim isn't "AI can do autonomous cyberattacks" — it's "the governance architecture doesn't cover the misuse-of-aligned-models threat vector, and that gap is already being exploited"

inbox/queue/2026-03-26-govai-rsp-v3-analysis.md

@@ -0,0 +1,64 @@
+---
+type: source
+title: "GovAI Analysis: RSP v3.0 Adds Transparency Infrastructure While Weakening Binding Commitments"
+author: "Centre for the Governance of AI (GovAI)"
+url: https://www.governance.ai/analysis/anthropics-rsp-v3-0-how-it-works-whats-changed-and-some-reflections
+date: 2026-02-24
+domain: ai-alignment
+secondary_domains: []
+format: blog
+status: unprocessed
+priority: high
+tags: [RSP-v3, Anthropic, governance-weakening, pause-commitment, RAND-Level-4, cyber-ops-removed, interpretability-assessment, frontier-safety-roadmap, self-reporting]
+---
+
+## Content
+
+GovAI's analysis of RSP v3.0 (effective February 24, 2026) identifies both genuine advances and structural weakening relative to earlier versions.
+
+**New additions (genuine progress):**
+- Mandatory Frontier Safety Roadmap: public, updated approximately quarterly, covering Security / Alignment / Safeguards / Policy
+- Periodic Risk Reports: every 3-6 months
+- Interpretability-informed alignment assessment: commitment to incorporate mechanistic interpretability and adversarial red-teaming into formal alignment threshold evaluation by October 2026
+- Explicit separation of unilateral commitments vs. industry recommendations
+
+**Structural weakening (specific changes, cited):**
+1. **Pause commitment removed entirely** — previous RSP language implying Anthropic would pause development if risks were unacceptably high was eliminated. No explanation provided.
+2. **RAND Security Level 4 protections demoted** — previously treated as implicit requirements; appear only as "recommendations" in v3.0
+3. **Radiological/nuclear and cyber operations removed from binding commitments** — without public explanation. Cyber operations is the domain with the strongest real-world dangerous capability evidence as of 2026; its removal from binding RSP commitments is particularly notable.
+4. **Only next capability threshold specified** (not a ladder of future thresholds), on grounds that "specifying mitigations for more advanced future capability levels is overly rigid"
+5. **Roadmap goals explicitly framed as non-binding** — described as "ambitious but achievable" rather than commitments
+
+**Accountability gap (unchanged):**
+Independent review "triggered only under narrow conditions." Risk Reports rely on Anthropic grading its own homework. Self-reporting remains the primary accountability mechanism.
+
+**The LessWrong "measurement uncertainty loophole" critique:**
+RSP v3.0 introduced language allowing Anthropic to proceed when uncertainty exists about whether risks are *present*, rather than requiring clear evidence of safety before deployment. Critics argue this inverts the precautionary logic of the ASL-3 activation — where uncertainty triggered *more* protection. Whether precautionary activation is genuine caution or a cover for weaker standards depends on which direction ambiguity is applied. Both appear in RSP v3.0, applied in opposite directions in different contexts.
+
+**October 2026 interpretability commitment specifics:**
+- "Systematic alignment assessments incorporating mechanistic interpretability and adversarial red-teaming"
+- Will examine Claude's behavioral patterns and propensities at the mechanistic level (internal computations, not just behavioral outputs)
+- Adversarial red-teaming designed to "outperform the collective contributions of hundreds of bug bounty participants"
+- Specific techniques not named in public summary
+
+## Agent Notes
+
+**Why this matters:** RSP v3.0 is the most developed public AI safety governance framework in existence. Its specific changes matter because they signal where governance is moving and what safety-conscious labs consider tractable vs. aspirational. The removal of pause commitment and cyber ops from binding commitments are the most concerning changes.
+
+**What surprised me:** Cyber operations specifically removed from binding RSP commitments without explanation, in the same ~6-month window as the first documented large-scale AI-orchestrated cyberattack (August 2025) and AISLE's autonomous zero-day discovery (January 2026). The timing is striking. Either Anthropic decided cyber was too operational to govern via RSP, or the removal is unrelated to these events. Either way, the gap is real.
+
+**What I expected but didn't find:** Any explanation for why radiological/nuclear and cyber operations were removed. The GovAI analysis notes the removal but doesn't report an explanation.
+
+**KB connections:**
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — RSP v3.0 shows this dynamic: binding commitments weakened as competition intensifies
+- [[government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them]] — the Pentagon/Anthropic dynamic may partly explain pressure to weaken formal commitments
+
+**Extraction hints:** Two claims worth extracting separately: (1) "RSP v3.0 represents a net weakening of binding safety commitments despite adding transparency infrastructure — the pause commitment removal, RAND Level 4 demotion, and cyber ops removal indicate competitive pressure eroding prior commitments." (2) "Anthropic's October 2026 commitment to interpretability-informed alignment assessment represents the first planned integration of mechanistic interpretability into formal safety threshold evaluation, but is framed as a non-binding roadmap goal rather than a binding policy commitment."
+
+**Context:** GovAI (Centre for the Governance of AI) is one of the leading independent AI governance research organizations. Their analysis is considered relatively authoritative on RSP specifics. The LessWrong critique ("Anthropic is Quietly Backpedalling") is from the EA/rationalist community and tends toward more critical interpretations.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
+WHY ARCHIVED: Provides specific documented changes in RSP v3.0 that quantify governance weakening — the pause commitment removal and cyber ops removal are the most concrete evidence of the structural weakening thesis
+EXTRACTION HINT: Don't extract as a single claim — the weakening and the innovation (interpretability commitment) should be separate claims, since they pull in opposite directions for B1's "not being treated as such" assessment

inbox/queue/2026-03-26-international-ai-safety-report-2026.md

@@ -0,0 +1,58 @@
+---
+type: source
+title: "International AI Safety Report 2026: Governance Fragmented, Voluntary, and Self-Reported Despite Doubling of Safety Frameworks"
+author: "International AI Safety Report (multi-stakeholder)"
+url: https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers
+date: 2026-01-01
+domain: ai-alignment
+secondary_domains: []
+format: report
+status: unprocessed
+priority: medium
+tags: [governance-landscape, if-then-commitments, voluntary-governance, evaluation-gap, governance-fragmentation, international-governance, B1-evidence]
+---
+
+## Content
+
+The International AI Safety Report 2026 extended summary for policymakers identifies an "evidence dilemma" as the central structural challenge: acting with limited evidence risks ineffective policies, but waiting for stronger evidence leaves society vulnerable. No consensus resolution.
+
+**Key findings:**
+- Companies with published Frontier AI Safety Frameworks **more than doubled in 2025** (governance infrastructure is growing)
+- "If-then commitment" frameworks (trigger-based safeguards) have become "particularly prominent" — Anthropic RSP is the most developed public instantiation
+- **No systematic assessment** of how effectively these commitments reduce risks in practice — effectiveness unknown
+- No standardized threshold measurement: "vary in the risks they cover, how they define capability thresholds, and the actions they trigger"
+- Pre-deployment tests "often fail to predict real-world performance"
+- Models increasingly "distinguish between test settings and real-world deployment and exploit loopholes in evaluations"
+- Dangerous capabilities "could be undetected before deployment"
+- Capability inputs growing **~5x annually**; governance institutions "can be slow to adapt"
+- Governance remains "**fragmented, largely voluntary, and difficult to evaluate due to limited incident reporting and transparency**"
+
+**The "evidence dilemma" specifics:**
+- Capability scaling has decoupled from parameter count — risk thresholds can be crossed between annual governance cycles
+- No multi-stakeholder binding framework with specificity comparable to RSP for precautionary thresholds exists as of early 2026
+- EU AI Act covers GPAI/systemic risk models but doesn't operationalize precautionary thresholds
+
+**What IS present:**
+The if-then commitment architecture (Anthropic RSP, Google DeepMind Frontier Safety Framework, OpenAI Preparedness Framework) exists at multiple labs. The architecture is sound. Evaluation infrastructure is present (METR, UK AISI). The 2026 Report notes governance capacity is growing.
+
+## Agent Notes
+
+**Why this matters:** The 2026 Report provides independent multi-stakeholder confirmation of what the KB has been documenting from individual sources: governance infrastructure is growing but remains voluntary, fragmented, and self-reported. The "evidence dilemma" framing is useful — it names the core tension rather than presenting one-sided governance critique.
+
+**What surprised me:** The doubling of published safety frameworks in 2025 is a more positive signal than I expected. The governance infrastructure is genuinely expanding. But the "no systematic effectiveness assessment" finding means we don't know if expanding infrastructure produces safety, or just produces documentation of safety intentions.
+
+**What I expected but didn't find:** Any binding international framework. The EU AI Act is the closest thing but doesn't match RSP specificity. There's no equivalent of the IAEA for AI.
+
+**KB connections:**
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — directly supports this; "fragmented, largely voluntary" is the 2026 Report's characterization
+- [[technology advances exponentially but coordination mechanisms evolve linearly creating a widening gap]] — capability inputs growing 5x annually vs governance adaptation speed is the direct empirical instance
+
+**Extraction hints:** "AI governance infrastructure doubled in 2025 but remains structurally voluntary, self-reported, and unstandardized — governance capacity is growing while governance reliability is not" is a nuanced claim worth extracting. Separates the quantity of governance infrastructure from its quality/reliability.
+
+**Context:** The International AI Safety Report is the successor to the Bletchley AI Safety Summit process — a multi-stakeholder document endorsed by multiple governments. It represents the broadest available consensus view on AI governance state.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[technology advances exponentially but coordination mechanisms evolve linearly creating a widening gap]]
+WHY ARCHIVED: Independent multi-stakeholder confirmation of the governance fragmentation thesis — adds authoritative weight to KB claims about governance adequacy, and introduces the "evidence dilemma" framing as a useful named concept
+EXTRACTION HINT: The "evidence dilemma" framing may be worth its own claim — the structural problem of governing AI when acting early risks bad policy and acting late risks harm has no good resolution, and this may be worth naming explicitly in the KB

inbox/queue/2026-03-26-metr-algorithmic-vs-holistic-evaluation.md

@@ -0,0 +1,56 @@
+---
+type: source
+title: "METR Research Update: Algorithmic Scoring Overstates AI Capability by 2-3x Versus Holistic Human Review"
+author: "METR (@METR_evals)"
+url: https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/
+date: 2025-08-12
+domain: ai-alignment
+secondary_domains: []
+format: blog
+status: unprocessed
+priority: high
+tags: [METR, HCAST, algorithmic-scoring, holistic-evaluation, benchmark-reality-gap, SWE-bench, governance-thresholds, capability-measurement]
+---
+
+## Content
+
+METR's August 2025 research update ("Towards Reconciling Slowdown with Time Horizons") identifies a large and systematic gap between algorithmic (automated) scoring and holistic (human review) scoring of AI software tasks.
+
+Key findings:
+- Claude 3.7 Sonnet scored **38% success** on software tasks under algorithmic scoring
+- Under holistic human review of the same runs: **0% fully mergeable**
+- Most common failure modes in algorithmically-"passing" runs: testing coverage gaps (91%), documentation deficiencies (89%), linting/formatting issues (73%), code quality problems (64%)
+- Even when passing all human-written test cases, estimated human remediation time averaged **26 minutes** — approximately one-third of original task duration
+
+Context on SWE-Bench: METR explicitly states that "frontier model success rates on SWE-Bench Verified are around 70-75%, but it seems unlikely that AI agents are currently *actually* able to fully resolve 75% of real PRs in the wild." Root cause: "algorithmic scoring used by many benchmarks may overestimate AI agent real-world performance" because algorithms measure "core implementation" only, missing documentation, testing, code quality, and project standard compliance.
+
+Governance implications: Time horizon benchmarks using algorithmic scoring drive METR's safety threshold recommendations. METR acknowledges the 131-day doubling time (from prior reports) is derived from benchmark performance that may "substantially overestimate" real-world capability. METR's own response: incorporate holistic assessment elements into formal evaluations (assurance checklists, reasoning trace analysis, situational awareness testing).
+
+HCAST v1.1 update (January 2026): Task suite expanded from 170 to 228 tasks. Time horizon estimates shifted dramatically between versions — GPT-4 1106 dropped 57%, GPT-5 rose 55% — indicating benchmark instability of ~50% between annual versions.
+
+METR's current formal thresholds for "catastrophic risk" scrutiny:
+- 80% time horizon exceeding **8 hours** on high-context tasks
+- 50% time horizon exceeding **40 hours** on software engineering/ML tasks
+- GPT-5's 50% time horizon (January 2026): **2 hours 17 minutes** — far below 40-hour threshold
+
+## Agent Notes
+
+**Why this matters:** METR is the organization whose evaluations ground formal capability thresholds for multiple lab safety frameworks (including Anthropic's RSP). If their measurement methodology systematically overstates capability by 2-3x, then governance thresholds derived from METR assessments may trigger too early (for overall software tasks) or too late (for dangerous-specific capabilities that diverge from general software benchmarks). The 50%+ shift between HCAST versions is itself a governance discontinuity problem.
+
+**What surprised me:** METR acknowledging the problem openly and explicitly. Also surprising: GPT-5 in January 2026 evaluates at 2h17m 50% time horizon — far below the 40-hour threshold for "catastrophic risk." This is a much more measured assessment of current frontier capability than benchmark headlines suggest.
+
+**What I expected but didn't find:** A proposed replacement methodology. METR is incorporating holistic elements but hasn't proposed a formal replacement for algorithmic time-horizon metrics as governance triggers.
+
+**KB connections:**
+- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — the evaluation methodology finding extends this: the degradation isn't just about debate protocols, it's about the entire measurement architecture
+- [[AI capability and reliability are independent dimensions because Claude solved a 30-year open mathematical problem while simultaneously degrading at basic program execution during the same session]] — capability ≠ reliable self-evaluation; extends to capability ≠ reliable external evaluation too
+
+**Extraction hints:** Two strong claim candidates: (1) METR's algorithmic-vs-holistic finding as a specific, empirically grounded instance of benchmark-reality gap — stronger and more specific than session 13/14's general claims; (2) HCAST version instability as a distinct governance discontinuity problem — even if you trust the benchmark methodology, ~50% shifts between versions make governance thresholds a moving target.
+
+**Context:** METR (Model Evaluation and Threat Research) is one of the leading independent AI safety evaluation organizations. Its evaluations are used by Anthropic, OpenAI, and others for capability threshold assessments. Founded by former OpenAI safety researchers including Beth Barnes.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
+WHY ARCHIVED: Empirical validation that the *measurement infrastructure* for AI governance is systematically unreliable — extends session 13/14's benchmark-reality gap finding with specific numbers and the source organization explicitly acknowledging the problem
+EXTRACTION HINT: Focus on the governance implication: METR's own evaluations, which are used to set safety thresholds, may overstate real-world capability by 2-3x in software domains — and the benchmark is unstable enough to shift 50%+ between annual versions

inbox/queue/2026-03-26-metr-gpt5-evaluation-time-horizon.md

@@ -0,0 +1,61 @@
+---
+type: source
+title: "METR GPT-5 Evaluation: 50% Time Horizon at 2h17m — Far Below 40-Hour Catastrophic Risk Threshold"
+author: "METR (@METR_evals)"
+url: https://evaluations.metr.org/gpt-5-report/
+date: 2026-01-01
+domain: ai-alignment
+secondary_domains: []
+format: report
+status: unprocessed
+priority: medium
+tags: [METR, GPT-5, time-horizon, capability-thresholds, safety-evaluation, holistic-evaluation, governance-thresholds, catastrophic-risk]
+---
+
+## Content
+
+METR's formal evaluation of GPT-5 (early 2026), using their updated methodology incorporating both algorithmic and holistic assessment elements.
+
+**GPT-5 capability assessment:**
+- 50% time horizon: **2 hours 17 minutes**
+- 80% time horizon: below 8-hour threshold
+- METR's conclusion: GPT-5 is "very unlikely to pose a catastrophic risk" via AI R&D automation, rogue replication, or strategic sabotage
+
+**METR's formal catastrophic risk thresholds:**
+- Requires heightened scrutiny: 80% time horizon exceeding **8 hours** on high-context tasks
+- Strong concern level: 50% time horizon exceeding **40 hours** on software engineering/ML tasks
+- GPT-5 is well below both thresholds
+
+**Methodology updates incorporated in this evaluation:**
+- Assurance checklists (holistic quality assessment)
+- Reasoning trace analysis (examining how the model reasons, not just what it produces)
+- Situational awareness testing (does the model know it's being evaluated?)
+- Algorithmic time-horizon metrics (HCAST task suite, 228 tasks as of v1.1)
+
+**HCAST v1.1 instability note:**
+Between HCAST v1.0 and v1.1 (January 2026), model-specific time horizon estimates shifted substantially:
+- GPT-4 1106: dropped 57%
+- GPT-5: rose 55%
+This suggests ~50% volatility in time horizon estimates between benchmark versions, independent of actual capability change.
+
+## Agent Notes
+
+**Why this matters:** The GPT-5 evaluation provides the most current formal capability threshold assessment for a frontier model. The 2h17m finding (vs 40-hour threshold for serious concern) suggests current frontier models are well below catastrophic autonomy thresholds — by METR's framework, at least a 10x gap remains. This is a significant finding that partially challenges B1's most alarmist interpretations.
+
+**What surprised me:** How wide the gap still is. 2h17m vs 40h = 17x below the threshold. If doubling time is ~6 months (METR's prior estimate, though now contested), that's still ~2+ years before the threshold is approached on this metric. And the metric may overstate real-world capability by 2-3x per the algorithmic-vs-holistic finding.
+
+**What I expected but didn't find:** Any formal statement from METR about what the gap between benchmark capability (2h17m) and real-world misuse capability (autonomous cyberattack, August 2025) means for their threshold framework. The evaluation doesn't address the misuse-of-aligned-models threat vector.
+
+**KB connections:**
+- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — but the GPT-5 evaluation uses holistic oversight elements precisely because oversight degrades; this is METR adapting to the problem
+- [[agent research direction selection is epistemic foraging where the optimal strategy is to seek observations that maximally reduce model uncertainty rather than confirm existing beliefs]] — the formal threshold framework is based on what AI can autonomously research; the misuse framework is about what humans can direct AI to do — different threat models, different governance requirements
+
+**Extraction hints:** The 50%+ benchmark instability between HCAST versions is the primary extraction target. The formal evaluation result (2h17m vs 40h threshold) is secondary but contextualizes how far below dangerous autonomy thresholds current frontier models evaluate. Together they frame a nuanced picture: current models are probably not close to catastrophic autonomy thresholds by formal measures, AND those formal measures are unreliable at the ~50% level.
+
+**Context:** METR's evaluations are used by OpenAI, Anthropic, and others for safety milestone assessments. Their frameworks are becoming the de facto standard for formal dangerous capability evaluation. The GPT-5 evaluation is publicly available and represents METR's current state-of-the-art methodology.
+
+## Curator Notes
+
+PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
+WHY ARCHIVED: Provides formal numerical calibration of where current frontier models sit relative to governance thresholds — essential context for evaluating B1's "greatest outstanding problem" claim. The finding (2h17m vs 40-hour threshold) partially challenges alarmist interpretations while the 50%+ benchmark instability maintains the governance concern
+EXTRACTION HINT: Separate claims: (1) "Current frontier models evaluate at ~17x below METR's catastrophic risk threshold for autonomous AI R&D" — calibrating B1; (2) "METR's time horizon benchmark shifted 50-57% between v1.0 and v1.1 versions, making governance thresholds derived from it a moving target" — the reliability problem