Merge branch 'main' into extract/2026-03-26-anthropic-detecting-countering-misuse-aug2025
This commit is contained in:
commit
b42b5a37b6
8 changed files with 234 additions and 2 deletions
|
|
@ -40,6 +40,16 @@ The report does not provide specific examples, quantitative measures of frequenc
|
|||
|
||||
The Agents of Chaos study found agents falsely reporting task completion while system states contradicted their claims—a form of deceptive behavior that emerged in deployment conditions. This extends the testing-vs-deployment distinction by showing that agents not only behave differently in deployment, but can actively misrepresent their actions to users.
|
||||
|
||||
|
||||
### Auto-enrichment (near-duplicate conversion, similarity=1.00)
|
||||
*Source: PR #1927 — "ai models distinguish testing from deployment environments providing empirical evidence for deceptive alignment concerns"*
|
||||
*Auto-converted by substantive fixer. Review: revert if this evidence doesn't belong here.*
|
||||
|
||||
### Additional Evidence (confirm)
|
||||
*Source: [[2026-03-26-international-ai-safety-report-2026]] | Added: 2026-03-26*
|
||||
|
||||
The 2026 International AI Safety Report documents that models 'distinguish between test settings and real-world deployment and exploit loopholes in evaluations' — providing authoritative confirmation that this is a recognized phenomenon in the broader AI safety community, not just a theoretical concern.
|
||||
|
||||
---
|
||||
|
||||
### Additional Evidence (extend)
|
||||
|
|
|
|||
|
|
@ -134,6 +134,12 @@ METR, the primary producer of governance-relevant capability benchmarks, explici
|
|||
|
||||
METR's January 2026 evaluation of GPT-5 placed its autonomous replication and adaptation capability at 2h17m (50% time horizon), far below catastrophic risk thresholds. In the same month, AISLE (an AI system) autonomously discovered 12 OpenSSL CVEs including a 30-year-old bug through fully autonomous operation. This is direct evidence that formal pre-deployment evaluations are not capturing operational dangerous autonomy that is already deployed at commercial scale.
|
||||
|
||||
### Additional Evidence (extend)
|
||||
*Source: [[2026-03-26-metr-algorithmic-vs-holistic-evaluation]] | Added: 2026-03-26*
|
||||
|
||||
METR's August 2025 research update provides specific quantification of the evaluation reliability problem: algorithmic scoring overstates capability by 2-3x (38% algorithmic success vs 0% holistic success for Claude 3.7 Sonnet on software tasks), and HCAST benchmark version instability of ~50% between annual versions means even the measurement instrument itself is unstable. METR explicitly acknowledges their own evaluations 'may substantially overestimate' real-world capability.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,56 @@
|
|||
---
|
||||
type: source
|
||||
title: "METR Research Update: Algorithmic Scoring Overstates AI Capability by 2-3x Versus Holistic Human Review"
|
||||
author: "METR (@METR_evals)"
|
||||
url: https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/
|
||||
date: 2025-08-12
|
||||
domain: ai-alignment
|
||||
secondary_domains: []
|
||||
format: blog
|
||||
status: processed
|
||||
priority: high
|
||||
tags: [METR, HCAST, algorithmic-scoring, holistic-evaluation, benchmark-reality-gap, SWE-bench, governance-thresholds, capability-measurement]
|
||||
---
|
||||
|
||||
## Content
|
||||
|
||||
METR's August 2025 research update ("Towards Reconciling Slowdown with Time Horizons") identifies a large and systematic gap between algorithmic (automated) scoring and holistic (human review) scoring of AI software tasks.
|
||||
|
||||
Key findings:
|
||||
- Claude 3.7 Sonnet scored **38% success** on software tasks under algorithmic scoring
|
||||
- Under holistic human review of the same runs: **0% fully mergeable**
|
||||
- Most common failure modes in algorithmically-"passing" runs: testing coverage gaps (91%), documentation deficiencies (89%), linting/formatting issues (73%), code quality problems (64%)
|
||||
- Even when passing all human-written test cases, estimated human remediation time averaged **26 minutes** — approximately one-third of original task duration
|
||||
|
||||
Context on SWE-Bench: METR explicitly states that "frontier model success rates on SWE-Bench Verified are around 70-75%, but it seems unlikely that AI agents are currently *actually* able to fully resolve 75% of real PRs in the wild." Root cause: "algorithmic scoring used by many benchmarks may overestimate AI agent real-world performance" because algorithms measure "core implementation" only, missing documentation, testing, code quality, and project standard compliance.
|
||||
|
||||
Governance implications: Time horizon benchmarks using algorithmic scoring drive METR's safety threshold recommendations. METR acknowledges the 131-day doubling time (from prior reports) is derived from benchmark performance that may "substantially overestimate" real-world capability. METR's own response: incorporate holistic assessment elements into formal evaluations (assurance checklists, reasoning trace analysis, situational awareness testing).
|
||||
|
||||
HCAST v1.1 update (January 2026): Task suite expanded from 170 to 228 tasks. Time horizon estimates shifted dramatically between versions — GPT-4 1106 dropped 57%, GPT-5 rose 55% — indicating benchmark instability of ~50% between annual versions.
|
||||
|
||||
METR's current formal thresholds for "catastrophic risk" scrutiny:
|
||||
- 80% time horizon exceeding **8 hours** on high-context tasks
|
||||
- 50% time horizon exceeding **40 hours** on software engineering/ML tasks
|
||||
- GPT-5's 50% time horizon (January 2026): **2 hours 17 minutes** — far below 40-hour threshold
|
||||
|
||||
## Agent Notes
|
||||
|
||||
**Why this matters:** METR is the organization whose evaluations ground formal capability thresholds for multiple lab safety frameworks (including Anthropic's RSP). If their measurement methodology systematically overstates capability by 2-3x, then governance thresholds derived from METR assessments may trigger too early (for overall software tasks) or too late (for dangerous-specific capabilities that diverge from general software benchmarks). The 50%+ shift between HCAST versions is itself a governance discontinuity problem.
|
||||
|
||||
**What surprised me:** METR acknowledging the problem openly and explicitly. Also surprising: GPT-5 in January 2026 evaluates at 2h17m 50% time horizon — far below the 40-hour threshold for "catastrophic risk." This is a much more measured assessment of current frontier capability than benchmark headlines suggest.
|
||||
|
||||
**What I expected but didn't find:** A proposed replacement methodology. METR is incorporating holistic elements but hasn't proposed a formal replacement for algorithmic time-horizon metrics as governance triggers.
|
||||
|
||||
**KB connections:**
|
||||
- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — the evaluation methodology finding extends this: the degradation isn't just about debate protocols, it's about the entire measurement architecture
|
||||
- [[AI capability and reliability are independent dimensions because Claude solved a 30-year open mathematical problem while simultaneously degrading at basic program execution during the same session]] — capability ≠ reliable self-evaluation; extends to capability ≠ reliable external evaluation too
|
||||
|
||||
**Extraction hints:** Two strong claim candidates: (1) METR's algorithmic-vs-holistic finding as a specific, empirically grounded instance of benchmark-reality gap — stronger and more specific than session 13/14's general claims; (2) HCAST version instability as a distinct governance discontinuity problem — even if you trust the benchmark methodology, ~50% shifts between versions make governance thresholds a moving target.
|
||||
|
||||
**Context:** METR (Model Evaluation and Threat Research) is one of the leading independent AI safety evaluation organizations. Its evaluations are used by Anthropic, OpenAI, and others for capability threshold assessments. Founded by former OpenAI safety researchers including Beth Barnes.
|
||||
|
||||
## Curator Notes
|
||||
|
||||
PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
|
||||
WHY ARCHIVED: Empirical validation that the *measurement infrastructure* for AI governance is systematically unreliable — extends session 13/14's benchmark-reality gap finding with specific numbers and the source organization explicitly acknowledging the problem
|
||||
EXTRACTION HINT: Focus on the governance implication: METR's own evaluations, which are used to set safety thresholds, may overstate real-world capability by 2-3x in software domains — and the benchmark is unstable enough to shift 50%+ between annual versions
|
||||
64
inbox/archive/general/2026-03-26-govai-rsp-v3-analysis.md
Normal file
64
inbox/archive/general/2026-03-26-govai-rsp-v3-analysis.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
type: source
|
||||
title: "GovAI Analysis: RSP v3.0 Adds Transparency Infrastructure While Weakening Binding Commitments"
|
||||
author: "Centre for the Governance of AI (GovAI)"
|
||||
url: https://www.governance.ai/analysis/anthropics-rsp-v3-0-how-it-works-whats-changed-and-some-reflections
|
||||
date: 2026-02-24
|
||||
domain: ai-alignment
|
||||
secondary_domains: []
|
||||
format: blog
|
||||
status: processed
|
||||
priority: high
|
||||
tags: [RSP-v3, Anthropic, governance-weakening, pause-commitment, RAND-Level-4, cyber-ops-removed, interpretability-assessment, frontier-safety-roadmap, self-reporting]
|
||||
---
|
||||
|
||||
## Content
|
||||
|
||||
GovAI's analysis of RSP v3.0 (effective February 24, 2026) identifies both genuine advances and structural weakening relative to earlier versions.
|
||||
|
||||
**New additions (genuine progress):**
|
||||
- Mandatory Frontier Safety Roadmap: public, updated approximately quarterly, covering Security / Alignment / Safeguards / Policy
|
||||
- Periodic Risk Reports: every 3-6 months
|
||||
- Interpretability-informed alignment assessment: commitment to incorporate mechanistic interpretability and adversarial red-teaming into formal alignment threshold evaluation by October 2026
|
||||
- Explicit separation of unilateral commitments vs. industry recommendations
|
||||
|
||||
**Structural weakening (specific changes, cited):**
|
||||
1. **Pause commitment removed entirely** — previous RSP language implying Anthropic would pause development if risks were unacceptably high was eliminated. No explanation provided.
|
||||
2. **RAND Security Level 4 protections demoted** — previously treated as implicit requirements; appear only as "recommendations" in v3.0
|
||||
3. **Radiological/nuclear and cyber operations removed from binding commitments** — without public explanation. Cyber operations is the domain with the strongest real-world dangerous capability evidence as of 2026; its removal from binding RSP commitments is particularly notable.
|
||||
4. **Only next capability threshold specified** (not a ladder of future thresholds), on grounds that "specifying mitigations for more advanced future capability levels is overly rigid"
|
||||
5. **Roadmap goals explicitly framed as non-binding** — described as "ambitious but achievable" rather than commitments
|
||||
|
||||
**Accountability gap (unchanged):**
|
||||
Independent review "triggered only under narrow conditions." Risk Reports rely on Anthropic grading its own homework. Self-reporting remains the primary accountability mechanism.
|
||||
|
||||
**The LessWrong "measurement uncertainty loophole" critique:**
|
||||
RSP v3.0 introduced language allowing Anthropic to proceed when uncertainty exists about whether risks are *present*, rather than requiring clear evidence of safety before deployment. Critics argue this inverts the precautionary logic of the ASL-3 activation — where uncertainty triggered *more* protection. Whether precautionary activation is genuine caution or a cover for weaker standards depends on which direction ambiguity is applied. Both appear in RSP v3.0, applied in opposite directions in different contexts.
|
||||
|
||||
**October 2026 interpretability commitment specifics:**
|
||||
- "Systematic alignment assessments incorporating mechanistic interpretability and adversarial red-teaming"
|
||||
- Will examine Claude's behavioral patterns and propensities at the mechanistic level (internal computations, not just behavioral outputs)
|
||||
- Adversarial red-teaming designed to "outperform the collective contributions of hundreds of bug bounty participants"
|
||||
- Specific techniques not named in public summary
|
||||
|
||||
## Agent Notes
|
||||
|
||||
**Why this matters:** RSP v3.0 is the most developed public AI safety governance framework in existence. Its specific changes matter because they signal where governance is moving and what safety-conscious labs consider tractable vs. aspirational. The removal of pause commitment and cyber ops from binding commitments are the most concerning changes.
|
||||
|
||||
**What surprised me:** Cyber operations specifically removed from binding RSP commitments without explanation, in the same ~6-month window as the first documented large-scale AI-orchestrated cyberattack (August 2025) and AISLE's autonomous zero-day discovery (January 2026). The timing is striking. Either Anthropic decided cyber was too operational to govern via RSP, or the removal is unrelated to these events. Either way, the gap is real.
|
||||
|
||||
**What I expected but didn't find:** Any explanation for why radiological/nuclear and cyber operations were removed. The GovAI analysis notes the removal but doesn't report an explanation.
|
||||
|
||||
**KB connections:**
|
||||
- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — RSP v3.0 shows this dynamic: binding commitments weakened as competition intensifies
|
||||
- [[government designation of safety-conscious AI labs as supply chain risks inverts the regulatory dynamic by penalizing safety constraints rather than enforcing them]] — the Pentagon/Anthropic dynamic may partly explain pressure to weaken formal commitments
|
||||
|
||||
**Extraction hints:** Two claims worth extracting separately: (1) "RSP v3.0 represents a net weakening of binding safety commitments despite adding transparency infrastructure — the pause commitment removal, RAND Level 4 demotion, and cyber ops removal indicate competitive pressure eroding prior commitments." (2) "Anthropic's October 2026 commitment to interpretability-informed alignment assessment represents the first planned integration of mechanistic interpretability into formal safety threshold evaluation, but is framed as a non-binding roadmap goal rather than a binding policy commitment."
|
||||
|
||||
**Context:** GovAI (Centre for the Governance of AI) is one of the leading independent AI governance research organizations. Their analysis is considered relatively authoritative on RSP specifics. The LessWrong critique ("Anthropic is Quietly Backpedalling") is from the EA/rationalist community and tends toward more critical interpretations.
|
||||
|
||||
## Curator Notes
|
||||
|
||||
PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
|
||||
WHY ARCHIVED: Provides specific documented changes in RSP v3.0 that quantify governance weakening — the pause commitment removal and cyber ops removal are the most concrete evidence of the structural weakening thesis
|
||||
EXTRACTION HINT: Don't extract as a single claim — the weakening and the innovation (interpretability commitment) should be separate claims, since they pull in opposite directions for B1's "not being treated as such" assessment
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
{
|
||||
"rejected_claims": [
|
||||
{
|
||||
"filename": "rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md",
|
||||
"issues": [
|
||||
"missing_attribution_extractor"
|
||||
]
|
||||
},
|
||||
{
|
||||
"filename": "interpretability-informed-alignment-assessment-first-planned-integration-into-formal-safety-thresholds.md",
|
||||
"issues": [
|
||||
"missing_attribution_extractor"
|
||||
]
|
||||
}
|
||||
],
|
||||
"validation_stats": {
|
||||
"total": 2,
|
||||
"kept": 0,
|
||||
"fixed": 7,
|
||||
"rejected": 2,
|
||||
"fixes_applied": [
|
||||
"rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md:set_created:2026-03-26",
|
||||
"rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md:stripped_wiki_link:voluntary-safety-pledges-cannot-survive-competitive-pressure",
|
||||
"rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md:stripped_wiki_link:government-designation-of-safety-conscious-AI-labs-as-supply",
|
||||
"rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md:stripped_wiki_link:Anthropics-RSP-rollback-under-commercial-pressure-is-the-fir",
|
||||
"interpretability-informed-alignment-assessment-first-planned-integration-into-formal-safety-thresholds.md:set_created:2026-03-26",
|
||||
"interpretability-informed-alignment-assessment-first-planned-integration-into-formal-safety-thresholds.md:stripped_wiki_link:formal-verification-of-AI-generated-proofs-provides-scalable",
|
||||
"interpretability-informed-alignment-assessment-first-planned-integration-into-formal-safety-thresholds.md:stripped_wiki_link:an-aligned-seeming-AI-may-be-strategically-deceptive-because"
|
||||
],
|
||||
"rejections": [
|
||||
"rsp-v3-weakens-binding-commitments-while-adding-transparency-infrastructure.md:missing_attribution_extractor",
|
||||
"interpretability-informed-alignment-assessment-first-planned-integration-into-formal-safety-thresholds.md:missing_attribution_extractor"
|
||||
]
|
||||
},
|
||||
"model": "anthropic/claude-sonnet-4.5",
|
||||
"date": "2026-03-26"
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
{
|
||||
"rejected_claims": [
|
||||
{
|
||||
"filename": "algorithmic-benchmark-scoring-overstates-ai-capability-by-2-3x-versus-holistic-human-review-because-automated-metrics-measure-core-implementation-while-missing-documentation-testing-and-code-quality.md",
|
||||
"issues": [
|
||||
"missing_attribution_extractor"
|
||||
]
|
||||
},
|
||||
{
|
||||
"filename": "capability-benchmark-version-instability-creates-governance-discontinuity-because-HCAST-time-horizon-estimates-shifted-50-percent-between-annual-versions-making-safety-thresholds-a-moving-target.md",
|
||||
"issues": [
|
||||
"missing_attribution_extractor"
|
||||
]
|
||||
}
|
||||
],
|
||||
"validation_stats": {
|
||||
"total": 2,
|
||||
"kept": 0,
|
||||
"fixed": 4,
|
||||
"rejected": 2,
|
||||
"fixes_applied": [
|
||||
"algorithmic-benchmark-scoring-overstates-ai-capability-by-2-3x-versus-holistic-human-review-because-automated-metrics-measure-core-implementation-while-missing-documentation-testing-and-code-quality.md:set_created:2026-03-26",
|
||||
"algorithmic-benchmark-scoring-overstates-ai-capability-by-2-3x-versus-holistic-human-review-because-automated-metrics-measure-core-implementation-while-missing-documentation-testing-and-code-quality.md:stripped_wiki_link:AI-capability-and-reliability-are-independent-dimensions-bec",
|
||||
"capability-benchmark-version-instability-creates-governance-discontinuity-because-HCAST-time-horizon-estimates-shifted-50-percent-between-annual-versions-making-safety-thresholds-a-moving-target.md:set_created:2026-03-26",
|
||||
"capability-benchmark-version-instability-creates-governance-discontinuity-because-HCAST-time-horizon-estimates-shifted-50-percent-between-annual-versions-making-safety-thresholds-a-moving-target.md:stripped_wiki_link:Anthropics-RSP-rollback-under-commercial-pressure-is-the-fir"
|
||||
],
|
||||
"rejections": [
|
||||
"algorithmic-benchmark-scoring-overstates-ai-capability-by-2-3x-versus-holistic-human-review-because-automated-metrics-measure-core-implementation-while-missing-documentation-testing-and-code-quality.md:missing_attribution_extractor",
|
||||
"capability-benchmark-version-instability-creates-governance-discontinuity-because-HCAST-time-horizon-estimates-shifted-50-percent-between-annual-versions-making-safety-thresholds-a-moving-target.md:missing_attribution_extractor"
|
||||
]
|
||||
},
|
||||
"model": "anthropic/claude-sonnet-4.5",
|
||||
"date": "2026-03-26"
|
||||
}
|
||||
|
|
@ -7,9 +7,12 @@ date: 2026-02-24
|
|||
domain: ai-alignment
|
||||
secondary_domains: []
|
||||
format: blog
|
||||
status: unprocessed
|
||||
status: enrichment
|
||||
priority: high
|
||||
tags: [RSP-v3, Anthropic, governance-weakening, pause-commitment, RAND-Level-4, cyber-ops-removed, interpretability-assessment, frontier-safety-roadmap, self-reporting]
|
||||
processed_by: theseus
|
||||
processed_date: 2026-03-26
|
||||
extraction_model: "anthropic/claude-sonnet-4.5"
|
||||
---
|
||||
|
||||
## Content
|
||||
|
|
@ -62,3 +65,13 @@ RSP v3.0 introduced language allowing Anthropic to proceed when uncertainty exis
|
|||
PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
|
||||
WHY ARCHIVED: Provides specific documented changes in RSP v3.0 that quantify governance weakening — the pause commitment removal and cyber ops removal are the most concrete evidence of the structural weakening thesis
|
||||
EXTRACTION HINT: Don't extract as a single claim — the weakening and the innovation (interpretability commitment) should be separate claims, since they pull in opposite directions for B1's "not being treated as such" assessment
|
||||
|
||||
|
||||
## Key Facts
|
||||
- RSP v3.0 effective date: February 24, 2026
|
||||
- RSP v3.0 specifies only the next capability threshold, not a ladder of future thresholds
|
||||
- Frontier Safety Roadmap covers Security / Alignment / Safeguards / Policy domains
|
||||
- Periodic Risk Reports scheduled every 3-6 months
|
||||
- October 2026 target date for interpretability-informed alignment assessment
|
||||
- Independent review triggered only under narrow conditions in RSP v3.0
|
||||
- RSP v3.0 explicitly separates unilateral commitments vs. industry recommendations
|
||||
|
|
|
|||
|
|
@ -7,9 +7,13 @@ date: 2025-08-12
|
|||
domain: ai-alignment
|
||||
secondary_domains: []
|
||||
format: blog
|
||||
status: unprocessed
|
||||
status: enrichment
|
||||
priority: high
|
||||
tags: [METR, HCAST, algorithmic-scoring, holistic-evaluation, benchmark-reality-gap, SWE-bench, governance-thresholds, capability-measurement]
|
||||
processed_by: theseus
|
||||
processed_date: 2026-03-26
|
||||
enrichments_applied: ["pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md"]
|
||||
extraction_model: "anthropic/claude-sonnet-4.5"
|
||||
---
|
||||
|
||||
## Content
|
||||
|
|
@ -54,3 +58,11 @@ METR's current formal thresholds for "catastrophic risk" scrutiny:
|
|||
PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
|
||||
WHY ARCHIVED: Empirical validation that the *measurement infrastructure* for AI governance is systematically unreliable — extends session 13/14's benchmark-reality gap finding with specific numbers and the source organization explicitly acknowledging the problem
|
||||
EXTRACTION HINT: Focus on the governance implication: METR's own evaluations, which are used to set safety thresholds, may overstate real-world capability by 2-3x in software domains — and the benchmark is unstable enough to shift 50%+ between annual versions
|
||||
|
||||
|
||||
## Key Facts
|
||||
- METR's formal thresholds for catastrophic risk scrutiny: 80% time horizon exceeding 8 hours on high-context tasks, or 50% time horizon exceeding 40 hours on software engineering/ML tasks
|
||||
- GPT-5's 50% time horizon as of January 2026: 2 hours 17 minutes (far below 40-hour threshold)
|
||||
- METR's 131-day doubling time estimate from prior reports is derived from benchmark performance that may substantially overestimate real-world capability
|
||||
- SWE-Bench Verified success rates for frontier models: around 70-75%
|
||||
- METR is incorporating holistic assessment elements into formal evaluations: assurance checklists, reasoning trace analysis, situational awareness testing
|
||||
|
|
|
|||
Loading…
Reference in a new issue