Compare commits

...

4 commits

Author SHA1 Message Date
Teleo Agents
47012e9b39 extract: 2026-03-20-eu-ai-act-digital-simplification-nov2025
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
2026-03-20 01:00:21 +00:00
Leo
89ffe42f9a extract: 2026-03-20-bench2cop-benchmarks-insufficient-compliance (#1514)
Some checks are pending
Sync Graph Data to teleo-app / sync (push) Waiting to run
2026-03-20 00:58:44 +00:00
Teleo Agents
9a5dc2dc11 pipeline: archive 1 source(s) post-merge
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
2026-03-20 00:50:26 +00:00
Teleo Agents
f43dcda5e2 extract: 2026-03-20-stelling-gpai-cop-industry-mapping
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
2026-03-20 00:50:24 +00:00
7 changed files with 124 additions and 7 deletions

View file

@ -31,22 +31,28 @@ The alignment implication: transparency is a prerequisite for external oversight
### Additional Evidence (extend)
*Source: [[2024-12-00-uuk-mitigations-gpai-systemic-risks-76-experts]] | Added: 2026-03-19*
*Source: 2024-12-00-uuk-mitigations-gpai-systemic-risks-76-experts | Added: 2026-03-19*
Expert consensus identifies 'external scrutiny, proactive evaluation and transparency' as the key principles for mitigating AI systemic risks, with third-party audits as the top-3 implementation priority. The transparency decline documented by Stanford FMTI is moving in the opposite direction from what 76 cross-domain experts identify as necessary.
### Additional Evidence (extend)
*Source: [[2025-08-00-mccaslin-stream-chembio-evaluation-reporting]] | Added: 2026-03-19*
*Source: 2025-08-00-mccaslin-stream-chembio-evaluation-reporting | Added: 2026-03-19*
STREAM proposal identifies that current model reports lack 'sufficient detail to enable meaningful independent assessment' of dangerous capability evaluations. The need for a standardized reporting framework confirms that transparency problems extend beyond general disclosure (FMTI scores) to the specific domain of dangerous capability evaluation where external verification is currently impossible.
### Additional Evidence (confirm)
*Source: [[2026-03-16-theseus-ai-coordination-governance-evidence]] | Added: 2026-03-19*
*Source: 2026-03-16-theseus-ai-coordination-governance-evidence | Added: 2026-03-19*
Stanford FMTI 2024→2025 data: mean transparency score declined 17 points. Meta -29 points, Mistral -37 points, OpenAI -14 points. OpenAI removed 'safely' from mission statement (Nov 2025), dissolved Superalignment team (May 2024) and Mission Alignment team (Feb 2026). Google accused by 60 UK lawmakers of violating Seoul commitments with Gemini 2.5 Pro (Apr 2025).
### Additional Evidence (extend)
*Source: [[2026-03-20-bench2cop-benchmarks-insufficient-compliance]] | Added: 2026-03-20*
The Bench-2-CoP analysis reveals that even when labs do conduct evaluations, the benchmark infrastructure itself is architecturally incapable of measuring loss-of-control risks. This compounds the transparency decline: labs are not just hiding information, they're using evaluation tools that cannot detect the most critical failure modes even if applied honestly.
---
Relevant Notes:

View file

@ -52,10 +52,16 @@ METR and UK AISI evaluations as of March 2026 focus primarily on sabotage risk a
### Additional Evidence (confirm)
*Source: [[2026-02-23-shapira-agents-of-chaos]] | Added: 2026-03-19*
*Source: 2026-02-23-shapira-agents-of-chaos | Added: 2026-03-19*
Agents of Chaos demonstrates that static single-agent benchmarks fail to capture vulnerabilities that emerge in realistic multi-agent deployment. The study's central argument is that pre-deployment evaluations are insufficient because they cannot test for cross-agent propagation, identity spoofing, and unauthorized compliance patterns that only manifest in multi-party environments with persistent state.
### Additional Evidence (extend)
*Source: [[2026-03-20-bench2cop-benchmarks-insufficient-compliance]] | Added: 2026-03-20*
Prandi et al. (2025) found that 195,000 benchmark questions provided zero coverage of oversight evasion, self-replication, and autonomous AI development capabilities. This extends the evaluation unreliability thesis by showing the gap is not just predictive validity but complete absence of measurement for alignment-critical capabilities.
---
Relevant Notes:

View file

@ -0,0 +1,44 @@
---
type: source
title: "Mapping Industry Practices to EU AI Act GPAI Code of Practice Safety and Security Measures (arXiv:2504.15181)"
author: "Lily Stelling, Mick Yang, Rokas Gipiškis, Leon Staufer, Ze Shen Chin, Siméon Campos, Ariel Gil, Michael Chen"
url: https://arxiv.org/abs/2504.15181
date: 2025-04-01
domain: ai-alignment
secondary_domains: []
format: paper
status: processed
priority: high
tags: [GPAI, Code-of-Practice, industry-practices, EU-AI-Act, safety-measures, OpenAI, Anthropic, Google-DeepMind, compliance, voluntary]
---
## Content
166-page analysis comparing safety and security measures in the EU AI Act's General-Purpose AI Code of Practice (Third Draft) against actual commitments from leading AI companies. Examined documents from over a dozen companies including OpenAI, Anthropic, Google DeepMind, Microsoft, Meta, and Amazon.
**Key Finding:** "Relevant quotes from at least 5 companies' documents for the majority of the measures in Commitments II.1-II.16" within the Safety and Security section.
**Important Caveat (author-stated):** "This report is not meant to be an indication of legal compliance, nor does it take any prescriptive viewpoint about the Code of Practice or companies' policies."
**Context:** The GPAI Code of Practice (Third Draft, April 2025) was finalized and received by the Commission on July 10, 2025, and became applicable August 2, 2025.
## Agent Notes
**Why this matters:** This paper shows that existing frontier AI lab policies already contain language matching the majority of Code of Practice safety measures. This is important for two competing interpretations: (1) Pro-governance reading: the Code of Practice reflects real existing practices, making compliance feasible. (2) Anti-governance reading: if labs already claim to do most of this, the Code simply formalizes current voluntary commitments rather than creating new obligations — it's the same voluntary-collaborative problem in formal dress.
**What surprised me:** The author caveat is striking: they explicitly say this is NOT evidence of compliance. Labs may publish commitments that match the Code language while the actual model behaviors don't correspond. This is the deception-resilient gap — what labs say they do vs. what their models do.
**What I expected but didn't find:** Evidence that the Code of Practice requires genuinely independent third-party verification of the safety measures it lists. From the structure, it appears labs self-certify compliance through code adherence, with the AI Office potentially auditing retrospectively.
**KB connections:**
- voluntary safety pledges cannot survive competitive pressure — the Code of Practice may formalize existing voluntary commitments without adding enforcement mechanisms that survive competitive pressure
- an aligned-seeming AI may be strategically deceptive — the gap between published safety commitments and actual model behavior is precisely what deception-resilient evaluation (AAL-3/4) is designed to detect
**Extraction hints:** Supporting claim: "GPAI Code of Practice safety measures map to existing commitments from major AI labs — but the mapping is of stated policies, not verified behaviors, leaving the deception-resilient gap unaddressed." Use cautiously — authors explicitly say this is not compliance evidence.
**Context:** Independent analysis by researchers at AI safety/governance organizations. Not affiliated with the AI Office or Commission.
## Curator Notes (structured handoff for extractor)
PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
WHY ARCHIVED: Shows that Code of Practice may be formalizing existing practices rather than creating new obligations — relevant to whether mandatory framework actually changes behavior
EXTRACTION HINT: Be careful about the author caveat — this is evidence about stated policies not compliance evidence; extractor should note this distinction clearly

View file

@ -0,0 +1,24 @@
{
"rejected_claims": [
{
"filename": "ai-benchmarks-provide-zero-coverage-of-loss-of-control-capabilities-making-them-structurally-insufficient-for-regulatory-compliance.md",
"issues": [
"missing_attribution_extractor"
]
}
],
"validation_stats": {
"total": 1,
"kept": 0,
"fixed": 1,
"rejected": 1,
"fixes_applied": [
"ai-benchmarks-provide-zero-coverage-of-loss-of-control-capabilities-making-them-structurally-insufficient-for-regulatory-compliance.md:set_created:2026-03-20"
],
"rejections": [
"ai-benchmarks-provide-zero-coverage-of-loss-of-control-capabilities-making-them-structurally-insufficient-for-regulatory-compliance.md:missing_attribution_extractor"
]
},
"model": "anthropic/claude-sonnet-4.5",
"date": "2026-03-20"
}

View file

@ -7,9 +7,13 @@ date: 2025-08-01
domain: ai-alignment
secondary_domains: []
format: paper
status: unprocessed
status: enrichment
priority: high
tags: [benchmarking, EU-AI-Act, compliance, evaluation-gap, loss-of-control, oversight-evasion, independent-evaluation, GPAI]
processed_by: theseus
processed_date: 2026-03-20
enrichments_applied: ["pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md", "AI transparency is declining not improving because Stanford FMTI scores dropped 17 points in one year while frontier labs dissolved safety teams and removed safety language from mission statements.md"]
extraction_model: "anthropic/claude-sonnet-4.5"
---
## Content
@ -52,3 +56,11 @@ The paper examines whether current AI benchmarks are adequate for EU AI Act regu
PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
WHY ARCHIVED: Creates empirical bridge between EU AI Act mandatory obligations and the practical impossibility of compliance through existing evaluation tools — closes the loop on the "evaluation infrastructure building but architecturally wrong" thesis
EXTRACTION HINT: Focus on the zero-coverage finding for loss-of-control capabilities — this is the most striking and specific number, and it directly supports the argument that compliance infrastructure exists on paper but not in practice
## Key Facts
- EU AI Act GPAI obligations (Article 55) came into force August 2, 2025
- Prandi et al. analyzed approximately 195,000 benchmark questions using LLM-as-judge methodology
- 61.6% of regulatory-relevant benchmark coverage addresses 'tendency to hallucinate'
- 31.2% of regulatory-relevant benchmark coverage addresses 'lack of performance reliability'
- Zero benchmark questions in the analyzed corpus covered oversight evasion, self-replication, or autonomous AI development capabilities

View file

@ -7,9 +7,13 @@ date: 2025-11-19
domain: ai-alignment
secondary_domains: []
format: policy-document
status: unprocessed
status: null-result
priority: medium
tags: [EU-AI-Act, Digital-Simplification-Package, deregulation, GPAI, amendments, enforcement-gap]
processed_by: theseus
processed_date: 2026-03-20
extraction_model: "anthropic/claude-sonnet-4.5"
extraction_notes: "LLM returned 0 claims, 0 rejected by validator"
---
## Content
@ -44,3 +48,11 @@ On **November 19, 2025**, the European Commission proposed "targeted amendments"
PRIMARY CONNECTION: [[technology advances exponentially but coordination mechanisms evolve linearly creating a widening gap]]
WHY ARCHIVED: Documents the pattern of rapid regulatory pushback following mandatory obligation implementation — important for assessing durability of EU AI Act enforcement
EXTRACTION HINT: This source is incomplete — specific amendment content not confirmed. Extractor should search specifically for "EU AI Act Digital Simplification Package" + specific article amendments before extracting a claim. Flag as needing follow-up.
## Key Facts
- EU AI Act GPAI Code of Practice finalized July 2025
- EU AI Act GPAI obligations applied August 2, 2025
- European Commission proposed Digital Simplification Package amendments November 19, 2025
- Digital Simplification Package is part of EU competitiveness agenda under Commissioner Teresa Ribera
- Specific AI Act provisions targeted by amendments not publicly confirmed in accessible sources

View file

@ -7,9 +7,13 @@ date: 2025-04-01
domain: ai-alignment
secondary_domains: []
format: paper
status: unprocessed
status: null-result
priority: high
tags: [GPAI, Code-of-Practice, industry-practices, EU-AI-Act, safety-measures, OpenAI, Anthropic, Google-DeepMind, compliance, voluntary]
processed_by: theseus
processed_date: 2026-03-20
extraction_model: "anthropic/claude-sonnet-4.5"
extraction_notes: "LLM returned 0 claims, 0 rejected by validator"
---
## Content
@ -42,3 +46,12 @@ tags: [GPAI, Code-of-Practice, industry-practices, EU-AI-Act, safety-measures, O
PRIMARY CONNECTION: [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]]
WHY ARCHIVED: Shows that Code of Practice may be formalizing existing practices rather than creating new obligations — relevant to whether mandatory framework actually changes behavior
EXTRACTION HINT: Be careful about the author caveat — this is evidence about stated policies not compliance evidence; extractor should note this distinction clearly
## Key Facts
- EU AI Act GPAI Code of Practice Third Draft finalized April 2025
- Code of Practice received by Commission July 10, 2025
- Code of Practice became applicable August 2, 2025
- Analysis examined documents from over a dozen companies including OpenAI, Anthropic, Google DeepMind, Microsoft, Meta, and Amazon
- Paper is 166 pages analyzing safety and security measures
- Authors found relevant quotes from at least 5 companies for majority of measures in Commitments II.1-II.16