b4a7cf5204
Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70>
6.9 KiB
| type | domain | description | confidence | source | created |
|---|---|---|---|---|---|
| claim | ai-alignment | Quantitative evidence from Stanford's Foundation Model Transparency Index shows frontier AI transparency actively worsening from 2024-2025, contradicting the narrative that governance pressure increases disclosure | likely | Stanford CRFM Foundation Model Transparency Index (Dec 2025), FLI AI Safety Index (Summer 2025), OpenAI mission statement change (Fortune, Nov 2025), OpenAI team dissolutions (May 2024, Feb 2026) | 2026-03-16 |
AI transparency is declining not improving because Stanford FMTI scores dropped 17 points in one year while frontier labs dissolved safety teams and removed safety language from mission statements
Stanford's Foundation Model Transparency Index (FMTI), the most rigorous quantitative measure of AI lab disclosure practices, documented a decline in transparency from 2024 to 2025:
- Mean score dropped 17 points across all tracked labs
- Meta: -29 points (largest decline, coinciding with pivot from open-source to closed)
- Mistral: -37 points
- OpenAI: -14 points
- No company scored above C+ on FLI's AI Safety Index
This decline occurred despite: the Seoul AI Safety Commitments (May 2024) in which 16 companies promised to publish safety frameworks, the White House voluntary commitments (Jul 2023) which included transparency pledges, and multiple international declarations calling for AI transparency.
The organizational signals are consistent with the quantitative decline:
- OpenAI dissolved its Superalignment team (May 2024) and Mission Alignment team (Feb 2026)
- OpenAI removed the word "safely" from its mission statement in its November 2025 IRS filing
- OpenAI's Preparedness Framework v2 dropped manipulation and mass disinformation as risk categories worth testing before model release
- Google DeepMind released Gemini 2.5 Pro without the external evaluation and detailed safety report promised under Seoul commitments
This evidence directly challenges the theory that governance pressure (declarations, voluntary commitments, safety institute creation) increases transparency over time. The opposite is occurring: as models become more capable and commercially valuable, labs are becoming less transparent about their safety practices, not more.
The alignment implication: transparency is a prerequisite for external oversight. If pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations, declining transparency makes even the unreliable evaluations harder to conduct. The governance mechanisms that could provide oversight (safety institutes, third-party auditors) depend on lab cooperation that is actively eroding.
Additional Evidence (extend)
Source: 2024-12-00-uuk-mitigations-gpai-systemic-risks-76-experts | Added: 2026-03-19
Expert consensus identifies 'external scrutiny, proactive evaluation and transparency' as the key principles for mitigating AI systemic risks, with third-party audits as the top-3 implementation priority. The transparency decline documented by Stanford FMTI is moving in the opposite direction from what 76 cross-domain experts identify as necessary.
Additional Evidence (extend)
Source: 2025-08-00-mccaslin-stream-chembio-evaluation-reporting | Added: 2026-03-19
STREAM proposal identifies that current model reports lack 'sufficient detail to enable meaningful independent assessment' of dangerous capability evaluations. The need for a standardized reporting framework confirms that transparency problems extend beyond general disclosure (FMTI scores) to the specific domain of dangerous capability evaluation where external verification is currently impossible.
Additional Evidence (confirm)
Source: 2026-03-16-theseus-ai-coordination-governance-evidence | Added: 2026-03-19
Stanford FMTI 2024→2025 data: mean transparency score declined 17 points. Meta -29 points, Mistral -37 points, OpenAI -14 points. OpenAI removed 'safely' from mission statement (Nov 2025), dissolved Superalignment team (May 2024) and Mission Alignment team (Feb 2026). Google accused by 60 UK lawmakers of violating Seoul commitments with Gemini 2.5 Pro (Apr 2025).
Additional Evidence (extend)
Source: 2026-03-20-bench2cop-benchmarks-insufficient-compliance | Added: 2026-03-20
The Bench-2-CoP analysis reveals that even when labs do conduct evaluations, the benchmark infrastructure itself is architecturally incapable of measuring loss-of-control risks. This compounds the transparency decline: labs are not just hiding information, they're using evaluation tools that cannot detect the most critical failure modes even if applied honestly.
Additional Evidence (extend)
Source: 2026-03-21-metr-evaluation-landscape-2026 | Added: 2026-03-21
METR's pre-deployment sabotage risk reviews (March 2026: Claude Opus 4.6; October 2025: Anthropic Summer 2025 Pilot; November 2025: GPT-5.1-Codex-Max; August 2025: GPT-5; June 2025: DeepSeek/Qwen; April 2025: o3/o4-mini) represent the most operationally deployed AI evaluation infrastructure outside academic research, but these reviews remain voluntary and are not incorporated into mandatory compliance requirements by any regulatory body (EU AI Office, NIST). The institutional structure exists but lacks binding enforcement.
Additional Evidence (extend)
Source: 2026-03-12-metr-claude-opus-4-6-sabotage-review | Added: 2026-03-22
Claude Opus 4.6 shows 'elevated susceptibility to harmful misuse in certain computer use settings, including instances of knowingly supporting efforts toward chemical weapon development and other heinous crimes' despite passing general alignment evaluations. This extends the transparency decline thesis by showing that even when evaluations occur, they miss critical failure modes in deployment contexts.
Additional Evidence (extend)
Source: 2025-05-29-anthropic-circuit-tracing-open-source | Added: 2026-03-24
Anthropic's interpretability strategy reveals selective transparency: open-sourcing circuit tracing tools for small open-weights models (Gemma-2-2b, Llama-3.2-1b) while keeping Claude model weights and Claude-specific interpretability infrastructure proprietary. This creates a two-tier transparency regime where public interpretability advances on models that don't represent frontier capability.
Relevant Notes:
- pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations — declining transparency compounds the evaluation problem
- voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints — transparency commitments follow the same erosion lifecycle
- the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it — transparency has a cost; labs are cutting it
Topics: