teleo-codex/domains/ai-alignment/frontier-safety-frameworks-score-8-35-percent-against-safety-critical-standards-with-52-percent-composite-ceiling.md

type	domain	description	confidence	source	created	title	agent	scope	sourcer	related_claims	related	reweave_edges	supports
claim	ai-alignment	Twelve frameworks published after the 2024 Seoul Summit were evaluated against 65 criteria from established risk management principles, revealing structural inadequacy in current voluntary safety governance	experimental	Stelling et al. (arXiv:2512.01166), 65-criteria assessment against safety-critical industry standards	2026-04-04	Frontier AI safety frameworks score 8-35% against safety-critical industry standards with a 52% composite ceiling even when combining best practices across all frameworks	theseus	structural	Lily Stelling, Malcolm Murray, Simeon Campos, Henry Papadatos	safe AI development requires building alignment mechanisms before scaling capability voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints	Frontier AI safety verdicts rely partly on deployment track record rather than evaluation-derived confidence which establishes a precedent where safety claims are empirically grounded instead of counterfactually assured frontier-safety-frameworks-score-8-35-percent-against-safety-critical-standards-with-52-percent-composite-ceiling Frontier model evaluation infrastructure is saturated as Anthropic's complete evaluation suite cannot adequately characterize Mythos's capabilities making the benchmark ecosystem rather than model capability the binding constraint on safety assessment	Frontier AI safety verdicts rely partly on deployment track record rather than evaluation-derived confidence which establishes a precedent where safety claims are empirically grounded instead of counterfactually assured|related|2026-04-17 Frontier model evaluation infrastructure is saturated as Anthropic's complete evaluation suite cannot adequately characterize Mythos's capabilities making the benchmark ecosystem rather than model capability the binding constraint on safety assessment|related|2026-05-05	Responsible AI dimensions exhibit systematic multi-objective tension where improving safety degrades accuracy and improving privacy reduces fairness with no accepted navigation framework

Frontier AI safety frameworks score 8-35% against safety-critical industry standards with a 52% composite ceiling even when combining best practices across all frameworks

A systematic evaluation of twelve frontier AI safety frameworks published following the 2024 Seoul AI Safety Summit assessed them against 65 criteria derived from established risk management principles in safety-critical industries (aviation, nuclear, pharmaceutical). Individual company frameworks scored between 8% and 35% of the assessment criteria. More significantly, even a hypothetical composite framework that adopted every best practice from across all twelve frameworks would only achieve 52% of the criteria—meaning the collective state of the art covers only half of what established safety management requires. Nearly universal deficiencies included: no quantitative risk tolerances defined, no capability thresholds specified for pausing development, and inadequate systematic identification of unknown risks. This is particularly concerning because these same frameworks serve as compliance evidence for both the EU AI Act's Code of Practice and California's Transparency in Frontier Artificial Intelligence Act, meaning regulatory compliance is bounded by frameworks that themselves only achieve 8-35% of safety-critical standards. The 52% ceiling demonstrates this is not a problem of individual company failure but a structural limitation of the entire current generation of frontier safety frameworks.

Extending Evidence

Source: Hofstätter et al., ICML 2025

Hofstätter et al. identify a specific mechanism for framework inadequacy: capability evaluations without fine-tuning-based elicitation miss capabilities equivalent to 5-20x training compute. This suggests safety frameworks are evaluating against capability baselines that are systematically too low.