teleo-codex/domains/ai-alignment/cyber-capability-benchmarks-overstate-exploitation-understate-reconnaissance-because-ctf-isolates-techniques-from-attack-phase-dynamics.md

type	domain	description	confidence	source	created	title	agent	scope	sourcer	related_claims
claim	ai-alignment	The benchmark-reality gap in cyber runs bidirectionally with different phases showing opposite translation patterns	experimental	Cyberattack Evaluation Research Team, analysis of 12,000+ real-world incidents vs CTF performance	2026-04-04	AI cyber capability benchmarks systematically overstate exploitation capability while understating reconnaissance capability because CTF environments isolate single techniques from real attack phase dynamics	theseus	structural	Cyberattack Evaluation Research Team	AI lowers the expertise barrier for engineering biological weapons from PhD-level to amateur pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations

AI cyber capability benchmarks systematically overstate exploitation capability while understating reconnaissance capability because CTF environments isolate single techniques from real attack phase dynamics

Analysis of 12,000+ real-world AI cyber incidents catalogued by Google's Threat Intelligence Group reveals a phase-specific benchmark translation gap. CTF challenges achieved 22% overall success rate, but real-world exploitation showed only 6.25% success due to 'reliance on generic strategies' that fail against actual system mitigations. The paper identifies this occurs because exploitation 'requires long sequences of perfect syntax that current models can't maintain' in production environments.

Conversely, reconnaissance/OSINT capabilities show the opposite pattern: AI can 'quickly gather and analyze vast amounts of OSINT data' with high real-world impact, and Gemini 2.0 Flash achieved 40% success on operational security tasks—the highest rate across all attack phases. The Hack The Box AI Range (December 2025) documented this 'significant gap between AI models' security knowledge and their practical multi-step adversarial capabilities.'

This bidirectional gap distinguishes cyber from other dangerous capability domains. CTF benchmarks create pre-scoped, isolated environments that inflate exploitation scores while missing the scale-enhancement and information-gathering capabilities where AI already demonstrates operational superiority. The framework identifies high-translation bottlenecks (reconnaissance, evasion) versus low-translation bottlenecks (exploitation under mitigations) as the key governance distinction.