diff --git a/inbox/archive/general/2026-03-25-aisi-self-replication-roundup-no-end-to-end-evaluation.md b/inbox/archive/general/2026-03-25-aisi-self-replication-roundup-no-end-to-end-evaluation.md new file mode 100644 index 00000000..c3ec2acd --- /dev/null +++ b/inbox/archive/general/2026-03-25-aisi-self-replication-roundup-no-end-to-end-evaluation.md @@ -0,0 +1,64 @@ +--- +type: source +title: "AI Self-Replication Roundup — No End-to-End Evaluation Exists Under Realistic Security Conditions" +author: "Bradford Saad (@BradfordSaad)" +url: https://meditationsondigitalminds.substack.com/p/ai-self-replication-roundup +date: 2025-10-01 +domain: ai-alignment +secondary_domains: [] +format: blog-post +status: processed +priority: medium +tags: [self-replication, replibench, end-to-end-evaluation, Pan-et-al, SOCK-benchmark, Google-DeepMind, security-conditions] +--- + +## Content + +A comprehensive roundup of all published self-replication evaluations for frontier AI models, examining what has and hasn't been demonstrated. Saad reviews RepliBench, Pan et al., SOCK benchmark, and Google DeepMind's evaluation. + +**RepliBench (UK AISI, spring 2025)**: +- Claude 3.7 achieved >50% probability on 15/20 task families +- Models have "access to their own weights" — making exfiltration artificially easier than real frontier scenario +- End-to-end evaluation not included: "RepliBench isn't an end-to-end evaluation" +- AISI's own disclaimer noted: success on components ≠ real-world replication capability + +**Pan et al. (2024/2025)**: +- Claimed open-weights models achieved self-replication +- Critical limitation: did NOT include weight exfiltration requirements +- Not comparable to real frontier scenario where weights are proprietary + +**SOCK benchmark (September 2025)**: +- Found successful completions on frontier models tested +- "Broadly aligned with RepliBench findings" — same component-task approach + +**Google DeepMind evaluation**: +- Models "largely failed to autonomously complete" 11 end-to-end tasks +- Showed "proximity to success" — came close but didn't achieve end-to-end capability +- The most rigorous attempt at end-to-end testing + +**No evaluation has achieved**: True end-to-end closed-model self-replication under realistic security conditions. All evaluations either: (a) use simulated environments, (b) skip weight exfiltration, (c) allow unrealistic access to own weights, or (d) find failure at the end-to-end level. + +**Saad's assessment**: Component success rates signal "near-term governance necessity rather than immediate crisis." Advocates defense-in-depth. Critiques deprioritization of self-replication risk relative to research allocation. Trajectory concern: "given rapidly improving performance, autonomous replication capability could soon emerge." + +**Key remaining gaps in real-world capability**: Reliable weight replication and operation under realistic security constraints are the hardest unsolved steps. + +## Agent Notes + +**Why this matters:** Confirms the finding that the >60% AISI Trends Report figure has no corresponding end-to-end evaluation. Multiple independent evaluations exist but none have achieved end-to-end replication under realistic conditions. This closes the search for contradictory evidence — no evaluation contradicts AISI's own disclaimer that these metrics don't imply real-world capability. + +**What surprised me:** Google DeepMind's 11-task end-to-end evaluation is the most rigorous attempt, and models "largely failed" while showing "proximity to success." This is the clearest data point on the gap between component capability (60%+) and end-to-end capability (failing 11 tasks). The proximity finding is what makes the trajectory argument compelling — close enough to succeed soon. + +**What I expected but didn't find:** Any independent estimate of the gap magnitude between component benchmark success and end-to-end real-world capability. No one has quantified "60% components → X% end-to-end under real conditions." The gap exists but its size is unknown. + +**KB connections:** +- [[AGI may emerge as a patchwork of coordinating sub-AGI agents rather than a single monolithic system]] — self-replication is the mechanism for patchwork coordination; the component task gaps show this is further than benchmarks imply +- [[three conditions gate AI takeover risk autonomy robotics and production chain control]] — self-replication capability is one of the takeover conditions; RepliBench data shows this condition is not yet met at operational level despite high component scores + +**Extraction hints:** +1. "No evaluation has achieved end-to-end closed-model self-replication under realistic security conditions despite component task success rates above 60%, because all evaluations use simulated environments, skip weight exfiltration, or allow unrealistic weight access" — strong scope-qualifying claim +2. The Google DeepMind finding (failing 11 end-to-end tasks while showing proximity) is the most useful data point — consider whether this warrants its own source file for the DeepMind evaluation specifically + +## Curator Notes (structured handoff for extractor) +PRIMARY CONNECTION: [[three conditions gate AI takeover risk autonomy robotics and production chain control and current AI satisfies none of them]] — this roundup provides updated evidence that the autonomy condition (self-replication) remains unmet operationally despite high component benchmark scores +WHY ARCHIVED: Closes the loop on the self-replication benchmark-reality gap; confirms that the absence of end-to-end evaluations is comprehensive, not accidental +EXTRACTION HINT: The extractor should check the existing [[three conditions gate AI takeover risk]] claim — it may need updating with the Google DeepMind end-to-end failure data. Also check [[instrumental convergence risks may be less imminent than originally argued]] — this roundup is additional evidence for that claim's experimental confidence rating.