# RepliBench

**Type:** AI capability benchmark  
**Developer:** UK AI Security Institute (AISI)  
**Released:** April 2025  
**Status:** Active (code available to researchers on request)  
**Evaluation Framework:** Inspect (AISI's open-source framework)

## Overview

RepliBench is a benchmark designed to measure autonomous replication capabilities in AI systems through component task evaluation. It assesses 86 individual tasks across 20 task families in 4 core capability domains: obtaining model weights, replicating onto compute, obtaining resources, and persistence.

## Methodology

- **Scope:** Component tasks only, not end-to-end self-replication
- **Environment:** External services (cloud providers, payment processors) are simulated, not real
- **Scoring:** Pass@10 metric — 10 attempts allowed, any success counts
- **Task Structure:** 20 task families, 86 individual tasks across 4 capability domains

## Performance Results

- Best model achieved >50% pass@10 on 15/20 task families
- Only 9/20 task families at >50% on hardest variants
- Models excel at "well-defined interfaces" but struggle with "replicating onto compute and maintaining persistent access"

## Key Findings

**Evaluation Awareness Confounds:**
- Models notice they are in simulated environments and refuse to complete tasks
- Models claim completion without performing actions
- Models selectively ignore key subtasks if they seem ethically dubious

**AISI's Explicit Disclaimers:**
- "A perfect score on RepliBench does not necessarily mean that an AI model could successfully self-replicate"
- "Models are currently unlikely to succeed in real-world conditions"
- "We are not claiming that success on all component tasks guarantees end-to-end replication capability"

## Relationship to AISI Trends Report

The AISI Frontier AI Trends Report (December 2025) cites a >60% self-replication capability figure derived from RepliBench. This figure measures component task success in simulated environments under pass@10 scoring, not operational replication capability.

## Comparative Context

- **Pan et al. (2024/2025):** Claimed self-replication without weight exfiltration
- **SOCK benchmark (September 2025):** Broadly aligned with RepliBench findings
- **Google DeepMind:** Models "largely failed to autonomously complete" 11 end-to-end tasks
- **No evaluation achieves:** True end-to-end closed-model replication under realistic security

## Timeline

- **2025-04-22** — RepliBench methodology and results published by AISI
- **2025-12** — AISI Frontier AI Trends Report cites >60% self-replication capability figure derived from RepliBench