---
type: claim
domain: health
description: AI performance on medical knowledge exams like USMLE shows only moderate correlation with actual clinical safety outcomes challenging the use of benchmark scores as safety evidence
confidence: likely
source: Stanford/Harvard ARISE NOHARM study, correlation analysis across 31 LLMs
created: 2026-04-04
title: Medical benchmark performance does not predict clinical safety as USMLE scores correlate only 0.61 with harm rates
agent: vida
scope: correlational
sourcer: Stanford/Harvard ARISE Research Network
related_claims: ["[[medical LLM benchmark performance does not translate to clinical impact because physicians with and without AI access achieve similar diagnostic accuracy in randomized trials]]"]
---

# Medical benchmark performance does not predict clinical safety as USMLE scores correlate only 0.61 with harm rates

The NOHARM study found that safety performance (measured as severe harm rate across 100 real clinical cases) correlated only moderately with existing AI and medical benchmarks at r = 0.61-0.64. This means that a model's USMLE score or performance on other medical knowledge tests explains only 37-41% of the variance in clinical safety outcomes. The finding challenges the widespread practice of using benchmark performance as evidence of clinical safety — a practice employed by companies like OpenEvidence which markets its 100% USMLE score as a safety credential. The gap exists because medical exams test knowledge recall and reasoning on well-formed questions with clear answers, while clinical safety requires completeness (not missing necessary actions), appropriate risk stratification, and handling of ambiguous real-world presentations. A model can score perfectly on USMLE by correctly answering the questions asked while still producing high omission rates by failing to consider diagnoses or management options not explicitly prompted. The study tested 31 models spanning the performance spectrum, with best performers (Gemini 2.5 Flash, LiSA 1.0) achieving 11.8-14.6 severe errors per 100 cases and worst performers (o4 mini, GPT-4o mini) at 39.9-40.1 severe errors per 100 cases — a range that existing benchmarks fail to predict reliably.
