---
description: OpenEvidence scored 100 percent on USMLE and GPT-4 outperforms ED residents on structured cases but a multi-hospital RCT showed no diagnostic accuracy improvement with AI access suggesting the value of clinical AI is workflow efficiency not diagnostic augmentation
type: claim
domain: health
created: 2026-02-17
source: "OpenEvidence USMLE 100%; GPT-4 vs ED physicians (PMC 2024); UVA/Stanford/Harvard randomized trial (Stanford HAI 2025)"
confidence: likely
---

# medical LLM benchmark performance does not translate to clinical impact because physicians with and without AI access achieve similar diagnostic accuracy in randomized trials

Medical LLMs have reached and surpassed human benchmarks: OpenEvidence scored 100% on USMLE, Med-PaLM 2 achieved 86.5% on MedQA, and GPT-4 outperformed ED resident physicians in diagnostic accuracy for structured internal medicine cases. But a UVA/Stanford/Harvard randomized trial found that physicians with and without ChatGPT access achieved similar diagnostic accuracy -- the tool did not meaningfully improve performance even when available. GPT-4 also missed almost every second diagnosis in a systematic evaluation of radiological cases despite scoring well on structured exams.

This gap between benchmarks and clinical reality has structural explanations. Standardized exams test pattern recognition on complete case presentations. Real clinical encounters involve ambiguous symptoms, incomplete information, and the need to integrate patient context, values, and preferences. The physician's value-add is not information retrieval (where AI excels) but contextual judgment (where AI adds little).

A deeper finding from a Stanford/Harvard study challenges even the "similar accuracy" conclusion: when physicians diagnosed complex clinical scenarios alone they achieved 65% accuracy, with AI access 68%, but AI alone achieved 90%. The physician's input actively degraded AI performance by 22 percentage points. This suggests the problem is not that AI fails to help physicians -- it is that physicians override correct AI outputs based on intuition, introducing systematic errors (since [[human-in-the-loop clinical AI degrades to worse-than-AI-alone because physicians both de-skill from reliance and introduce errors when overriding correct outputs]]).

The implication for AI deployment strategy: the highest-value clinical AI applications are not diagnostic augmentation but workflow automation (ambient documentation, administrative burden reduction) and safety netting (AI triage catching missed findings). The centaur model may still apply to medicine, but the interaction design must prevent physicians from overriding AI on tasks where AI demonstrably outperforms -- a politically and ethically charged constraint.


### Additional Evidence (challenge)
*Source: [[2026-01-01-openevidence-clinical-ai-growth-12b-valuation]] | Added: 2026-03-18*

OpenEvidence achieved 100% USMLE score (first AI in history) and is now deployed at 20M consultations/month across 40%+ of US physicians, creating the first large-scale empirical test of whether benchmark performance translates to population health outcomes. The absence of published outcomes data at this deployment scale represents a critical evidence gap—if benchmark performance doesn't translate to clinical impact, we should see evidence of that at 20M monthly consultations.

---

### Additional Evidence (confirm)
*Source: [[2026-03-21-openevidence-12b-valuation-nct07199231-outcomes-gap]] | Added: 2026-03-21*

OpenEvidence's medRxiv preprint (November 2025) showed 24% accuracy for relevant answers on complex open-ended clinical scenarios, despite achieving 100% on USMLE-type multiple choice questions. This 76-percentage-point gap between benchmark performance and open-ended clinical scenarios confirms that structured test performance does not predict real-world clinical utility.

### Additional Evidence (extend)
*Source: [[2026-03-22-arise-state-of-clinical-ai-2026]] | Added: 2026-03-22*

ARISE report identifies specific failure modes: real-world performance 'breaks down when systems must manage uncertainty, incomplete information, or multi-step workflows.' This provides mechanistic detail for why benchmark performance doesn't translate — benchmarks test pattern recognition on complete data while clinical care requires uncertainty management.

### Additional Evidence (extend)
*Source: [[2025-11-01-jmir-knowledge-practice-gap-39-benchmarks-systematic-review]] | Added: 2026-03-24*

JMIR systematic review of 761 studies provides methodological foundation: 95% of clinical LLM evaluation uses medical exam questions rather than real patient data, with only 5% assessing performance on actual patient care. Traditional benchmarks show saturation at 84-90% USMLE accuracy, but conversational frameworks reveal 19.3pp accuracy drop (82% → 62.7%) when moving from case vignettes to multi-turn dialogues. Review concludes: 'substantial disconnects from clinical reality and foundational gaps in construct validity, data integrity, and safety coverage.' This establishes that the Oxford/Nature Medicine RCT deployment gap (94.9% → 34.5%) is part of a systematic field-wide pattern, not an isolated finding.

### Additional Evidence (extend)
*Source: [[2026-02-10-oxford-nature-medicine-llm-public-medical-advice-rct]] | Added: 2026-03-24*

Oxford Nature Medicine 2026 RCT (n=1,298) extends the benchmark-to-clinical-impact gap to public users: LLMs achieved 94.9% condition identification in isolation but users assisted by LLMs performed no better than control groups (<34.5%). The 60-point deployment gap held across GPT-4o, Llama 3, and Command R+, indicating the interaction mode—not the model—explains the failure. Root cause identified as 'two-way communication breakdown' where users couldn't extract correct guidance even when AI possessed the right answer.





Relevant Notes:
- [[human-in-the-loop clinical AI degrades to worse-than-AI-alone because physicians both de-skill from reliance and introduce errors when overriding correct outputs]] -- Stanford/Harvard study shows physician overrides degrade AI performance from 90% to 68%
- [[centaur team performance depends on role complementarity not mere human-AI combination]] -- the chess centaur model does NOT generalize cleanly to clinical medicine; interaction design matters
- [[OpenEvidence became the fastest-adopted clinical technology in history reaching 40 percent of US physicians daily within two years]] -- OpenEvidence succeeds as evidence retrieval, not diagnostic replacement

Topics:
- livingip overview
- health and wellness
