5.2 KiB
| type | title | author | url | date | domain | secondary_domains | format | status | priority | tags | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| source | Automation Bias in LLM-Assisted Diagnostic Reasoning Among AI-Trained Physicians (RCT, medRxiv August 2025) | Multi-institution research team (Pakistan Medical and Dental Council physician cohort) | https://www.medrxiv.org/content/10.1101/2025.08.23.25334280v1 | 2025-08-26 | health |
|
research paper | unprocessed | high |
|
Content
Published medRxiv August 26, 2025. Registered as NCT06963957 ("Automation Bias in Physician-LLM Diagnostic Reasoning").
Study design:
- Single-blind randomized clinical trial
- Timeframe: June 20 to August 15, 2025
- Participants: Physicians registered with the Pakistan Medical and Dental Council (MBBS degrees), participating in-person or via remote video
- All participants completed 20-hour AI-literacy training covering LLM capabilities, prompt engineering, and critical evaluation of AI output
- Randomized 1:1: 6 clinical vignettes, 75-minute session
- Control arm: Received correct ChatGPT-4o recommendations
- Treatment arm: Received recommendations with deliberate errors in 3 of 6 vignettes
Key results:
- Erroneous LLM recommendations significantly degraded physicians' diagnostic accuracy in the treatment arm
- This effect occurred even among AI-trained physicians (20 hours of AI-literacy training)
- "Voluntary deference to flawed AI output highlights critical patient safety risk"
- "Necessitating robust safeguards to ensure human oversight before widespread clinical deployment"
Related work: JAMA Network Open "LLM Influence on Diagnostic Reasoning" randomized clinical trial (June 2025, PMID: 2825395). ClinicalTrials.gov NCT07328815: "Mitigating Automation Bias in Physician-LLM Diagnostic Reasoning Using Behavioral Nudges" — a follow-on study specifically testing behavioral interventions to reduce automation bias.
Meta-analysis on LLM effect on diagnostic accuracy (medRxiv December 2025) synthesizing these trials.
Agent Notes
Why this matters: The centaur model — AI for pattern recognition, physicians for judgment — is Belief 5's proposed solution to clinical AI safety risks. This RCT directly challenges the centaur assumption: if 20 hours of AI-literacy training is insufficient to protect physicians from automation bias when AI gives DELIBERATELY wrong answers, then the "physician oversight catches AI errors" safety mechanism is much weaker than assumed. The physicians in this study were trained to critically evaluate AI output and still failed.
What surprised me: The training duration (20 hours) is substantial — most "AI literacy" programs are far shorter. If 20 hours doesn't prevent automation bias against deliberately erroneous AI, shorter or no training almost certainly doesn't either. Also noteworthy: the emergence of NCT07328815 (follow-on trial testing "behavioral nudges" to mitigate automation bias) suggests the field recognizes the problem and is actively searching for solutions — which itself confirms the problem's existence.
What I expected but didn't find: I expected to see some granularity on WHICH types of clinical errors triggered the most automation bias. The summary doesn't specify — this is a gap in the current KB for understanding when automation bias is highest-risk.
KB connections:
- Directly challenges the "centaur model" safety assumption in Belief 5
- Connects to Session 19 finding (Catalini verification bandwidth): verification bandwidth is even more constrained if automation bias reduces the quality of physician review
- Cross-domain: connects to Theseus's alignment work on human oversight robustness — this is a domain-specific instance of the general problem of humans failing to catch AI errors at scale
Extraction hints: Primary claim: AI-literacy training is insufficient to prevent automation bias in physician-LLM diagnostic settings (RCT evidence). Secondary: the existence of NCT07328815 ("Behavioral Nudges to Mitigate Automation Bias") as evidence that the field has recognized the problem and is searching for solutions.
Context: Published during a period of rapid clinical AI deployment. The Pakistan physician cohort may limit generalizability, but the automation bias effect is directionally consistent with US and European literature. The NCT07328815 follow-on study suggests US-based researchers are testing interventions — that trial results will be high KB value when available.
Curator Notes (structured handoff for extractor)
PRIMARY CONNECTION: "clinical AI augments physicians but creates novel safety risks requiring centaur design" (Belief 5's centaur assumption) WHY ARCHIVED: First RCT showing that even AI-trained physicians fail to catch erroneous AI recommendations — the centaur model's "physician catches errors" safety assumption is empirically weaker than stated EXTRACTION HINT: Extract the automation-bias-despite-AI-training finding as a challenge to the centaur design assumption. Note the follow-on NCT07328815 trial as evidence the field recognizes the problem requires specific intervention.