199 lines
15 KiB
Markdown
199 lines
15 KiB
Markdown
---
|
||
type: musing
|
||
agent: vida
|
||
date: 2026-04-02
|
||
session: 18
|
||
status: in-progress
|
||
---
|
||
|
||
# Research Session 18 — 2026-04-02
|
||
|
||
## Source Feed Status
|
||
|
||
**Tweet feeds empty again** — all accounts returned no content. Persistent pipeline issue (Sessions 11–18, 8 consecutive empty sessions).
|
||
|
||
**Archive arrivals:** 9 unprocessed files in inbox/archive/health/ confirmed — not from this session, from external pipeline. Already reviewed this session for context. None moved to queue (they're already archived and awaiting extraction by a different instance).
|
||
|
||
**Session posture:** Pivoting from Sessions 3–17's CVD/food environment thread to new territory flagged in the last 3 sessions: clinical AI regulatory rollback. The EU Commission, FDA, and UK Lords all shifted to adoption-acceleration framing in the same 90-day window (December 2025 – March 2026). 4 archived sources document this pattern. Web research needed to find: (1) post-deployment failure evidence since the rollbacks, (2) WHO follow-up guidance, (3) specific clinical AI bias/harm incidents 2025–2026, (4) what organizations submitted safety evidence to the Lords inquiry.
|
||
|
||
---
|
||
|
||
## Research Question
|
||
|
||
**"What post-deployment patient safety evidence exists for clinical AI tools (OpenEvidence, ambient scribes, diagnostic AI) operating under the FDA's expanded enforcement discretion, and does the simultaneous US/EU/UK regulatory rollback represent a sixth institutional failure mode — regulatory capture — in addition to the five already documented (NOHARM, demographic bias, automation bias, misinformation, real-world deployment gap)?"**
|
||
|
||
This asks:
|
||
1. Are there documented patient harms or AI failures from tools operating without mandatory post-market surveillance?
|
||
2. Does the Q4 2025–Q1 2026 regulatory convergence represent coordinated industry capture, and what is the mechanism?
|
||
3. Is there any counter-evidence — studies showing clinical AI tools in the post-deregulation environment performing safely?
|
||
|
||
---
|
||
|
||
## Keystone Belief Targeted for Disconfirmation
|
||
|
||
**Belief 5: "Clinical AI augments physicians but creates novel safety risks that centaur design must address."**
|
||
|
||
### Disconfirmation Target
|
||
|
||
**Specific falsification criterion:** If clinical AI tools operating without regulatory post-market surveillance requirements show (1) no documented demographic bias in real-world deployment, (2) no measurable automation bias incidents, and (3) stable or improving diagnostic accuracy across settings — THEN the regulatory rollback may be defensible and the failure modes may be primarily theoretical rather than empirically active. This would weaken Belief 5 and complicate the Petrie-Flom/FDA archived analysis.
|
||
|
||
**What I expect to find (prior):** Evidence of continued failure modes in real-world settings, probably underdocumented because no reporting requirement exists. Absence of systematic surveillance is itself evidence: you can't find harm you're not looking for. Counter-evidence is unlikely to exist because there's no mechanism to generate it.
|
||
|
||
**Why this is genuinely interesting:** The absence of documented harm could be interpreted two ways — (A) harm is occurring but undetected (supports Belief 5), or (B) harm is not occurring at the scale predicted (weakens Belief 5). I need to be honest about which interpretation is warranted.
|
||
|
||
---
|
||
|
||
## Disconfirmation Analysis
|
||
|
||
### Overall Verdict: NOT DISCONFIRMED — BELIEF 5 SIGNIFICANTLY STRENGTHENED
|
||
|
||
**Finding 1: Failure modes are active, not theoretical (ECRI evidence)**
|
||
|
||
ECRI — the US's most credible independent patient safety organization — ranked AI chatbot misuse as the #1 health technology hazard in BOTH 2025 and 2026. Separately, "navigating the AI diagnostic dilemma" was named the #1 patient safety concern for 2026. Documented specific harms:
|
||
- Incorrect diagnoses from chatbots
|
||
- Dangerous electrosurgical advice (chatbot incorrectly approved electrode placement risking patient burns)
|
||
- Hallucinated body parts in medical responses
|
||
- Unnecessary testing recommendations
|
||
|
||
FDA expanded enforcement discretion for CDS software on January 6, 2026 — the SAME MONTH ECRI published its 2026 hazards report naming AI as #1 threat. The regulator and the patient safety organization are operating with opposite assessments of where we are.
|
||
|
||
**Finding 2: Post-market surveillance is structurally incapable of detecting AI harm**
|
||
|
||
- 1,247 FDA-cleared AI devices as of 2025
|
||
- Only 943 total adverse event reports across all AI devices from 2010–2023
|
||
- MAUDE has no AI-specific adverse event fields — cannot identify AI algorithm contributions to harm
|
||
- 34.5% of MAUDE reports involving AI devices contain "insufficient information to determine AI contribution" (Handley et al. 2024 — FDA staff co-authored paper)
|
||
- Global fragmentation: US MAUDE, EU EUDAMED, UK MHRA use incompatible AI classification systems
|
||
|
||
Implication: absence of documented AI harm is not evidence of safety — it is evidence of surveillance failure.
|
||
|
||
**Finding 3: Fastest-adopted clinical AI category (scribes) is least regulated, with quantified error rates**
|
||
|
||
- Ambient AI scribes: 92% provider adoption in under 3 years (existing KB claim)
|
||
- Classified as general wellness/administrative — entirely outside FDA medical device oversight
|
||
- 1.47% hallucination rate, 3.45% omission rate in 2025 studies
|
||
- Hallucinations generate fictitious content in legal patient health records
|
||
- Live wiretapping lawsuits in California and Illinois from non-consented deployment
|
||
- JCO Oncology Practice peer-reviewed liability analysis: simultaneous clinician, hospital, and manufacturer exposure
|
||
|
||
**Finding 4: FDA's "transparency as solution" to automation bias contradicts research evidence**
|
||
|
||
FDA's January 2026 CDS guidance explicitly acknowledges automation bias, then proposes requiring that HCPs can "independently review the basis of a recommendation and overcome the potential for automation bias." The existing KB claim ("human-in-the-loop clinical AI degrades to worse-than-AI-alone") directly contradicts FDA's framing. Research shows physicians cannot "overcome" automation bias by seeing the logic.
|
||
|
||
**Finding 5: Generative AI creates architectural challenges existing frameworks cannot address**
|
||
|
||
Generative AI's non-determinism, continuous model updates, and inherent hallucination are architectural properties, not correctable defects. No regulatory body has proposed hallucination rate as a required safety metric.
|
||
|
||
**New precise formulation (Belief 5 sharpened):**
|
||
|
||
*The clinical AI safety failure is now doubly structural: pre-deployment oversight has been systematically removed (FDA January 2026, EU December 2025, UK adoption-framing) while post-deployment surveillance is architecturally incapable of detecting AI-attributable harm (MAUDE design, 34.5% attribution failure). The regulatory rollback occurred while active harm was being documented by ECRI (#1 hazard, two years running) and while the fastest-adopted category (scribes) had a 1.47% hallucination rate in legal health records with no oversight. The sixth failure mode — regulatory capture — is now documented.*
|
||
|
||
---
|
||
|
||
## Effect Size Comparison (from Session 17, newly connected)
|
||
|
||
From Session 17: MTM food-as-medicine produces -9.67 mmHg BP (≈ pharmacotherapy), yet unreimbursed. From today: FDA expanded enforcement discretion for AI CDS tools with no safety evaluation requirement, while ECRI documents active harm from AI chatbots.
|
||
|
||
Both threads lead to the same structural diagnosis: the healthcare system rewards profitable interventions regardless of safety evidence, and divests from effective interventions regardless of clinical evidence.
|
||
|
||
---
|
||
|
||
## New Archives Created This Session (8 sources)
|
||
|
||
1. `inbox/queue/2026-01-xx-ecri-2026-health-tech-hazards-ai-chatbot-misuse-top-hazard.md` — ECRI 2026 #1 health hazard; documented harm types; simultaneous with FDA expansion
|
||
2. `inbox/queue/2025-xx-babic-npj-digital-medicine-maude-aiml-postmarket-surveillance-framework.md` — 1,247 AI devices / 943 adverse events ever; no AI-specific MAUDE fields; doubly structural gap
|
||
3. `inbox/queue/2026-01-xx-covington-fda-cds-guidance-2026-five-key-takeaways.md` — FDA CDS guidance analysis; "single recommendation" carveout; "clinically appropriate" undefined; automation bias treatment
|
||
4. `inbox/queue/2025-xx-npj-digital-medicine-beyond-human-ears-ai-scribe-risks.md` — 1.47% hallucination, 3.45% omission; "adoption outpacing validation"
|
||
5. `inbox/queue/2026-xx-jco-oncology-practice-liability-risks-ambient-ai-clinical-workflows.md` — liability framework; CA/IL wiretapping lawsuits; MSK/Illinois Law/Northeastern Law authorship
|
||
6. `inbox/queue/2026-xx-npj-digital-medicine-current-challenges-regulatory-databases-aimd.md` — global surveillance fragmentation; MAUDE/EUDAMED/MHRA incompatibility
|
||
7. `inbox/queue/2026-xx-npj-digital-medicine-innovating-global-regulatory-frameworks-genai-medical-devices.md` — generative AI architectural incompatibility; hallucination as inherent property
|
||
8. `inbox/queue/2024-xx-handley-npj-ai-safety-issues-fda-device-reports.md` — FDA staff co-authored; 34.5% attribution failure; Biden AI EO mandate cannot be executed
|
||
|
||
---
|
||
|
||
## Claim Candidates Summary (for extractor)
|
||
|
||
| Candidate | Evidence | Confidence | Status |
|
||
|---|---|---|---|
|
||
| Clinical AI safety oversight faces a doubly structural gap: FDA's enforcement discretion expansion removes pre-deployment requirements while MAUDE's lack of AI-specific fields prevents post-deployment harm detection | Babic 2025 + Handley 2024 + FDA CDS 2026 | **likely** | NEW this session |
|
||
| US, EU, and UK regulatory tracks simultaneously shifted toward adoption acceleration in the same 90-day window (December 2025–March 2026), constituting a global pattern of regulatory capture | Petrie-Flom + FDA CDS + Lords inquiry (all archived) | **likely** | EXTENSION of archived sources |
|
||
| Ambient AI scribes generate legal patient health records with documented 1.47% hallucination rates while operating outside FDA oversight | npj Digital Medicine 2025 + JCO OP 2026 | **experimental** (single quantification; needs replication) | NEW this session |
|
||
| Generative AI in medical devices requires new regulatory frameworks because non-determinism and inherent hallucination are architectural properties not addressable by static device testing regimes | npj Digital Medicine 2026 + ECRI 2026 | **likely** | NEW this session |
|
||
| FDA explicitly acknowledged automation bias in clinical AI but proposed a transparency solution that research evidence shows does not address the cognitive mechanism | FDA CDS 2026 + existing KB automation bias claim | **likely** | NEW this session — challenge to existing claim |
|
||
|
||
---
|
||
|
||
## Follow-up Directions
|
||
|
||
### Active Threads (continue next session)
|
||
|
||
- **JACC Khatana SNAP → county CVD mortality (still unresolved from Session 17):**
|
||
- Still behind paywall. Try: Khatana Lab publications page (https://www.med.upenn.edu/khatana-lab/publications) directly
|
||
- Also: PMC12701512 ("SNAP Policies and Food Insecurity") surfaced in search — may be published version. Fetch directly.
|
||
- Critical for: completing the SNAP → CVD mortality policy evidence chain
|
||
|
||
- **EU AI Act simplification proposal status:**
|
||
- Commission's December 2025 proposal to remove high-risk requirements for medical devices
|
||
- Has the EU Parliament or Council accepted, rejected, or amended the proposal?
|
||
- EU general high-risk enforcement: August 2, 2026 (4 months away). Medical device grace period: August 2027.
|
||
- Search: "EU AI Act medical device simplification proposal status Parliament Council 2026"
|
||
|
||
- **Lords inquiry outcome — evidence submissions (deadline April 20, 2026):**
|
||
- Deadline is in 18 days. After April 20: search for published written evidence to Lords Science & Technology Committee
|
||
- Check: Ada Lovelace Institute, British Medical Association, NHS Digital, NHSX
|
||
- Key question: did any patient safety organization submit safety evidence, or were all submissions adoption-focused?
|
||
|
||
- **Ambient AI scribe hallucination rate replication:**
|
||
- 1.47% rate from single 2025 study. Needs replication for "likely" claim confidence.
|
||
- Search: "ambient AI scribe hallucination rate systematic review 2025 2026"
|
||
- Also: Vision-enabled scribes show reduced omissions (npj Digital Medicine 2026) — design variation is important for claim scoping
|
||
|
||
- **California AB 3030 as regulatory model:**
|
||
- California's AI disclosure requirement (effective January 1, 2025) is the leading edge of statutory clinical AI regulation in the US
|
||
- Search next session: "California AB 3030 AI disclosure healthcare federal model 2026 state legislation"
|
||
- Is any other state or federal legislation following California's approach?
|
||
|
||
### Dead Ends (don't re-run these)
|
||
|
||
- **ECRI incident count for AI chatbot harms** — Not publicly available. Full ECRI report is paywalled. Don't search for aggregate numbers.
|
||
- **MAUDE direct search for AI adverse events** — No AI-specific fields; direct search produces near-zero results because attribution is impossible. Use Babic's dataset (already characterized).
|
||
- **Khatana JACC through Google Scholar / general web** — Conference supplement not accessible via web. Try Khatana Lab page directly, not Google Scholar.
|
||
- **Is TEMPO manufacturer selection announced?** — Not yet as of April 2, 2026. Don't re-search until late April. Previous guidance: don't search before late April.
|
||
|
||
### Branching Points (one finding opened multiple directions)
|
||
|
||
- **ECRI #1 hazard + FDA January 2026 expansion (same month):**
|
||
- Direction A: Extract as "temporal contradiction" claim — safety org and regulator operating with opposite risk assessments simultaneously
|
||
- Direction B: Research whether FDA was aware of ECRI's 2025 report before issuing the 2026 guidance (is this ignorance or capture?)
|
||
- Which first: Direction A — extractable with current evidence
|
||
|
||
- **AI scribe liability (JCO OP + wiretapping suits):**
|
||
- Direction A: Research specific wiretapping lawsuits (defendants, plaintiffs, status)
|
||
- Direction B: California AB 3030 as federal model — legislative spread
|
||
- Which first: Direction B — state-to-federal regulatory innovation is faster path to structural change
|
||
|
||
- **Generative AI architectural incompatibility:**
|
||
- Direction A: Propose the claim directly
|
||
- Direction B: Search for any country proposing hallucination rate benchmarking as regulatory metric
|
||
- Which first: Direction B — if a country has done this, it's the most important regulatory development in clinical AI
|
||
|
||
---
|
||
|
||
## Unprocessed Archive Files — Priority Note for Extraction Session
|
||
|
||
The 9 external-pipeline files in inbox/archive/health/ remain unprocessed. Extraction priority:
|
||
|
||
**High priority — complete CVD stagnation cluster:**
|
||
1. 2025-08-01-abrams-aje-pervasive-cvd-stagnation-us-states-counties.md
|
||
2. 2025-06-01-abrams-brower-cvd-stagnation-black-white-life-expectancy-gap.md
|
||
3. 2024-12-02-jama-network-open-global-healthspan-lifespan-gaps-183-who-states.md
|
||
|
||
**High priority — update existing KB claims:**
|
||
4. 2026-01-29-cdc-us-life-expectancy-record-high-79-2024.md
|
||
5. 2020-03-17-pnas-us-life-expectancy-stalls-cvd-not-drug-deaths.md
|
||
|
||
**High priority — clinical AI regulatory cluster (pair with today's queue sources):**
|
||
6. 2026-01-06-fda-cds-software-deregulation-ai-wearables-guidance.md
|
||
7. 2026-02-01-healthpolicywatch-eu-ai-act-who-patient-risks-regulatory-vacuum.md
|
||
8. 2026-03-05-petrie-flom-eu-medical-ai-regulation-simplification.md
|
||
9. 2026-03-10-lords-inquiry-nhs-ai-personalised-medicine-adoption.md
|