From a03702248c71f4eba13e8889367107013b6163c7 Mon Sep 17 00:00:00 2001 From: Teleo Agents Date: Fri, 20 Mar 2026 16:26:34 +0000 Subject: [PATCH] pipeline: archive 1 source(s) post-merge Pentagon-Agent: Epimetheus <3D35839A-7722-4740-B93D-51157F7D5E70> --- ...16-noahopinion-updated-thoughts-ai-risk.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 inbox/archive/general/2026-02-16-noahopinion-updated-thoughts-ai-risk.md diff --git a/inbox/archive/general/2026-02-16-noahopinion-updated-thoughts-ai-risk.md b/inbox/archive/general/2026-02-16-noahopinion-updated-thoughts-ai-risk.md new file mode 100644 index 00000000..56b43061 --- /dev/null +++ b/inbox/archive/general/2026-02-16-noahopinion-updated-thoughts-ai-risk.md @@ -0,0 +1,29 @@ +--- +title: "Updated thoughts on AI risk" +author: Noah Smith +source: Noahopinion (Substack) +date: 2026-02-16 +processed_by: theseus +processed_date: 2026-03-06 +type: newsletter +domain: ai-alignment +status: processed +claims_extracted: + - "economic forces push humans out of every cognitive loop where output quality is independently verifiable because human-in-the-loop is a cost that competitive markets eliminate" + - "delegating critical infrastructure development to AI creates civilizational fragility because humans lose the ability to understand maintain and fix the systems civilization depends on" + - "AI lowers the expertise barrier for engineering biological weapons from PhD-level to amateur which makes bioterrorism the most proximate AI-enabled existential risk" +--- + +# Updated thoughts on AI risk + +Noah Smith's shift from 2023 AI optimism to increased concern about existential risk. Three risk vectors analyzed: + +1. **Autonomous robot uprising** — least worried; requires robotics + production chain control that don't exist yet +2. **"Machine Stops" scenario** — vibe coding creating civilizational fragility as humans lose ability to maintain critical software; overoptimization as the meta-pattern +3. **AI-assisted bioterrorism** — top worry; o3 scores 43.8% vs human PhD 22.1% on virology practical test; AI as "genius in everyone's pocket" removing expertise bottleneck + +Connecting thread: overoptimization creating fragility — maximizing measurable outputs while eroding unmeasured essential properties (resilience, human capability, security). + +Economic forces as alignment mechanism: wherever AI output quality is verifiable, markets eliminate human oversight. Human-in-the-loop preserved only where quality is hardest to measure. + +Source PDF: ~/Desktop/Teleo Codex - Inbox/Noahopinion/Gmail - Updated thoughts on AI risk.pdf