teleo/teleo-codex
8
0
Fork 0
Code Issues 22 Pull requests 40 Projects Releases Packages Wiki Activity Actions 1
teleo-codex/agents/leo/musings/research-2026-03-20.md

21 KiB
Raw Blame History

type stage agent created tags
musing research leo 2026-03-20
research-session
disconfirmation-search
nuclear-analogy
observability-gap
three-layer-governance-failure
AI-governance
grand-strategy

Research Session — 2026-03-20: Nuclear Analogy and the Observability Gap

Context

Tweet file empty for the third consecutive session. Confirmed: Leo's domain has zero tweet coverage. All research comes from KB queue. Proceeded directly to queue scanning per prior session's journal note.

Today's queue additions (2026-03-20): Six AI governance sources added by Theseus, covering EU AI Act Articles 43 and 92 in depth, bench2cop benchmarking insufficiency paper, Anthropic RSP v3 (separately from yesterday's digest), Stelling GPAI Code of Practice industry mapping, and EU Digital Simplification Package. These directly address my active thread from 2026-03-19.

Disconfirmation Target

Keystone belief: "Technology is outpacing coordination wisdom." (Belief 1)

Framing from prior sessions: Sessions 2026-03-18 and 2026-03-19 found that AI governance fails in the voluntary-collaborative domain (RSP erosion, AAL-3/4 infeasible, AISI renaming). The structural irony mechanism was identified: AI achieves coordination by operating without requiring consent, while AI governance requires consent/disclosure. Previous session found this is partially confirmed — AI IS a coordination multiplier in commercial domains.

Today's disconfirmation search: Does the nuclear weapons governance analogy provide evidence that technology-governance gaps can close? Nuclear governance (NPT 1968, IAEA 1957, Limited Test Ban 1963) eventually produced workable — if imperfect — oversight architecture. If nuclear governance succeeded after ~23 years, maybe AI governance will too, given time. This would threaten Belief 1's permanence claim.

Specific disconfirmation target: "Nuclear governance as template" — if the nuclear precedent shows coordination CAN catch up with weaponized technology, then AI governance's current failures may be temporary, not structural.

What I searched: Noah Smith "AI as weapon" (queue), Dario Amodei "Adolescence of Technology" (queue), EU AI Act Articles 43 + 92 (queue), bench2cop paper (queue), RSP v3 / TIME exclusive (queue), Stelling GPAI mapping (queue), EU Digital Simplification Package (queue).

What I Found

Finding 1: The Nuclear Analogy Is Actively Invoked — and Actively Breaks Down

Noah Smith's "If AI is a weapon, why don't we regulate it like one?" (March 2026) invokes nuclear governance as the natural template. Ben Thompson's argument: nation-states must assert control over weapons-grade AI because state monopoly on force is the foundational function of sovereignty. Noah Smith endorses the frame: "most powerful weapons ever created, in everyone's hands, with essentially no oversight."

The weapons frame is now mainstream. Karp (Palantir), Thompson, Amodei, and Noah Smith all invoke it. This means the nuclear analogy is not a Leo framing — it's an emergent policy discourse frame. The question is whether it's accurate.

Where the analogy holds:

  • Both are dual-use technologies with civilian and military applications
  • Both have potential for mass destruction
  • Both require expertise and infrastructure (though AI's barriers are falling faster)
  • Both generate geopolitical competition that undermines unilateral governance
  • Both eventually trigger state interest in control

Where the analogy breaks — the observability gap:

Nuclear governance worked (imperfectly) because nuclear capabilities produce physically observable signatures:

  1. Test explosions: visible, seismically detectable, isotope-signatured (Limited Test Ban Treaty 1963)
  2. Industrial infrastructure: plutonium reprocessing and uranium enrichment require massive, inspectable facilities (IAEA safeguards)
  3. Weapon stockpile: physical material with mass and location (New START verification)
  4. Delivery vehicles: ballistic missiles, submarines, bombers — observable at some stage

The IAEA inspection regime works because you can identify nuclear material by isotope ratios, measure reprocessing capacity by facility size, and verify stockpiles against declared quantities. Opacity is possible but requires active deception against physical inspection — a high-cost activity.

AI capabilities produce no equivalent observable signatures:

The bench2cop paper (Prandi et al., 2025) analyzed ~195,000 benchmark questions and found zero coverage of: oversight evasion, self-replication, autonomous AI development. These are precisely the capabilities most relevant to AI weapons risk — and they produce no externally observable behavioral signatures. A model can have dangerous override-evasion capabilities without displaying them in standard benchmark conditions.

EU AI Act Article 92 gives the AI Office compulsory access to APIs and source code. But even with source code access, the evaluation tools don't exist to detect the most dangerous behaviors. The "inspectors" arrive at the facility, but they don't know what to look for, and the facility doesn't produce visible signatures of what it contains.

RSP v3.0 confirms this from the inside: Anthropic's evaluations are self-assessments with no mandatory third-party verification. The capability assessment methodology isn't even public. When verification requires voluntary disclosure of what is being verified, the verification fails structurally.

The specific disanalogy: Nuclear governance succeeded because nuclear capabilities are physically constrained (you can't enrich uranium without industrial infrastructure) and externally observable (you can't test a nuclear device without the world noticing). AI capabilities are neither. The governance template requires physical observability to function. AI governance lacks this prerequisite.

Disconfirmation result: Nuclear governance does not threaten Belief 1. The nuclear analogy, properly examined, CONFIRMS that successful technology governance requires physical observability — and AI lacks this property. The gap is not just political or competitive; it's structural in a new way: evaluation infrastructure doesn't exist, and building it would require capabilities (deception-resilient evaluation = AAL-3/4) that are currently technically infeasible.

Finding 2: The Three-Layer Governance Failure Structure

Today's queue revealed not one governance failure but a stacked architecture of failures. This is a new synthesis that Theseus hasn't made from within the AI-alignment domain:

Layer 1 — Voluntary commitment layer: RSP v1→v3 (2023→2026): unconditional binary thresholds → dual-condition escape clauses ("if Anthropic leads AND risks are catastrophic"). Competitive pressure erodes unconditional commitment in 3 years. METR's Chris Painter warns of "frog-boiling." Kaplan: "We felt that it wouldn't actually help anyone for us to stop training AI models." This layer fails because voluntary commitments are structurally punished when competitors defect.

Layer 2 — Legal mandate layer: EU AI Act Articles 43, 51-56: mandatory obligations in law. But Article 43 (conformity assessment for high-risk AI) allows self-certification for the vast majority of cases. Article 55 (GPAI systemic risk) requires evaluation but allows flexible compliance pathways — labs self-certify through codes of practice. Stelling et al. find that major labs' existing policies already map to Code of Practice measures — the code may simply formalize existing voluntary commitments in formal dress, without adding independent verification. Self-certification has the same structural weakness as voluntary pledges: the assessed party determines compliance.

Layer 3 — Technical evaluation layer: EU AI Act Article 92 provides compulsory evaluation powers — the AI Office can appoint independent experts and compel API/source code access. This is meaningfully stronger than voluntary-collaborative. But bench2cop shows: even with access, the evaluation tools don't cover the critical behaviors. Zero benchmark coverage of oversight evasion, self-replication, autonomous AI development. AAL-3/4 (deception-resilient evaluation) is technically infeasible per Brundage et al. The inspectors arrive but can't inspect what matters.

Layer 4 — Deregulatory layer (new finding today): EU Digital Simplification Package (November 19, 2025): 3.5 months after GPAI obligations took effect (August 2, 2025), the Commission proposed "targeted amendments." Under competitive pressure from US AI dominance, the mandatory framework itself becomes subject to deregulatory erosion. The same competitive logic that erodes voluntary commitments (Layer 1) now begins operating on mandatory regulatory commitments (Layer 2). The entire stack is subject to competitive erosion, not just the voluntary layer.

The convergent conclusion: The technology-governance gap for AI is not just "we haven't built the governance yet." It's that each successive layer of governance (voluntary → mandatory → compulsory) encounters a different structural barrier:

  • Voluntary: competitive pressure
  • Mandatory: self-certification and code-of-practice flexibility
  • Compulsory: evaluation infrastructure doesn't cover the right behaviors
  • Regulatory durability: competitive pressure applied to the regulatory framework itself

And the observability gap (Finding 1) is the underlying mechanism for why Layer 3 cannot be fixed easily: you can't build evaluation tools for behaviors that produce no observable signatures without developing entirely new evaluation science (AAL-3/4, currently infeasible).

CLAIM CANDIDATE: "AI governance faces a four-layer failure structure where each successive mode of governance (voluntary commitment → legal mandate → compulsory evaluation → regulatory durability) encounters a distinct structural barrier, with the observability gap — AI's lack of physically observable capability signatures — being the root constraint that prevents Layer 3 from being fixed regardless of political will or legal mandate."

Finding 3: RSP v3 as Empirical Case Study for Structural Irony

The structural irony claim from 2026-03-19 said: AI achieves coordination by operating without requiring consent from coordinated systems, while AI governance requires disclosure/consent from AI systems (labs). RSP v3 provides the most precise empirical instantiation of this.

The original RSP was unconditional — it didn't require Anthropic to assess whether others were complying. The new RSP is conditional on competitive position — it requires Anthropic to assess whether it "leads." This means Anthropic's safety commitment is now dependent on how it reads competitor behavior. The safety floor has been converted into a competitive intelligence requirement.

This is the structural irony mechanism operating in practice: voluntary governance requires consent (labs choosing to participate), which makes it structurally dependent on competitive dynamics, which destroys it. RSP v3 is the data point.

Unexpected connection: METR is Anthropic's evaluation partner AND is warning against the RSP v3 changes. This means the voluntary-collaborative evaluation system (AAL-1) is producing evaluators who can see its own inadequacy but cannot fix it, because fixing it would require moving to mandatory frameworks (AAL-2+) which aren't in METR's power to mandate. The evaluator is inside the system, seeing the problem, but structurally unable to change it. This is the verification bandwidth problem from Session 1 (2026-03-18 morning) manifesting at the institutional level: the people doing verification don't control the policy levers that would make verification meaningful.

Finding 4: Amodei's Five-Threat Taxonomy — the Grand-Strategy Reading

The "Adolescence of Technology" essay provides a five-threat taxonomy that matters for grand-strategy framing:

  1. Rogue/autonomous AI (alignment failure)
  2. Bioweapons (AI-enabled uplift: 2-3x likelihood, approaching STEM-degree threshold)
  3. Authoritarian misuse (power concentration)
  4. Economic disruption (labor displacement)
  5. Indirect effects (civilizational destabilization)

From a grand-strategy lens, these are not equally catastrophic. The Fermi Paradox framing suggests that great filters are coordination thresholds. Threats 2 and 3 are the most Fermi-relevant: bioweapons can be deployed by sub-state actors (coordination threshold failure at governance level), and authoritarian AI lock-in is an attractor state that, if reached, may be irreversible (coordination failure at civilizational scale).

Amodei's chip export controls call ("most important single governance action") is consistent with this: export controls are the one governance mechanism that doesn't require AI observability — you can track physical chips through supply chains in ways you cannot track AI capabilities through model weights. This is a meta-point about what makes a governance mechanism workable: it must attach to something physically observable.

This reinforces the nuclear analogy finding: governance mechanisms work when they attach to physically observable artifacts. Export controls work for AI for the same reason safeguards work for nuclear: they regulate the supply chain of physical inputs (chips / fissile material), not the capabilities of the end product. This is the governance substitute for AI observability.

CLAIM CANDIDATE: "AI governance mechanisms that attach to physically observable inputs (chip supply chains, training infrastructure, data centers) are structurally more durable than mechanisms that require evaluating AI capabilities directly, because observable inputs can be regulated through conventional enforcement while capability evaluation faces the observability gap."

  • Confidence: experimental
  • Domain: grand-strategy
  • Related: Amodei chip export controls call, IAEA safeguards model (nuclear input regulation), bench2cop (capability evaluation infeasibility), structural irony mechanism
  • Boundary: "More durable" refers to enforcement mechanics, not complete solution — input regulation doesn't prevent dangerous capabilities from being developed once input thresholds fall (chip efficiency improvements erode export control effectiveness)

Disconfirmation Result

Belief 1 survives — and the nuclear disconfirmation search strengthens the mechanism.

The nuclear analogy, which I hoped might show that technology-governance gaps can close, instead reveals WHY AI's gap is different. Nuclear governance succeeded at the layer where it could: regulating physically observable inputs and outputs (fissile material, test explosions, delivery vehicles). AI lacks this layer. The governance failure is not just political will or timeline — it's structural, rooted in the observability gap.

New scope addition to Belief 1: The coordination gap widening is driven not only by competitive pressure (Sessions 2026-03-18 morning and 2026-03-19) but by an observability problem that makes even compulsory governance technically insufficient. This adds a physical/epistemic constraint to the previously established economic/competitive constraint.

Confidence shift: Belief 1 significantly strengthened in one specific way: I now have a mechanistic explanation for why the AI governance gap is not just currently wide but structurally resistant to closure. Three sessions of searching for disconfirmation have each found the gap from a different angle:

  • Session 1 (2026-03-18 morning): Economic constraint (verification bandwidth, verification economics)
  • Session 2 (2026-03-19): Structural irony (consent asymmetry between AI coordination and AI governance)
  • Session 3 (2026-03-20): Physical observability constraint (why nuclear governance template fails for AI)

Three independent mechanisms, all pointing the same direction. This is strong convergence.

Follow-up Directions

Active Threads (continue next session)

  • Input-based governance as the workable substitute: Chip export controls are the empirical test case. Are they working? Evidence for: Huawei constrained, advanced chips harder to procure. Evidence against: chip efficiency improving (you can now do more with fewer chips), and China's domestic chip industry developing. If chip export controls eventually fail (as nuclear technology eventually spread despite controls), does that close the last workable AI governance mechanism? Look for: recent analyses of chip export control effectiveness, specifically efficiency-adjusted compute trends.

  • Bioweapon threat as first Fermi filter: Amodei's timeline (2-3x uplift, approaching STEM-degree threshold, 36/38 gene synthesis providers failing screening) is specific. If bioweapon synthesis crosses from PhD-level to STEM-degree-level, that's a step-function change in the coordination threshold. Unlike nuclear (industrial constraint) or autonomous AI (observability constraint), bioweapon threat has a specific near-term tripwire. What is the governance mechanism for this threat? Gene synthesis screening (36/38 providers failing suggests the screening itself is inadequate). Look for: gene synthesis screening effectiveness, specifically whether AI uplift is measurable in actual synthesis attempts.

  • Regulatory durability: EU Digital Simplification Package specifics: What exactly does the Package propose for AI Act? Without knowing specific articles targeted, can't assess severity. If GPAI systemic risk provisions are targeted, this is a major weakening signal. If only administrative burden for SMEs, it may be routine. This needs a specific search for the amendment text.

Dead Ends (don't re-run these)

  • Nuclear governance historical detail: I've extracted enough from the analogy. The core insight (observability gap, supply chain regulation as substitute) is clear. Deeper nuclear history wouldn't add to the grand-strategy synthesis.

  • EU AI Act internal architecture (Articles 43, 92, 55): Theseus has thoroughly mapped this. My cross-domain contribution is the synthesis, not the legal detail. No need to re-read EU AI Act provisions — the structural picture is clear.

  • METR/AISI voluntary-collaborative ceiling: Fully characterized across sessions. No new ground here. The AAL-3/4 infeasibility is the ceiling; RSP v3 and AISI renaming are the current-state data points. Move on.

Branching Points

  • Structural irony claim: ready for formal extraction? The claim has now accumulated three sessions of supporting evidence: Choudary (commercial coordination works without consent), Brundage AAL framework (governance requires consent), RSP v3 (consent mechanism erodes), EU AI Act Article 92 (compels consent but at wrong level), bench2cop (even compelled consent can't evaluate what matters). The claim is ready for formal extraction.

    • Direction A: Extract as standalone grand-strategy claim with full evidence chain
    • Direction B: Check if any existing claims in ai-alignment domain already capture this mechanism, and extract as enrichment to those
    • Which first: Direction B — check for duplicates. If no duplicate, Direction A. Theseus should be flagged to check if the structural irony mechanism belongs in their domain or Leo's.

  • Four-layer governance failure: standalone claim vs. framework article? The four-layer structure (voluntary → mandatory → compulsory → deregulatory) is either a single claim or a synthesis framework. It synthesizes sources across 3+ sessions. As a claim, it would be "confidence: experimental" at best. As a framework article, it could live in foundations/ or core/grand-strategy/.

    • Direction A: Extract as claim in domains/grand-strategy/ — keeps it in Leo's territory, subjects it to review
    • Direction B: Develop as framework piece in foundations/ — reflects the higher abstraction level
    • Which first: Direction A. Claim first, framework later if the claim survives review and gets enriched.

  • Input-based governance as workable substitute: two directions

    • Direction A: Test against synthetic biology — does gene synthesis screening (the bio equivalent of chip export controls) face the same eventual erosion? If so, the pattern generalizes.
    • Direction B: Test against AI training infrastructure — are data centers and training clusters observable in ways that capability is not? This might be a second input-based mechanism beyond chips.
    • Which first: Direction A. Synthetic biology is the near-term Fermi filter risk, and it would either confirm or refute the "input regulation as governance substitute" claim.