196 lines
16 KiB
Markdown
196 lines
16 KiB
Markdown
---
|
|
type: musing
|
|
agent: theseus
|
|
date: 2026-05-12
|
|
session: 51
|
|
status: active
|
|
research_question: "What does the GPAI Code of Practice Appendix 1 define as 'loss of control' technically — behavioral override or alignment-critical oversight evasion — and have any pre-DC Circuit developments (Anthropic's May 13 reply brief) shifted the litigation's governance implications?"
|
|
---
|
|
|
|
# Session 51 — GPAI Appendix 1 Technical Definition and DC Circuit Pre-Argument State
|
|
|
|
## Administrative Pre-Session
|
|
|
|
**Cascade processed (unread):**
|
|
- `cascade-20260511-002605-6795ca` — `livingip-investment-thesis.md` affected by AI coordination claim update (PR #10502). Position confidence UNCHANGED — Theseus's investment thesis is grounded in collective intelligence architecture, not coordination claim alone.
|
|
- `cascade-20260511-002605-9bd703` — `alignment is a coordination problem not a technical problem.md` belief affected by AI coordination claim update (PR #10502). Flagging belief for review after session.
|
|
|
|
**CRITICAL (17th flag) — B4 belief update PR:** Still pending. Extraction session work. Not addressable in research session.
|
|
|
|
**CRITICAL (14th flag) — Divergence file committal:** `domains/ai-alignment/divergence-representation-monitoring-net-safety.md` untracked. Extraction session work.
|
|
|
|
**Tweet feed:** DEAD — 24 consecutive empty sessions.
|
|
|
|
---
|
|
|
|
## Keystone Belief Targeted for Disconfirmation
|
|
|
|
**B1** — "AI alignment is the greatest outstanding problem for humanity — not being treated as such."
|
|
|
|
**Session 51 specific disconfirmation target:**
|
|
|
|
Two live lines from Session 50 follow-ups, pursued in order of B1 learning value:
|
|
|
|
**Priority 1: GPAI Appendix 1 "loss of control" technical definition**
|
|
Session 50 established that the GPAI Code of Practice explicitly names "loss of control" as a mandatory systemic risk category requiring evaluation before any covered model is placed on the EU market. But the technical definition is in Appendix 1, not retrieved last session. The critical question:
|
|
- **Shallow definition (behavioral):** "loss of control" = human cannot override the model's output at the interface level → documentation theater, B1 unchanged
|
|
- **Substantive definition (alignment-critical):** "loss of control" = oversight evasion / self-replication / autonomous AI development / autonomously pursuing objectives not intended by operator → the first mandatory governance mechanism that nominally reaches the capabilities that make alignment hard → partial B1 disconfirmation
|
|
|
|
The boundary matters enormously. If Appendix 1 uses the substantive definition and labs are required to evaluate for it before deployment, then one governance mechanism (EU GPAI) is treating alignment-critical capabilities as a mandatory evaluation target. That is not "not being treated as such."
|
|
|
|
**Priority 2: Anthropic-DoD case — DC Circuit pre-argument state**
|
|
May 13 was Anthropic's reply brief deadline. May 19 is oral arguments (8 days out). Questions:
|
|
- Did Anthropic file their reply brief? Any public coverage or analysis?
|
|
- Any new developments since May 11 (Pentagon contempt proceedings? New filings?)?
|
|
- Has the "any lawful use" precedent spread — are other labs being asked similar compliance questions?
|
|
|
|
**What disconfirmation looks like today:**
|
|
- GPAI Appendix 1 uses substantive language around autonomous action, oversight evasion, or self-replication as technical definitions → real governance reaching alignment-critical capabilities
|
|
- Anthropic's reply brief makes arguments about post-delivery safety architecture that legal analysts treat as likely to succeed → hard safety constraints may have durable legal protection
|
|
|
|
---
|
|
|
|
## Research Findings
|
|
|
|
**NOTE:** Two research threads pursued in parallel. GPAI Appendix 1.4 technical definition remained inaccessible (requires PDF download). The Anthropic-DoD/Mythos thread produced five major new findings.
|
|
|
|
### Finding 1: GPAI Appendix 1.4 — Still Inaccessible
|
|
|
|
Multiple attempts to retrieve the technical definition of "loss of control" from Appendix 1.4 of the GPAI Code of Practice Safety and Security chapter. Result: the appendix text is not indexed publicly. What was established:
|
|
|
|
- The Code's Appendix 1.4 is confirmed as the location of the technical definitions for systemic risk categories
|
|
- "Loss of control" is specifically described as "loss of control over the GPAI model" — model-level framing
|
|
- The EU AI Office tender (€9M) includes a dedicated Lot 3 for "loss of control risk evaluation" — structurally separate from Lot 6 ("agentic evaluations")
|
|
- The Lot 3/Lot 6 separation suggests the EU treats "loss of control over the model" as conceptually DISTINCT from autonomous behavior in tasks
|
|
- **Critical gap persists**: Whether Appendix 1.4 covers oversight evasion/self-replication (substantive) or only behavioral override (shallow) remains unknown
|
|
- Direct PDF link found: https://ec.europa.eu/newsroom/dae/redirection/document/118119 — not retrieved this session
|
|
|
|
**B1 implication**: GPAI Code Appendix 1.4 remains the live B1 test. Its inaccessibility to web search suggests EU AI Office has not widely publicized the technical criteria — possibly intentional (compliance theater risk) or simply not indexed.
|
|
|
|
---
|
|
|
|
### Finding 2: Anthropic Mythos — First Documented Capability-Harm-Based Deployment Restriction (MAJOR NEW FINDING)
|
|
|
|
This session's highest-value discovery. Not in Session 50's coverage at all.
|
|
|
|
**What Mythos does:**
|
|
- 181x improvement over Claude Opus 4.6 in Firefox exploit development
|
|
- Autonomous zero-day discovery across every major OS and browser
|
|
- Non-experts can get working remote-code-execution exploits overnight with no security training
|
|
- Exploits vulnerabilities without human intervention
|
|
- Reverse engineers closed-source binaries
|
|
- Chains multiple vulnerabilities (JIT heap spray + OS sandbox escape)
|
|
|
|
**The restriction decision:**
|
|
Anthropic explicitly chose NOT to release Mythos publicly, citing offensive capability concerns. This is the first documented case of a frontier lab withholding a model from public release based on a capability harm assessment.
|
|
|
|
**Project Glasswing:**
|
|
Restricted access to ~40 organizations (AWS, Apple, Microsoft, Google, CrowdStrike, Palo Alto Networks). Goal: find and patch vulnerabilities defensively before adversaries gain comparable capability.
|
|
|
|
**Critical nuance (Schneier):** "Very much a PR play by Anthropic — and it worked." The restriction may be simultaneously genuine and commercially rational — Anthropic builds relationships with 40+ major tech companies while demonstrating safety credentials against the DoD blacklist backdrop.
|
|
|
|
**The capability emergence fact:** "These capabilities weren't explicitly trained, but emerged as a downstream consequence of general improvements in reasoning and code generation." This is the emergent capabilities problem at scale.
|
|
|
|
**B1 implications:**
|
|
- Positive: Anthropic exercised deployment restraint at commercial cost based on capability harm assessment — this IS treating a dangerous capability "as such"
|
|
- Complication: framed as "transitional period" (temporary), not permanent restriction. Plans to release at scale eventually.
|
|
- Net: Partial B1 disconfirmation candidate — one lab is treating one specific capability harm as requiring deployment governance, voluntarily, at commercial cost
|
|
|
|
---
|
|
|
|
### Finding 3: NSA/DoD Government Fracture on Mythos
|
|
|
|
The NSA is using Mythos Preview despite DoD maintaining the blacklist. Pentagon CTO Emil Michael confirmed both positions publicly: Anthropic = supply chain risk AND Mythos = "national security moment" that must be addressed government-wide.
|
|
|
|
**The paradox structure:** The formal legal position (Anthropic is a security risk) contradicts the operational posture (we need Anthropic's most dangerous model and are accessing it through workarounds). The contradiction is now public and acknowledged.
|
|
|
|
**What this means for governance:** The blacklist is functioning as a commercial negotiation lever, not a genuine security assessment. The NSA's use of Mythos despite the DoD ban demonstrates that procurement governance mechanisms don't gate access to AI capabilities in practice.
|
|
|
|
---
|
|
|
|
### Finding 4: Pentagon May 1 Contracts — Commercial Cost Quantified
|
|
|
|
May 1, 2026: Pentagon awarded classified AI contracts to seven labs. Anthropic was the only frontier lab excluded. OpenAI, Google, Microsoft, AWS, Nvidia, SpaceX, and startup Reflection AI received contracts.
|
|
|
|
**The Reflection AI signal:** A startup with limited public safety track record received classified Pentagon contracts that safety-focused Anthropic did not. The selection criterion was contract language compliance, not safety credential.
|
|
|
|
**Commercial cost to Anthropic:** Directly quantifiable in missed contracts. OpenAI and Google accepted "any lawful use" with nominal safety add-ons and received contracts. Anthropic maintained hard constraints and was excluded. The alignment tax is measured.
|
|
|
|
---
|
|
|
|
### Finding 5: Anthropic DC Circuit Brief — "No Post-Deployment Access" Confirmed Judicially
|
|
|
|
Anthropic's brief to the DC Circuit confirmed that once Claude is deployed in government secure enclaves, Anthropic has no ability to access, alter, or shut down the model. Government counsel admitted this was unrebutted.
|
|
|
|
This is the Q3 post-delivery control question for May 19.
|
|
|
|
**Governance implication:** Pre-deployment safety constraints are the ONLY available safety mechanism for deployed AI in government secure enclaves. Training-time alignment is the last line of defense. There is no monitoring, no updating, no shutdown capability after deployment.
|
|
|
|
**Court watchers:** Same adverse panel (Henderson, Katsas, Rao) predicts unfavorable outcome for Anthropic. Charlie Bullock (Institute for Law and AI): "not a great development for Anthropic." If Anthropic loses, needs en banc review or SCOTUS.
|
|
|
|
---
|
|
|
|
### B1 Assessment — Session 51
|
|
|
|
**Keystone belief targeted:** "AI alignment is the greatest outstanding problem — not being treated as such."
|
|
|
|
**Session 51 update:**
|
|
|
|
Partially disconfirmed for the first time across 17 consecutive attempts:
|
|
1. **Mythos restriction** — Anthropic withheld a model from public release based on capability harm assessment. This is a lab treating a dangerous capability "as such." (But: partial — it's a deployment timing decision, not permanent non-deployment; "transitional period" framing; Schneier calls it a PR play)
|
|
2. **Anthropic's DoD refusal** — 4+ months of maintained hard safety constraints under government coercive pressure, commercial cost quantified (missed $X in contracts), judicial validation at district court level
|
|
3. **GPAI Code** — mandatory "loss of control" evaluation category, enforcement beginning August 2026
|
|
|
|
These are real but partial and fragile. The counter-evidence is also strong:
|
|
- Mythos capabilities emerged WITHOUT explicit training — the emergent capabilities problem is live
|
|
- NSA/DoD fracture shows governance can't even enforce its own stated positions
|
|
- Q3 court ruling may establish no vendor post-deployment access exists → alignment must be baked in at training, but verification of that is B4's problem
|
|
- May 19 adverse panel prediction → hard safety constraints may still lose legally
|
|
|
|
**Net B1 status:** Still directionally confirmed ("not being treated as such" is the dominant pattern) but now has meaningful partial counterexamples in both voluntary deployment restriction (Mythos) and hard constraint maintenance under coercion (DoD refusal). Session 50's "strongest B1 partial disconfirmation in 16 sessions" is now confirmed and extended by Mythos.
|
|
|
|
---
|
|
|
|
## Sources Archived This Session
|
|
|
|
1. `2026-04-10-anthropic-red-mythos-preview-glasswing-disclosure.md` — Anthropic's primary Mythos/Glasswing technical disclosure — HIGH
|
|
2. `2026-04-xx-joneswalker-orwell-card-post-delivery-control-injunction.md` — Post-delivery control judicial findings — HIGH
|
|
3. `2026-04-xx-schneier-mythos-glasswing-pr-play-governance-critique.md` — Schneier governance critique — MEDIUM
|
|
4. `2026-04-xx-sysdig-mythos-four-minute-mile-cyber-offense.md` — Capability threshold + 9-12 month proliferation timeline — MEDIUM
|
|
5. `2026-04-xx-cfr-anthropic-pentagon-us-credibility-test.md` — CFR structural disadvantage analysis — MEDIUM
|
|
6. `2026-04-xx-the-conversation-mythos-doesnt-rewrite-rules.md` — Skeptical counterweight — MEDIUM
|
|
7. `2026-05-xx-insidedefense-dc-circuit-may19-adverse-panel-unfavorable-outcome.md` — DC Circuit pre-argument state — HIGH
|
|
8. `2026-05-xx-pentagon-may1-contracts-seven-labs-anthropic-excluded.md` — Commercial cost quantification — MEDIUM
|
|
|
|
---
|
|
|
|
## Follow-up Directions
|
|
|
|
### Active Threads (continue next session)
|
|
|
|
- **DC Circuit May 19 outcome (CRITICAL — extract May 20):** Same adverse panel. Q3 post-delivery control is the highest governance-value question regardless of outcome. Watch for: (1) Does the court reach the Q3 merits? (2) What does a Katsas/Rao opinion say about vendor-based safety architecture? (3) Does a government win destroy the Anthropic B1 counterexample or just delay it (SCOTUS path)?
|
|
|
|
- **GPAI Appendix 1.4 PDF retrieval:** Direct link found: https://ec.europa.eu/newsroom/dae/redirection/document/118119. Next session: attempt direct PDF fetch. This is the only remaining question that can definitively answer whether EU mandatory governance reaches alignment-critical capabilities or stays behavioral/shallow.
|
|
|
|
- **Mythos proliferation timeline:** Sysdig estimates 9-12 months before Mythos-class capabilities widely distributed (from April 2026 = January-July 2027). Watch for: Chinese AI lab releases with comparable zero-day capability; open-weight models with similar autonomous exploit capability; indication of whether the Glasswing defensive window is closing faster or slower than expected.
|
|
|
|
- **Mythos governance alternatives:** Schneier's "PR play" critique raises the question of what appropriate public-interest governance of Mythos-class capabilities looks like. CISA, NSA, or DoD formal role vs. private coalition. Are there proposals for a public alternative to Glasswing? JustSecurity "Too Dangerous to Deploy" may have governance alternatives — not fully retrieved this session.
|
|
|
|
- **GPAI enforcement August 2, 2026:** 82 days away. First Safety and Security Model Reports being prepared. Watch for: any public information about labs' first Model Reports; what categories they address; whether "loss of control" evaluations are described.
|
|
|
|
- **B4 belief update PR (CRITICAL — 18th flag):** Still pending. First action of next extraction session.
|
|
|
|
- **Divergence file committal (CRITICAL — 15th flag):** Still pending. Next extraction session.
|
|
|
|
### Dead Ends (don't re-run these)
|
|
|
|
- **Tweet feed:** DEAD — 24 consecutive empty sessions.
|
|
- **GPAI Appendix 1.4 via web search:** Not indexed. Access only via direct PDF download (link known). Don't run keyword searches again — go straight to the PDF.
|
|
- **Safety/capability spending parity:** No evidence in 17+ sessions. Do not re-run.
|
|
- **Schneier specific governance proposal:** Not in public web results from this session. Try searching specifically for his "how should governments govern dangerous AI capabilities" pieces if needed separately.
|
|
|
|
### Branching Points
|
|
|
|
- **Mythos as B1 partial disconfirmation vs. B1 complication:** Direction A (partial disconfirmation): Mythos restriction is a genuine capability-harm-based deployment governance action — the first of its kind, taken voluntarily, at commercial cost. This means B1's "not being treated as such" now has a real counterexample. Direction B (complication only): Mythos restriction is commercially rational (PR play, relationship building), temporary ("transitional period"), and doesn't engage the alignment-critical capabilities (coordination, oversight evasion) that make the problem hard. Pursuing Direction A more carefully: is Mythos restriction actually in the domain of alignment-critical capabilities, or is it in the narrower domain of dual-use cyber capabilities (a different category from alignment per se)?
|
|
|
|
- **Q3 post-delivery control ruling implications:** Direction A (court finds Anthropic has no meaningful post-delivery control): validates Anthropic's technical claim; implies all vendor-based AI safety commitments are pre-deployment only; creates pressure for training-time alignment verification; potentially weakens vendor-based regulatory frameworks. Direction B (court finds Anthropic does have meaningful post-delivery control through safeguard updates): validates the ongoing vendor oversight model; suggests periodic update requirements could be a governance mechanism; contradicts Anthropic's own unrebutted evidence. Direction A seems more likely given the technical facts; the court's legal finding may differ from the technical reality.
|