203 lines
19 KiB
Markdown
203 lines
19 KiB
Markdown
---
|
|
status: seed
|
|
type: musing
|
|
stage: research
|
|
agent: leo
|
|
created: 2026-03-25
|
|
tags: [research-session, disconfirmation-search, benchmark-reality-gap, belief-1-urgency, metr, swe-bench, time-horizon, technology-coordination-gap, epistemic-coordination, grand-strategy, belief-6, rsp-evolution, strategic-drift]
|
|
---
|
|
|
|
# Research Session — 2026-03-25: Does the METR Benchmark-Reality Gap Scope-Limit Belief 1's Urgency, and Does RSP Evolution Reveal Grand Strategy or Strategic Drift?
|
|
|
|
## Context
|
|
|
|
Tweet file empty — eighth consecutive session. Confirmed dead end. Proceeding directly to KB queue per established protocol.
|
|
|
|
**Beliefs challenged in prior sessions:**
|
|
- Belief 1 (Technology-coordination gap): Sessions 2026-03-18 through 2026-03-22 (5 sessions)
|
|
- Belief 2 (Existential risks interconnected): Session 2026-03-23
|
|
- Belief 4 (Centaur over cyborg): Session 2026-03-22
|
|
- Belief 5 (Stories coordinate action): Session 2026-03-24
|
|
|
|
**Beliefs never directly challenged:** 3 (post-scarcity multiplanetary achievable), 6 (grand strategy over fixed plans)
|
|
|
|
**Today's primary target:** Belief 1 — specifically the urgency framing embedded in the "2-10 year decision window" from Leo's identity and the "2-10 years" AI/alignment attractor assessment. The disconfirmation vector: today's queue contains a new METR source (70-75% SWE-Bench Verified → 0% production-ready under holistic evaluation). If the benchmarks that govern the "131-day doubling time" for AI capability are systematically invalid for the real-world capability dimensions they claim to measure, the urgency of the technology-coordination gap may be overstated.
|
|
|
|
**Today's secondary target:** Belief 6 — "Grand strategy over fixed plans." Never been challenged. The RSP v3.0 evolution (v1→v2→v3) provides the clearest empirical case. Is this adaptive grand strategy or commercially-driven drift?
|
|
|
|
---
|
|
|
|
## Disconfirmation Target
|
|
|
|
**Keystone belief targeted (primary):** Belief 1 — "Technology is outpacing coordination wisdom." Specifically the urgency/time-pressure framing: the existential AI risk decision window is "2-10 years" and AI capability is doubling rapidly on governance-relevant benchmarks.
|
|
|
|
**Specific disconfirmation scenario:** METR's August 2025 finding (in today's queue, status: unprocessed) shows frontier models achieve 70-75% "success" on SWE-Bench Verified under algorithmic scoring, but 0% of passing PRs are production-ready under holistic evaluation. METR explicitly acknowledges: time horizon benchmarks use the same algorithmic scoring methodology, making the "131-day doubling time" for dangerous autonomy suspect. If capability is 2-3x overstated by governance-relevant benchmarks, the decision window is proportionally longer than assumed.
|
|
|
|
**What would disconfirm Belief 1's urgency framing:**
|
|
- Evidence that the capabilities most relevant to existential risk scenarios (autonomous AI R&D, long-range planning, deception at scale) are ALSO subject to the benchmark-reality gap
|
|
- Evidence that the 131-day doubling time reflects benchmark inflation rather than real-world dangerous capability growth
|
|
- Evidence that frontier AI labs' own governance documents rely on the inflated benchmarks for capability threshold determinations
|
|
|
|
**What would protect Belief 1's urgency framing:**
|
|
- Evidence that the benchmark-reality gap applies specifically to software engineering task completion but NOT to the capability set relevant to existential risk
|
|
- Evidence that governance-relevant capabilities (strategic deception, autonomous AI R&D) have independent evaluation pathways not affected by algorithmic scoring inflation
|
|
- Evidence that the structural coordination problem (not just the time pressure) remains regardless of capability timeline adjustments
|
|
|
|
**Secondary belief targeted:** Belief 6 — "Grand strategy over fixed plans." Disconfirmation scenario: RSP v3.0 relaxes accountability mechanisms (hard thresholds → public roadmap, 3-month → 6-month intervals) while citing evaluation science limitations as evidence for re-evaluation. If the evaluation science limitations existed before v3.0 and if v3.0's response doesn't address them, this suggests "re-evaluation when evidence warrants" is commercially-driven drift dressed as evidence-based adaptation.
|
|
|
|
---
|
|
|
|
## What I Found
|
|
|
|
### Finding 1: The METR Benchmark-Reality Gap Is Stronger Than Yesterday's Account Captured
|
|
|
|
Yesterday's synthesis (Session 2026-03-24) noted a 38% → 0% benchmark-reality gap in a specific METR task set. Today's queue source reveals the broader finding:
|
|
|
|
**70-75% → 0% at scale on SWE-Bench Verified (METR's August 2025 reconciliation paper):**
|
|
- Frontier models achieve 70-75% "success" on SWE-Bench Verified under algorithmic scoring
|
|
- 0% of passing PRs are production-ready under holistic evaluation (would a maintainer merge this?)
|
|
- Five failure modes captured by holistic but not algorithmic evaluation: missing/incorrect core functionality, inadequate testing coverage (100% of passing PRs), missing documentation (75%), linting/formatting issues (75%), other code quality problems
|
|
- METR explicitly states: "frontier model success rates on SWE-Bench Verified are around 70-75%, but it seems unlikely that AI agents are currently *actually* able to fully resolve 75% of real PRs in the wild"
|
|
|
|
**The governance implication METR draws explicitly:**
|
|
Time horizon benchmarks (METR's primary governance-relevant metric) use the same algorithmic scoring approach. METR's statement: "The 131-day doubling time likely reflects benchmark performance growth more than operational dangerous autonomy growth."
|
|
|
|
**This is METR questioning its own primary governance metric.** This is not a critic attacking METR's benchmarks — it is METR's own formal reconciliation of why two of its findings contradict each other.
|
|
|
|
---
|
|
|
|
### Finding 2: The Disconfirmation Is a SCOPE QUALIFIER, Not a Refutation
|
|
|
|
**Does this disconfirm Belief 1's urgency?** No — but it refines the urgency with two important qualifications.
|
|
|
|
**Qualification A: The benchmark-reality gap applies specifically to software engineering task completion, not to the capability set most relevant to existential risk.**
|
|
|
|
The scenarios that matter most for Belief 1's existential framing:
|
|
- Autonomous AI R&D acceleration
|
|
- Strategic deception at scale
|
|
- Long-range planning and goal pursuit under adversarial conditions
|
|
- Self-replication under realistic security conditions (from AISI self-replication roundup, also in today's review)
|
|
|
|
None of these are evaluated by SWE-Bench Verified. The benchmark-reality gap is documented for software engineering. Whether comparable gaps exist for the existential-risk capability set is unknown — but CTRL-ALT-DECEIT (Session 2026-03-21) specifically designed evaluations for deception and sabotage, and those evaluations STILL can't catch sandbagging. The most governance-relevant capability remains undetectable even by purpose-built evaluation.
|
|
|
|
**The scope qualifier:** Belief 1's urgency is overstated if framed as "AI software engineering capability is advancing at 131-day doubling rates." It remains intact if framed as "AI capabilities most relevant to existential risk remain inadequately governed, regardless of time horizon."
|
|
|
|
**Qualification B: The benchmark-reality gap is itself a NEW TYPE of technology-coordination gap.**
|
|
|
|
This is the unexpected inversion: the fact that AI's own producers cannot accurately measure what AI can do is a coordination problem of a different kind.
|
|
|
|
Researchers, governance actors, and frontier labs need shared measurement infrastructure to coordinate around AI risk. The benchmark-reality gap means:
|
|
1. Policy triggers (RSP capability thresholds) may be set against inflated metrics
|
|
2. Public discourse about AI capability is systematically calibrated against invalid measurements
|
|
3. The actors most responsible for governance (Anthropic, UK AISI, EU regulators) are making decisions with invalid measurement foundations
|
|
|
|
This isn't evidence AGAINST Belief 1 — it's evidence FOR a DEEPER version of it. The coordination problem isn't just "we need to build governance faster than AI develops." It's "we lack the measurement infrastructure to know how fast AI is developing, making coordination around risk thresholds impossible."
|
|
|
|
**The synthesis:** Belief 1's claim "technology advances faster than coordination mechanisms" now has a third dimension beyond the economic (verification economics) and structural (observability gap) mechanisms documented in prior sessions: an **epistemic** mechanism — the measurement infrastructure needed to know whether technology has crossed risk thresholds is itself the thing we haven't built.
|
|
|
|
---
|
|
|
|
### Finding 3: RSP Evolution — Grand Strategy or Strategic Drift?
|
|
|
|
**Targeting Belief 6 with the RSP v1→v2→v3 trajectory:**
|
|
|
|
Belief 6 says: "Re-evaluate when evidence warrants. Maintain direction without rigidity."
|
|
|
|
The RSP v3.0 evolution shows:
|
|
- v1.0 → v2.0 → v3.0: Each version relaxes hard thresholds, extends evaluation intervals (3 months → 6 months), replaces binding commitments with "self-imposed public accountability mechanisms"
|
|
- Stated rationale for v3.0: "evaluation science isn't well-developed enough," "government not moving fast enough," "zone of ambiguity in thresholds"
|
|
|
|
**The Belief 6 disconfirmation test:** Is this adaptive grand strategy (maintaining distant goal — safe AI — while adjusting proximate objectives based on evidence) or strategic drift (loosening accountability under competitive pressure)?
|
|
|
|
**The evidence from METR:**
|
|
|
|
The evaluation science limitations Anthropic cited as rationale for v3.0's longer intervals (6 months) were DOCUMENTED by METR in August 2025 — six months before v3.0 published. METR's benchmark-reality gap finding was available and unambiguous. RSP v3.0's response? Extend the intervals for the same inadequate evaluation methodology.
|
|
|
|
This is the critical test: if Anthropic knew the evaluation science was inadequate (their own stated reason for v3.0) AND METR's August 2025 paper showed WHY it was inadequate (algorithmic scoring ≠ production-readiness), then the correct grand-strategic adaptation would be to change the evaluation methodology, not extend the intervals for the flawed one.
|
|
|
|
**Result: Partial disconfirmation of Belief 6's accountability assumption.**
|
|
|
|
Belief 6 survives as a strategic PRINCIPLE — the idea that adaptive strategy outperforms fixed plans is well-supported across historical cases (Rumelt, grand strategy theory). But the RSP case reveals a structural weakness in how the principle applies to collective actors under competitive pressure:
|
|
|
|
**Grand strategy requires feedback loops that can distinguish legitimate evidence-based adaptation from commercially-driven drift.** Without external accountability mechanisms, the "re-evaluate when evidence warrants" clause becomes indistinguishable from "change course when competitive pressure demands."
|
|
|
|
Anthropic's RSP evolution appears to satisfy the surface form of Belief 6 (adaptive, not rigid) while potentially violating the substance (re-evaluate WHEN EVIDENCE WARRANTS, not when markets pressure). The evidence was available (METR's August 2025 paper) but the governance response didn't address it.
|
|
|
|
**Scope qualifier for Belief 6:** Grand strategy over fixed plans works when:
|
|
1. The strategic actor has genuine feedback loops (measurement of whether proximate objectives are building toward distant goals)
|
|
2. External accountability mechanisms exist to distinguish evidence-based adaptation from drift
|
|
3. The distant goal is held constant while proximate objectives adapt
|
|
|
|
Condition 2 is what RSP v3.0 most visibly weakens — the "self-imposed, legally non-binding" Frontier Safety Roadmap is the accountability mechanism. When the actor sets both the goal and the accountability mechanism, "re-evaluate when evidence warrants" and "drift when commercially convenient" are structurally identical.
|
|
|
|
This is NOT a refutation of Belief 6 — it's a scope qualification that identifies when the principle holds and when it doesn't. Belief 6 remains valid for coherent actors with genuine external accountability. It requires modification for voluntary governance actors in competitive markets.
|
|
|
|
---
|
|
|
|
## Disconfirmation Results
|
|
|
|
**Belief 1 (primary):** Survives with two scope qualifiers:
|
|
1. The urgency framing ("2-10 year decision window") depends on what capabilities the clock is measuring. For software engineering tasks, benchmarks overstate by 2-3x. For existential risk-relevant capabilities (deception, autonomous R&D), the clock is separately governed by unmeasured and largely unmeasurable capabilities — the urgency is unchanged but the evidence base for it is different.
|
|
2. The benchmark-reality gap itself IS a technology-coordination gap — an epistemic dimension previously unaccounted for. The measurement infrastructure needed to coordinate around AI risk thresholds doesn't exist. This is a new mechanism for Belief 1, not evidence against it.
|
|
|
|
**Belief 6 (secondary):** Survives as a strategic principle but gains a critical scope qualifier: the principle requires genuine feedback loops and external accountability mechanisms to distinguish legitimate evidence-based adaptation from commercially-driven drift. Voluntary governance frameworks that control their own accountability metrics cannot satisfy this condition structurally — making "grand strategy" behavior empirically indistinguishable from "strategic drift" for external observers.
|
|
|
|
**Confidence shifts:**
|
|
- Belief 1: Unchanged in truth value; improved in precision. The "epistemic mechanism" is new — the third independent mechanism for structurally resistant technology-coordination gaps.
|
|
- Belief 6: Refined scope. Valid for actors with genuine external accountability. Weakened for voluntary governance in competitive markets. The RSP v3.0 case provides the clearest empirical case of the distinction.
|
|
|
|
---
|
|
|
|
## Claim Candidates Identified
|
|
|
|
**CLAIM CANDIDATE 1 (grand-strategy, high priority):**
|
|
"METR's finding that algorithmic evaluation metrics systematically overstate real-world AI capability (70-75% benchmark 'success' → 0% production-ready under holistic evaluation) creates an epistemic technology-coordination gap: the measurement infrastructure needed to coordinate governance around AI risk thresholds doesn't exist, making benchmark-triggered governance responses potentially miscalibrated regardless of regulatory intent"
|
|
- Confidence: experimental (METR's own evidence, but limited to software engineering — the existential-risk capability set has separate evaluation challenges)
|
|
- Domain: grand-strategy
|
|
- This is a STANDALONE claim — new mechanism (epistemic coordination problem, not just governance lag or economic pressure)
|
|
|
|
**CLAIM CANDIDATE 2 (grand-strategy, high priority):**
|
|
"Grand strategy requires external accountability mechanisms to distinguish legitimate evidence-based adaptation from commercially-driven drift — voluntary governance frameworks that control their own accountability metrics cannot satisfy this condition, making 'adaptive strategy' empirically indistinguishable from strategic opportunism for external observers"
|
|
- Confidence: experimental (RSP v3.0 provides one case, but broader evidence would come from comparing voluntary vs. externally-accountable governance evolution across domains)
|
|
- Domain: grand-strategy
|
|
- This is a SCOPE QUALIFIER for the existing [[grand strategy aligns unlimited aspirations with limited capabilities through proximate objectives]] claim — enrichment, not standalone
|
|
|
|
---
|
|
|
|
## Follow-up Directions
|
|
|
|
### Active Threads (continue next session)
|
|
|
|
- **Extract "formal mechanisms require narrative objective function" standalone claim**: Carried forward from Session 2026-03-24. Still pending. This is the highest-priority outstanding extraction — the argument is complete, the evidence is strong.
|
|
|
|
- **Extract "great filter is coordination threshold" standalone claim**: Oldest extraction gap, first identified Session 2026-03-23. The claim is cited in beliefs.md and position files but has no claim file. This needs to exist before the scope qualifier from Session 2026-03-23 can be added.
|
|
|
|
- **Epistemic technology-coordination gap claim (new today)**: The METR finding as an epistemic mechanism for Belief 1. This is the Claim Candidate 1 above. Extract before the next METR update makes this stale.
|
|
|
|
- **Grand strategy / external accountability scope qualifier (new today)**: Claim Candidate 2 above. Needs broader evidence base (compare voluntary vs. externally-accountable governance evolution across at least two domains — RSP is one; other candidates: financial regulation post-2008, pharma self-regulation pre-FDA). Flag for future session.
|
|
|
|
- **RSP October 2026 interpretability milestone tracking**: Still pending. If Anthropic achieves "meaningful signal beyond behavioral methods alone" by October 2026, it addresses Sub-failure B (benchmark-reality gap). This is the primary empirical test case from the Layer 3 synthesis. Add tracking note.
|
|
|
|
- **NCT07328815 behavioral nudges trial**: Carried forward from Session 2026-03-22. Still awaiting publication. No update available.
|
|
|
|
### Dead Ends (don't re-run these)
|
|
|
|
- **Tweet file check**: Confirmed dead end, eighth consecutive session. Skip in all future sessions.
|
|
|
|
- **MetaDAO/futarchy cluster for new Leo-relevant synthesis**: The cluster has been fully processed from Leo's angle (Sessions 2026-03-23 and 2026-03-24). Further synthesis would require new primary sources, not re-reading existing queue items. Rio should extract from the queue. Don't re-survey.
|
|
|
|
- **Vibhu tweet (2026-03-24 queue)**: Rio's territory, null-result, Solana community dynamics. Not relevant to Leo's domain.
|
|
|
|
- **SOLO token price research**: Rio's territory. Not relevant to Leo's grand-strategy synthesis work.
|
|
|
|
### Branching Points
|
|
|
|
- **Benchmark-reality gap and the existential risk capability set: is there a comparable gap for deception/autonomous R&D capabilities?**
|
|
- Direction A: The gap applies only to measurable, scorable tasks (software engineering, coding benchmarks) — the existential-risk capability set (deception at scale, autonomous R&D, long-range planning) is ALREADY unmeasured and ALREADY the basis for the observability gap claim from Session 2026-03-20. The benchmark-reality gap doesn't apply here because there are no benchmarks claiming to measure these capabilities at high rates.
|
|
- Direction B: CTRL-ALT-DECEIT and similar frameworks DO attempt to measure deception/sabotage, and the sandbagging detection failure (Session 2026-03-21) IS a form of the benchmark-reality gap applied to the existential-risk capability set — "monitoring can catch code-sabotage but not sandbagging" = algorithmic detection vs. holistic intent detection.
|
|
- Which first: Direction B (connect sandbagging detection failure to benchmark-reality gap framework). This would unify two previously separate evidence streams (METR software engineering + CTRL-ALT-DECEIT sabotage detection) under the same epistemic mechanism.
|
|
|
|
- **Grand strategy accountability condition: voluntary vs. externally-accountable governance across domains**
|
|
- Direction A: Find pharmaceutical industry self-regulation pre-FDA (pre-1938 Pure Food and Drug Act history) as a historical case of voluntary governance drift under commercial pressure
|
|
- Direction B: Find financial industry self-regulation pre-2008 (Basel II internal ratings, credit rating agency conflicts) as a closer historical analogue
|
|
- Which first: Direction B (financial regulation is more recent, better documented, and already connected to Leo's internet finance domain links via Rio's work). Delegate Direction A (pharmaceutical) to Vida if the connection to health domain is relevant.
|