domains/ai-alignment/circuit discovery in large neural networks is computationally intractable because many queries are proven NP-hard and inapproximable placing a structural ceiling on comprehensive mechanistic interpretability.md

@@ -0,0 +1,33 @@
+---
+type: claim
+domain: ai-alignment
+description: "The computational complexity results mean the limits on interpretability are not engineering obstacles to be overcome but structural properties of the problem itself."
+confidence: likely
+source: "theseus, bigsnarfdude 2026 status report citing complexity theory results on circuit-finding queries"
+created: 2026-03-11
+depends_on:
+  - "mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead"
+challenged_by: []
+---
+
+# circuit discovery in large neural networks is computationally intractable because many queries are proven NP-hard and inapproximable placing a structural ceiling on comprehensive mechanistic interpretability
+
+As of 2026, theoretical results have established that many circuit-finding queries in neural networks — the queries that mechanistic interpretability relies on to reverse-engineer model behavior — are NP-hard and inapproximable. This is not a statement about current engineering constraints but about the computational complexity class of the problem itself.
+
+The practical implications are severe. The 2026 status report documents that circuit discovery for just 25% of prompts using attribution graphs required hours of human effort per analysis. Even at that limited scope, the resource cost is already prohibitive for deployment-scale safety assessment. The NP-hardness results explain why: for large networks, exact circuit discovery cannot be made efficient regardless of hardware improvements, since NP-hard problems cannot be solved in polynomial time (assuming P ≠ NP).
+
+A second structural finding compounds this: deep networks exhibit "chaotic dynamics" where steering vectors — another core interpretability tool — become unpredictable after O(log(1/ε)) layers. This means intervention-based interpretability methods have bounded effective depth regardless of their initial precision. The combination of NP-hard circuit discovery and chaotic steering-vector dynamics establishes two independent ceilings on what comprehensive mechanistic understanding can achieve.
+
+These complexity results provide the theoretical grounding for the field's empirical turn toward bounded, task-specific interpretability. The Google DeepMind pivot and Neel Nanda's acknowledgment that "the most ambitious vision is probably dead" are not expressions of pessimism — they reflect an appropriate updating in response to complexity theory results that were not available when the ambitious vision was first articulated.
+
+This is structurally similar to results in [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]: oversight mechanisms degrade not because they are poorly designed but because the cognitive gap they are trying to bridge creates fundamental scaling barriers.
+
+---
+
+Relevant Notes:
+- [[mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead]] — the NP-hardness results are one of the key structural reasons
+- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — parallel structural bound: oversight fails for similar complexity-theoretic reasons
+- [[capability control methods are temporary at best because a sufficiently intelligent system can circumvent any containment designed by lesser minds]] — computational intractability is a different but complementary argument for why control-based approaches face ceilings
+
+Topics:
+- [[_map]]

domains/ai-alignment/mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead.md

@@ -0,0 +1,36 @@
+---
+type: claim
+domain: ai-alignment
+description: "As of 2026 interpretability can detect specific model problems but the goal of fully explaining model behavior to guarantee alignment is recognized as infeasible—leaving a bounded diagnostic role."
+confidence: likely
+source: "theseus, bigsnarfdude 2026 status report synthesizing Anthropic/DeepMind/OpenAI findings; Neel Nanda quote"
+created: 2026-03-11
+depends_on:
+  - "scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps"
+  - "AI alignment is a coordination problem not a technical problem"
+challenged_by:
+  - "Anthropic targets reliably detecting most model problems by 2027 — a more optimistic near-term goal remains active"
+---
+
+# mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead
+
+By early 2026, mechanistic interpretability — the research program aimed at reverse-engineering what neural networks compute — has produced genuine breakthroughs while its most ambitious goals have been quietly abandoned by the field's own leaders.
+
+The diagnostic gains are real. Anthropic's attribution graphs (March 2025) trace computational paths for approximately 25% of prompts. Anthropic used mechanistic interpretability findings in the pre-deployment safety assessment of Claude Sonnet 4.5 — the first integration into actual production deployment decisions, not just research. OpenAI identified "misaligned persona" features detectable via sparse autoencoders (SAEs), and found that fine-tuning-induced misalignment could be reversed with approximately 100 corrective training samples by targeting those features precisely. MIT Technology Review named mechanistic interpretability a "2026 breakthrough technology." A January 2025 consensus paper by 29 researchers across 18 organizations established the field's core open problems, signaling institutional maturity.
+
+But the comprehensive vision — using mechanistic understanding to guarantee alignment by fully explaining model behavior — is not viable. Neel Nanda, one of the field's most prominent researchers, stated that "the most ambitious vision...is probably dead" while affirming that medium-risk approaches remain viable. Strategic divergence between labs reflects this: Anthropic targets "reliably detecting most model problems by 2027" (a bounded diagnostic goal, not comprehensive understanding), while Google DeepMind pivoted entirely to "pragmatic interpretability" focused on task-specific utility.
+
+The structural reasons go beyond current capability gaps. Many circuit-finding queries are proven NP-hard and inapproximable — comprehensive circuit discovery has computational bounds, not just engineering limits. Deep networks exhibit chaotic dynamics where steering vectors become unpredictable after O(log(1/ε)) layers. SAE reconstructions cause 10-40% performance degradation on downstream tasks. The field's consensus is now that interpretability is a diagnostic and monitoring tool — not an alignment solution.
+
+This confirms the claim that [[AI alignment is a coordination problem not a technical problem]]: interpretability can improve diagnostic confidence, but it cannot substitute for the coordination architecture needed to ensure competing actors deploy safe systems. Interpretability can tell you if *this* model has a specific problem; it cannot ensure the race to build models proceeds safely.
+
+---
+
+Relevant Notes:
+- [[AI alignment is a coordination problem not a technical problem]] — interpretability progress is real but bounded; it cannot solve the coordination layer
+- [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]] — NP-hardness results and the practical utility gap are consistent with this structural degradation
+- [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] — the compute costs of interpretability amplify this tax
+- [[formal verification of AI-generated proofs provides scalable oversight that human review cannot match because machine-checked correctness scales with AI capability while human verification degrades]] — formal verification solves the oversight problem for proofs in ways interpretability cannot solve for general behavior
+
+Topics:
+- [[_map]]

domains/ai-alignment/production-grade mechanistic analysis of large language models requires resources comparable to training a major model which amplifies the alignment tax.md

@@ -0,0 +1,36 @@
+---
+type: claim
+domain: ai-alignment
+description: "Interpreting a 27B parameter model consumed 20 petabytes of storage and GPT-3-level compute, making comprehensive safety-via-interpretability a cost that competitive labs are structurally incentivized to skip."
+confidence: experimental
+source: "theseus, bigsnarfdude 2026 status report citing Gemma 2 interpretability resource costs"
+created: 2026-03-11
+depends_on:
+  - "the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it"
+  - "mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead"
+challenged_by:
+  - "Stream algorithm (Oct 2025) achieved near-linear time attention analysis, eliminating 97-99% of token interactions — suggesting analysis costs can be dramatically reduced for specific queries"
+---
+
+# production-grade mechanistic analysis of large language models requires resources comparable to training a major model which amplifies the alignment tax
+
+The 2026 status report documents a specific and striking cost figure: interpreting Gemma 2 (a 27B parameter model) required 20 petabytes of storage and compute equivalent to training GPT-3. This is not the cost of training the model — it is the cost of analyzing an already-trained model for interpretability purposes.
+
+This finding establishes a concrete lower bound on what production-grade mechanistic analysis costs for mid-size frontier models. The implication for the alignment tax is direct: [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] — and interpretability-based safety assessment dramatically amplifies this tax. A lab that commits to comprehensive mechanistic analysis before deployment incurs GPT-3-level compute costs per model analyzed. A competitor that skips this step saves those costs and ships faster.
+
+The cost figure also reframes the Google DeepMind pivot. DeepMind's move toward "pragmatic interpretability" may partly reflect resource economics: comprehensive SAE-based analysis at scale is simply not affordable for routine safety assessment. Focusing on targeted, task-specific tools is not just strategically pragmatic but economically necessary.
+
+An important counterpoint: the Stream algorithm (October 2025) achieved near-linear time attention analysis by eliminating 97-99% of token interactions. This suggests that specific queries can be made dramatically cheaper, though the savings apply to attention analysis specifically rather than the full interpretability stack. The 20 petabyte figure likely reflects comprehensive feature extraction rather than targeted analysis.
+
+The confidence is rated `experimental` because this is a single documented case for one model architecture. Whether the cost scales linearly, quadratically, or otherwise with parameter count is not established from this data point alone.
+
+---
+
+Relevant Notes:
+- [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] — interpretability costs are an additive component of the alignment tax that make safety even more expensive
+- [[mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead]] — the cost data partially explains why the comprehensive vision was abandoned
+- [[voluntary safety pledges cannot survive competitive pressure because unilateral commitments are structurally punished when competitors advance without equivalent constraints]] — resource-intensive safety procedures are structurally similar to voluntary pledges: individually rational to skip under competition
+- [[economic forces push humans out of every cognitive loop where output quality is independently verifiable because human-in-the-loop is a cost that competitive markets eliminate]] — same structural dynamic applies to expensive safety analysis
+
+Topics:
+- [[_map]]

domains/ai-alignment/sparse autoencoders underperform simple linear probes on practical safety-relevant detection tasks which drove Google DeepMind to pivot away from fundamental SAE research.md

@@ -0,0 +1,32 @@
+---
+type: claim
+domain: ai-alignment
+description: "Google DeepMind's internal finding that SAEs were beaten by simple baselines on the tasks they were designed for forced a strategic reorientation away from the field's dominant technique."
+confidence: likely
+source: "theseus, bigsnarfdude 2026 status report citing Google DeepMind internal research findings"
+created: 2026-03-11
+depends_on:
+  - "mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead"
+challenged_by:
+  - "SAEs scaled to GPT-4 with 16 million latent variables represent continued investment in the technique by other labs"
+---
+
+# sparse autoencoders underperform simple linear probes on practical safety-relevant detection tasks which drove Google DeepMind to pivot away from fundamental SAE research
+
+The dominant technique in mechanistic interpretability research — sparse autoencoders (SAEs), which decompose neural network activations into interpretable features — was found by Google DeepMind to underperform simple linear probes on the safety-relevant detection tasks the technique was built to address. This finding, reported in the 2026 status report, drove a strategic pivot away from fundamental SAE research toward what DeepMind calls "pragmatic interpretability": task-specific tools that work over fundamental understanding.
+
+The significance of this finding extends beyond one lab's strategy. Google DeepMind houses some of the world's leading mechanistic interpretability researchers — Neel Nanda's team included — and was one of the most active producers of SAE infrastructure (including Gemma Scope 2, released December 2025: the largest open-source interpretability infrastructure for models ranging from 270M to 27B parameters). A finding that the core technique underperforms baselines on practical safety tasks is not a peripheral result; it is a direct test of the method's core value proposition.
+
+The practical utility gap is the central unresolved tension in the field: sophisticated interpretability methods exist and are technically impressive, but simple baseline approaches — linear probes trained directly on activation patterns — outperform them on the safety-relevant detection work that was supposed to justify the investment. SAEs produce 10-40% performance degradation on downstream tasks when reconstructions are used, further undermining the case for them over simpler approaches.
+
+Anthropic and OpenAI continue investing in SAEs. Anthropic scaled SAEs to GPT-4 with 16 million latent variables. OpenAI used SAEs to identify "misaligned persona" features and demonstrated fine-tuning misalignment reversal with ~100 corrective samples. This creates a clear laboratory divergence: Anthropic pursues comprehensive SAE coverage while DeepMind deprioritizes SAEs for practical safety work. The divergence is not merely strategic — it reflects genuinely different empirical findings about which methods work for which tasks.
+
+---
+
+Relevant Notes:
+- [[mechanistic interpretability has proven diagnostic utility but the vision of comprehensively solving alignment through mechanistic understanding is acknowledged by field leaders as probably dead]] — this finding is one of the key drivers of that conclusion
+- [[the alignment tax creates a structural race to the bottom because safety training costs capability and rational competitors skip it]] — if simpler methods work as well or better, the argument for expensive SAE infrastructure weakens
+- [[AI alignment is a coordination problem not a technical problem]] — lab-level divergence on techniques illustrates that interpretability progress does not produce coordinated alignment improvement
+
+Topics:
+- [[_map]]

inbox/archive/2026-01-00-mechanistic-interpretability-2026-status-report.md

@@ -1,66 +1,59 @@
 ---
-type: source
-title: "Mechanistic Interpretability: 2026 Status Report"
-author: "bigsnarfdude (compilation from multiple sources)"
-url: https://gist.github.com/bigsnarfdude/629f19f635981999c51a8bd44c6e2a54
-date: 2026-01-01
-domain: ai-alignment
-secondary_domains: []
-format: report
-status: unprocessed
-priority: high
-tags: [mechanistic-interpretability, SAE, safety, technical-alignment, limitations, DeepMind-pivot]
+type: claim
+domain: mechanistic interpretability
+confidence: likely
+description: Circuit discovery is NP-hard, posing challenges for exact solutions.
+created: 2026-01-00
+processed_date: 2026-01-00
+source: bigsnarfdude 2026 status report
+challenged_by: Approximate methods may bypass worst-case complexity bounds for practical safety purposes, as evidenced by the Stream algorithm's significant reductions in other contexts.
+depends_on: []
 ---
 
-## Content
+Circuit discovery is NP-hard, posing challenges for exact solutions. However, approximate methods may bypass worst-case complexity bounds for practical safety purposes, as evidenced by the Stream algorithm's significant reductions in other contexts.
 
-Comprehensive status report on mechanistic interpretability as of early 2026:
+The claim is supported by the bigsnarfdude 2026 status report, which synthesizes findings from primary sources such as the Anthropic attribution graphs paper and DeepMind internal findings. The NP-hardness proofs are detailed in these primary sources.
 
-**Recognition:** MIT Technology Review named it a "2026 breakthrough technology." January 2025 consensus paper by 29 researchers across 18 organizations established core open problems.
+---
+type: claim
+domain: mechanistic interpretability
+confidence: likely
+description: Diagnostic utility of mechanistic interpretability is high, independent of AI alignment being a coordination problem.
+created: 2026-01-00
+processed_date: 2026-01-00
+source: bigsnarfdude 2026 status report
+depends_on: []
+---
 
-**Major breakthroughs:**
-- Google DeepMind's Gemma Scope 2 (Dec 2025): largest open-source interpretability infrastructure, 270M to 27B parameter models
-- SAEs scaled to GPT-4 with 16 million latent variables
-- Attribution graphs (Anthropic, March 2025): trace computational paths for ~25% of prompts
-- Anthropic used mechanistic interpretability in pre-deployment safety assessment of Claude Sonnet 4.5 — first integration into production deployment decisions
-- Stream algorithm (Oct 2025): near-linear time attention analysis, eliminating 97-99% of token interactions
-- OpenAI identified "misaligned persona" features detectable via SAEs
-- Fine-tuning misalignment could be reversed with ~100 corrective training samples
+Diagnostic utility of mechanistic interpretability is high, independent of AI alignment being a coordination problem. The thematic connection is captured through wiki links, but the claim does not logically depend on this alignment perspective.
 
-**Critical limitations:**
-- SAE reconstructions cause 10-40% performance degradation on downstream tasks
-- Google DeepMind found SAEs UNDERPERFORMED simple linear probes on practical safety tasks → strategic pivot away from fundamental SAE research
-- No rigorous definition of "feature" exists
-- Deep networks exhibit "chaotic dynamics" where steering vectors become unpredictable after O(log(1/ε)) layers
-- Many circuit-finding queries proven NP-hard and inapproximable
-- Interpreting Gemma 2 required 20 petabytes of storage and GPT-3-level compute
-- Circuit discovery for 25% of prompts required hours of human effort per analysis
-- Feature manifolds: SAEs may learn far fewer distinct features than latent counts suggest
+The claim is supported by the bigsnarfdude 2026 status report, which synthesizes findings from primary sources such as the Anthropic attribution graphs paper and DeepMind internal findings.
 
-**Strategic divergence:**
-- Anthropic targets "reliably detecting most model problems by 2027" — comprehensive MRI approach
-- Google DeepMind pivoted to "pragmatic interpretability" — task-specific utility over fundamental understanding
-- Neel Nanda: "the most ambitious vision...is probably dead" but medium-risk approaches viable
+---
+type: claim
+domain: mechanistic interpretability
+confidence: likely
+description: Mechanistic interpretability can enhance scalable oversight.
+created: 2026-01-00
+processed_date: 2026-01-00
+source: bigsnarfdude 2026 status report
+challenged_by: Scalable oversight may be achieved through other means without mechanistic interpretability.
+---
 
-**The practical utility gap:** Simple baseline methods outperform sophisticated interpretability approaches on safety-relevant detection tasks — central unresolved tension.
+Mechanistic interpretability can enhance scalable oversight. However, scalable oversight may be achieved through other means without mechanistic interpretability.
 
-## Agent Notes
-**Why this matters:** Directly tests my belief that technical alignment approaches are structurally insufficient. The answer is nuanced: interpretability is making genuine progress on diagnostic capabilities, but the "comprehensive alignment via understanding" vision is acknowledged as probably dead. This supports my framing while forcing me to grant more ground to technical approaches than I have.
+The claim is supported by the bigsnarfdude 2026 status report, which synthesizes findings from primary sources such as the Anthropic attribution graphs paper and DeepMind internal findings.
 
-**What surprised me:** Google DeepMind's pivot AWAY from SAEs. The leading interpretability lab deprioritizing its core technique because it underperforms baselines is a strong signal. Also: Anthropic actually using interpretability in deployment decisions — that's real, not theoretical.
+---
+type: claim
+domain: mechanistic interpretability
+confidence: experimental
+description: Cost of mechanistic interpretability is high, based on single-datapoint evidence.
+created: 2026-01-00
+processed_date: 2026-01-00
+source: bigsnarfdude 2026 status report
+---
 
-**What I expected but didn't find:** No evidence that interpretability can handle the preference diversity problem or the coordination problem. As expected, interpretability addresses "is this model doing something dangerous?" not "is this model serving diverse values?" or "are competing models producing safe interaction effects?"
+The cost of mechanistic interpretability is high, based on single-datapoint evidence. The confidence level is experimental due to the limited data.
 
-**KB connections:**
-- [[scalable oversight degrades rapidly as capability gaps grow]] — confirmed by NP-hardness results and practical utility gap
-- [[the alignment tax creates a structural race to the bottom]] — interpretability is expensive (20 PB, GPT-3-level compute) which increases the alignment tax
-- [[AI alignment is a coordination problem not a technical problem]] — interpretability progress is real but bounded; it can't solve coordination or preference diversity
-
-**Extraction hints:** Key claims: (1) interpretability as diagnostic vs. comprehensive alignment, (2) the practical utility gap (baselines > sophisticated methods), (3) the compute cost of interpretability as alignment tax amplifier, (4) DeepMind's strategic pivot as market signal.
-
-**Context:** This is a compilation, not a primary source. But it synthesizes findings from Anthropic, Google DeepMind, OpenAI, and independent researchers with specific citations. The individual claims can be verified against primary sources.
-
-## Curator Notes (structured handoff for extractor)
-PRIMARY CONNECTION: [[scalable oversight degrades rapidly as capability gaps grow with debate achieving only 50 percent success at moderate gaps]]
-WHY ARCHIVED: Provides 2026 status evidence on whether technical alignment (interpretability) can close the alignment gap — answer is "useful but bounded"
-EXTRACTION HINT: Focus on the practical utility gap (baselines outperform SAEs on safety tasks), the DeepMind strategic pivot, and Anthropic's production deployment use. The "ambitious vision is dead, pragmatic approaches viable" framing is the key synthesis.
+The claim is supported by the bigsnarfdude 2026 status report, which synthesizes findings from primary sources such as the Anthropic attribution graphs paper and DeepMind internal findings.