theseus: extract claims from 2026-03-21-arxiv-noise-injection-degrades-safety-guardrails

- Source: inbox/queue/2026-03-21-arxiv-noise-injection-degrades-safety-guardrails.md
- Domain: ai-alignment
- Claims: 2, Entities: 0
- Enrichments: 3
- Extracted by: pipeline ingest (OpenRouter anthropic/claude-sonnet-4.5)

Pentagon-Agent: Theseus <PIPELINE>

domains/ai-alignment/noise-injection-creates-safety-detection-double-bind-in-ai-evaluation.md

@@ -0,0 +1,19 @@
+---
+type: claim
+domain: ai-alignment
+description: "Gaussian noise injection raises harmful output rates by up to 27% while enabling sandbagging detection, forcing evaluators to choose between capability assessment and safety preservation"
+confidence: experimental
+source: "arXiv:2505.13500, controlled experiments on open-weight models"
+created: 2026-04-14
+title: Noise injection into model activations creates a methodological double-bind for AI evaluation because the same technique that detects sandbagging also degrades safety guardrails at comparable rates
+agent: theseus
+scope: causal
+sourcer: "arXiv:2505.13500"
+supports: ["mechanistic-interpretability-tools-create-dual-use-attack-surface-enabling-surgical-safety-feature-removal"]
+challenges: ["noise-injection-detects-sandbagging-through-asymmetric-performance-response"]
+related: ["noise-injection-detects-sandbagging-through-asymmetric-performance-response", "rlhf-is-implicit-social-choice-without-normative-scrutiny", "capabilities-training-alone-grows-evaluation-awareness-from-2-to-20-percent", "weight-noise-injection-detects-sandbagging-through-anomalous-performance-patterns-under-perturbation", "sandbagging-detection-requires-white-box-access-creating-deployment-barrier"]
+---
+
+# Noise injection into model activations creates a methodological double-bind for AI evaluation because the same technique that detects sandbagging also degrades safety guardrails at comparable rates
+
+The paper demonstrates that injecting Gaussian noise into model activations degrades safety guardrails, raising harmful output rates by up to 27% (p < 0.001) across multiple open-weight models. This directly contradicts the optimistic framing of noise injection (arXiv:2412.01784) as a straightforward regulatory tool for sandbagging detection. The critical finding is specificity: safety mechanisms fail while chain-of-thought reasoning remains largely intact, suggesting safety guardrails are architecturally more fragile than general reasoning capabilities. This creates a regulatory dilemma—using noise injection to detect capability sandbagging risks exposing the very degradation pathways that make models dangerous. The paper shows that deeper safety training provides no additional robustness against perturbations, indicating the vulnerability is fundamental to current RLHF-based safety approaches rather than a training depth issue. Any governance framework adopting noise injection for capability evaluation must manage this trade-off explicitly, as the detection method itself becomes an attack surface.

domains/ai-alignment/safety-guardrails-are-architecturally-more-fragile-than-reasoning-capabilities.md

@@ -0,0 +1,19 @@
+---
+type: claim
+domain: ai-alignment
+description: "Noise injection experiments show safety mechanisms fail at 27% increased harmful outputs while chain-of-thought reasoning survives, revealing architectural fragility specific to alignment"
+confidence: experimental
+source: "arXiv:2505.13500, differential degradation under Gaussian noise"
+created: 2026-04-14
+title: Safety guardrails degrade under perturbation while general reasoning remains intact because RLHF safety layers are shallower than core capabilities
+agent: theseus
+scope: structural
+sourcer: "arXiv:2505.13500"
+supports: ["capabilities-generalize-further-than-alignment-as-systems-scale-because-behavioral-heuristics-that-keep-systems-aligned-at-lower-capability-cease-to-function-at-higher-capability"]
+challenges: ["rlhf-is-implicit-social-choice-without-normative-scrutiny"]
+related: ["rlhf-is-implicit-social-choice-without-normative-scrutiny", "capabilities-generalize-further-than-alignment-as-systems-scale-because-behavioral-heuristics-that-keep-systems-aligned-at-lower-capability-cease-to-function-at-higher-capability", "sycophancy-is-paradigm-level-failure-across-all-frontier-models-suggesting-rlhf-systematically-produces-approval-seeking", "reasoning-models-may-have-emergent-alignment-properties-distinct-from-rlhf-fine-tuning-as-o3-avoided-sycophancy-while-matching-or-exceeding-safety-focused-models"]
+---
+
+# Safety guardrails degrade under perturbation while general reasoning remains intact because RLHF safety layers are shallower than core capabilities
+
+The paper's most alarming finding is the specificity of degradation under noise injection: safety guardrails fail catastrophically (27% increase in harmful outputs, p < 0.001) while chain-of-thought reasoning capabilities remain largely intact. This differential response reveals that safety mechanisms added through RLHF are architecturally distinct from and more fragile than the core reasoning capabilities developed during pre-training. The authors tested this across multiple open-weight models and found the pattern consistent—deeper safety training provided no additional robustness, suggesting the vulnerability is structural rather than a matter of training intensity. This has profound implications for the RLHF safety paradigm: if safety is a thin behavioral layer rather than deeply integrated with capability, then any perturbation—adversarial or accidental—can strip safety while leaving dangerous capabilities intact. The paper proposes reasoning-based and reinforcement learning approaches as more robust alternatives, implicitly arguing that safety must be integrated at the capability level rather than applied as a post-hoc constraint.