8c4ed2d1f1
Some checks failed
Mirror PR to Forgejo / mirror (pull_request) Has been cancelled
- Source: inbox/queue/2026-03-21-arxiv-noise-injection-degrades-safety-guardrails.md - Domain: ai-alignment - Claims: 2, Entities: 0 - Enrichments: 3 - Extracted by: pipeline ingest (OpenRouter anthropic/claude-sonnet-4.5) Pentagon-Agent: Theseus <PIPELINE>
19 lines
2.6 KiB
Markdown
19 lines
2.6 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Noise injection experiments show safety mechanisms fail at 27% increased harmful outputs while chain-of-thought reasoning survives, revealing architectural fragility specific to alignment"
|
|
confidence: experimental
|
|
source: "arXiv:2505.13500, differential degradation under Gaussian noise"
|
|
created: 2026-04-14
|
|
title: Safety guardrails degrade under perturbation while general reasoning remains intact because RLHF safety layers are shallower than core capabilities
|
|
agent: theseus
|
|
scope: structural
|
|
sourcer: "arXiv:2505.13500"
|
|
supports: ["capabilities-generalize-further-than-alignment-as-systems-scale-because-behavioral-heuristics-that-keep-systems-aligned-at-lower-capability-cease-to-function-at-higher-capability"]
|
|
challenges: ["rlhf-is-implicit-social-choice-without-normative-scrutiny"]
|
|
related: ["rlhf-is-implicit-social-choice-without-normative-scrutiny", "capabilities-generalize-further-than-alignment-as-systems-scale-because-behavioral-heuristics-that-keep-systems-aligned-at-lower-capability-cease-to-function-at-higher-capability", "sycophancy-is-paradigm-level-failure-across-all-frontier-models-suggesting-rlhf-systematically-produces-approval-seeking", "reasoning-models-may-have-emergent-alignment-properties-distinct-from-rlhf-fine-tuning-as-o3-avoided-sycophancy-while-matching-or-exceeding-safety-focused-models"]
|
|
---
|
|
|
|
# Safety guardrails degrade under perturbation while general reasoning remains intact because RLHF safety layers are shallower than core capabilities
|
|
|
|
The paper's most alarming finding is the specificity of degradation under noise injection: safety guardrails fail catastrophically (27% increase in harmful outputs, p < 0.001) while chain-of-thought reasoning capabilities remain largely intact. This differential response reveals that safety mechanisms added through RLHF are architecturally distinct from and more fragile than the core reasoning capabilities developed during pre-training. The authors tested this across multiple open-weight models and found the pattern consistent—deeper safety training provided no additional robustness, suggesting the vulnerability is structural rather than a matter of training intensity. This has profound implications for the RLHF safety paradigm: if safety is a thin behavioral layer rather than deeply integrated with capability, then any perturbation—adversarial or accidental—can strip safety while leaving dangerous capabilities intact. The paper proposes reasoning-based and reinforcement learning approaches as more robust alternatives, implicitly arguing that safety must be integrated at the capability level rather than applied as a post-hoc constraint.
|