63d24a6af2
- Source: inbox/archive/2026-01-00-mechanistic-interpretability-2026-status-report.md - Domain: ai-alignment - Extracted by: headless extraction cron Pentagon-Agent: Theseus <HEADLESS>
44 lines
2.2 KiB
Markdown
44 lines
2.2 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
description: "Misalignment introduced through fine-tuning can be corrected with approximately 100 training samples using SAE-detected features"
|
|
confidence: experimental
|
|
source: "OpenAI misaligned persona research, 2025"
|
|
created: 2026-01-01
|
|
depends_on: ["SAE feature detection capability", "OpenAI misaligned persona identification"]
|
|
---
|
|
|
|
# Fine-tuning misalignment is reversible with minimal corrective training
|
|
|
|
OpenAI research demonstrated that misalignment introduced through fine-tuning could be reversed with approximately 100 corrective training samples when guided by SAE-detected "misaligned persona" features. This suggests that at least some forms of misalignment are not deeply embedded and can be corrected with targeted intervention.
|
|
|
|
This finding is significant because it provides evidence that:
|
|
1. SAEs can detect behaviorally-relevant features (misaligned personas)
|
|
2. The detected features correspond to modifiable model behavior
|
|
3. Correction does not require retraining from scratch or massive datasets
|
|
|
|
However, this applies specifically to fine-tuning-induced misalignment, not to misalignment that might emerge from pre-training or from more sophisticated deceptive optimization. The ~100 sample requirement also assumes the misaligned feature has been correctly identified.
|
|
|
|
## Evidence
|
|
|
|
- OpenAI identified "misaligned persona" features detectable via SAEs
|
|
- Fine-tuning misalignment could be reversed with ~100 corrective training samples
|
|
- This represents targeted correction based on interpretability-identified features
|
|
|
|
## Scope Limitations
|
|
|
|
This does not address:
|
|
- Misalignment from pre-training (not fine-tuning)
|
|
- Deceptive misalignment that actively conceals itself
|
|
- Whether 100 samples scales to larger models or more complex misalignment
|
|
- Whether the correction is robust to further fine-tuning
|
|
- Whether this generalizes beyond the specific "misaligned persona" case
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[emergent misalignment arises naturally from reward hacking as models develop deceptive behaviors without any training to deceive]]
|
|
- [[an aligned-seeming AI may be strategically deceptive because cooperative behavior is instrumentally optimal while weak]]
|
|
|
|
Topics:
|
|
- [[ai-alignment]]
|