---
type: claim
domain: ai-alignment
description: AuditBench found that black-box scaffolding (using one model to systematically probe another) was the most effective alignment auditing technique overall, suggesting behavioral probing may be more tractable than mechanistic understanding
confidence: experimental
source: Anthropic Fellows / Alignment Science Team, AuditBench comparative evaluation of 13 tool configurations
created: 2026-03-29
attribution:
  extractor:
    - handle: "theseus"
  sourcer:
    - handle: "anthropic-fellows-/-alignment-science-team"
      context: "Anthropic Fellows / Alignment Science Team, AuditBench comparative evaluation of 13 tool configurations"
related:
- alignment auditing tools fail through tool to agent gap not tool quality
reweave_edges:
- alignment auditing tools fail through tool to agent gap not tool quality|related|2026-03-31
- interpretability effectiveness anti correlates with adversarial training making tools hurt performance on sophisticated misalignment|challenges|2026-03-31
- white box interpretability fails on adversarially trained models creating anti correlation with threat model|challenges|2026-03-31
challenges:
- interpretability effectiveness anti correlates with adversarial training making tools hurt performance on sophisticated misalignment
- white box interpretability fails on adversarially trained models creating anti correlation with threat model
---

# Scaffolded black-box tools where an auxiliary model generates diverse prompts for the target are most effective at uncovering hidden behaviors, outperforming white-box interpretability approaches

Across AuditBench's 56 models with hidden behaviors, scaffolded black-box tools proved most effective overall. In this approach, an auxiliary AI model generates diverse prompts designed to elicit the target model's hidden behaviors, without requiring access to the target's internal representations. This outperformed white-box interpretability tools that attempted to understand hidden behaviors through mechanistic analysis of model internals. The finding suggests that behavioral probing — systematically exploring a model's input-output behavior — may be more tractable for alignment auditing than mechanistic interpretability. This has implications for research prioritization: if black-box scaffolding is more effective and more robust across different model training regimes, alignment research may get better returns from investing in sophisticated prompting strategies than in interpretability tools. However, the tool-to-agent gap still applies — even the most effective tools fail when investigator agents cannot use them properly.

---

Relevant Notes:
- pre-deployment-AI-evaluations-do-not-predict-real-world-risk-creating-institutional-governance-built-on-unreliable-foundations.md

Topics:
- [[_map]]
