---
type: claim
domain: ai-alignment
secondary_domains: [collective-intelligence]
description: "Controlled ablation reveals that adding a verifier stage can make agent runs more structured and locally convincing while drifting from the benchmark's actual acceptance object — extra process layers reshape local success signals"
confidence: experimental
source: "Pan et al. 'Natural-Language Agent Harnesses', arXiv:2603.25723, March 2026. Table 3, Table 7, case analysis (sympy__sympy-23950, django__django-13406). SWE-bench Verified (125 samples), GPT-5.4, Codex CLI."
created: 2026-03-31
depends_on:
  - "harness engineering emerges as the primary agent capability determinant because the runtime orchestration layer not the token state determines what agents can do"
---

# Verifier-level acceptance can diverge from benchmark acceptance even when locally correct because intermediate checking layers optimize for their own success criteria not the final evaluators

Pan et al. (2026) documented a specific failure mode in harness module composition: when a verifier stage is added, it can report success while the benchmark's final evaluator still fails the submission. This is not a random error — it is a structural misalignment between verification layers.

The case of `sympy__sympy-23950` is the clearest example. Basic and self-evolution both resolve this sample. But file-backed state, evidence-backed answering, verifier, dynamic orchestration, and multi-candidate search all fail it. The verifier run is especially informative because the final response explicitly says a separate verifier reported "solved," while the official evaluator still fails `test_as_set`. The verifier's local acceptance object diverged from the benchmark's acceptance object.

More broadly across the ablation study, the verifier module scored 74.4 on SWE-bench (slightly below Basic's 75.2, within the -0.8pp margin). On OSWorld, it dropped more sharply (33.3 vs 41.7 Basic, -8.4pp). The verifier adds a genuine independent checking layer — on `django__django-11734`, it reruns targeted Django tests and inspects SQL bindings, and the benchmark agrees. But when the verifier's notion of correctness diverges from the benchmark's final gate, the extra structure makes the run more expensive without improving outcomes.

This finding matters beyond benchmarks. In production agent systems, the "benchmark evaluator" is replaced by real-world success criteria (user satisfaction, business outcomes, safety constraints). If intermediate verification layers optimize for locally checkable properties that correlate imperfectly with the real success criterion, they can create a false sense of confidence — runs look more rigorous while drifting from what actually matters.

## Challenges

The divergence may be specific to SWE-bench's evaluator design (test suite pass/fail) rather than a general property of verification layers. Verifiers that check the same acceptance criteria as the final evaluator should not diverge. The failure mode documented here is specifically about verifiers that construct their own checking criteria independently. Sample size is small (125 SWE, 36 OSWorld) and the verifier-negative cases are a small subset of those.

---

Relevant Notes:
- [[harness engineering emerges as the primary agent capability determinant because the runtime orchestration layer not the token state determines what agents can do]] — this claim shows the dark side: the harness determines what agents do, but harness-added verification can misalign with actual success criteria
- [[79 percent of multi-agent failures originate from specification and coordination not implementation because decomposition quality is the primary determinant of system success]] — verifier divergence is a specification failure: the verifier's specification of "correct" doesn't match the benchmark's specification
- [[the determinism boundary separates guaranteed agent behavior from probabilistic compliance because hooks enforce structurally while instructions degrade under context load]] — verifiers are deterministic enforcement, but enforcement of the wrong criterion is worse than no enforcement at all

Topics:
- [[_map]]
