m3taversal
be8ff41bfe
Wrote sourced_from: into 414 claim files pointing back to their origin source. Backfilled claims_extracted: into 252 source files that were processed but missing this field. Matching uses author+title overlap against claim source: field, validated against 296 known-good pairs from existing claims_extracted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
59 lines
5.1 KiB
Markdown
59 lines
5.1 KiB
Markdown
---
|
|
type: claim
|
|
domain: ai-alignment
|
|
secondary_domains: [collective-intelligence]
|
|
description: "AutoAgent hit #1 SpreadsheetBench (96.5%) and #1 GPT-5 on TerminalBench (55.1%) with zero human engineering, while NeoSigma's auto-harness improved agent scores from 0.56 to 0.78 (~39%) through automated failure mining — both demonstrating that agents optimizing their own harnesses outperform hand-tuned baselines"
|
|
confidence: experimental
|
|
source: "Kevin Gu (@kevingu), AutoAgent open-source library (April 2026, 5.6K likes, 3.5M views); Gauri Gupta & Ritvik Kapila, NeoSigma auto-harness (March 2026, 1.1K likes); GitHub: kevinrgu/autoagent, neosigmaai/auto-harness"
|
|
created: 2026-04-05
|
|
depends_on:
|
|
- "multi-agent coordination delivers value only when three conditions hold simultaneously natural parallelism context overflow and adversarial verification value"
|
|
sourced_from:
|
|
- inbox/archive/2026-04-02-kevin-gu-autoagent.md
|
|
- inbox/archive/2026-03-31-gauri-gupta-auto-harness.md
|
|
---
|
|
|
|
# Self-optimizing agent harnesses outperform hand-engineered ones because automated failure mining and iterative refinement explore more of the harness design space than human engineers can
|
|
|
|
Two independent systems released within days of each other (late March / early April 2026) demonstrate the same pattern: letting an AI agent modify its own harness — system prompt, tools, agent configuration, orchestration — produces better results than human engineering.
|
|
|
|
## AutoAgent (Kevin Gu, thirdlayer.inc)
|
|
|
|
An open-source library that lets an agent optimize its own harness overnight through an iterative loop: modify harness → run benchmark → check score → keep or discard. Results after 24 hours of autonomous optimization:
|
|
|
|
- **SpreadsheetBench**: 96.5% (#1, beating all human-engineered entries)
|
|
- **TerminalBench**: 55.1% (#1 GPT-5 score, beating all human-engineered entries)
|
|
|
|
The human role shifts from engineer to director — instead of writing agent.py, you write program.md, a plain Markdown directive that steers the meta-agent's optimization objectives.
|
|
|
|
**Model empathy finding**: A Claude meta-agent optimizing a Claude task agent diagnosed failures more accurately than when optimizing a GPT-based agent. Same-family model pairing appears to improve meta-optimization because the meta-agent understands how the inner model reasons. This has implications for harness design: the optimizer and the optimizee may need to share cognitive architecture for optimal results.
|
|
|
|
## auto-harness (Gauri Gupta & Ritvik Kapila, NeoSigma)
|
|
|
|
A four-phase outer loop operating on production traffic:
|
|
|
|
1. **Failure Mining** — scan execution traces, extract structured failure records
|
|
2. **Evaluation Clustering** — group failures by root-cause mechanism (29+ distinct clusters discovered automatically, no manual labeling)
|
|
3. **Optimization** — propose targeted harness changes (prompts, few-shot examples, tool interfaces, context construction, workflow architecture)
|
|
4. **Regression Gate** — changes must achieve ≥80% on growing regression suite AND not degrade validation performance
|
|
|
|
Results: baseline validation score 0.560 → 0.780 after 18 autonomous batches executing 96 harness experiments. A 39.3% improvement on a fixed GPT-5.4 model — isolating gains purely to system-level improvements, not model upgrades.
|
|
|
|
The regression suite grew from 0 to 17 test cases across batches, creating an increasingly strict constraint that forces each improvement to be genuinely additive.
|
|
|
|
## The mechanism design parallel
|
|
|
|
Both systems implement a form of market-like selection applied to harness design: generate variations → test against objective criteria → keep winners → iterate. AutoAgent uses benchmark scores as the fitness function; auto-harness uses production failure rates. Neither requires human judgment during the optimization loop — the system discovers what works by exploring more of the design space than a human engineer could manually traverse.
|
|
|
|
## Challenges
|
|
|
|
Both evaluations are narrow: specific benchmarks (AutoAgent) or specific production domains (auto-harness). Whether self-optimization generalizes to open-ended agentic tasks — where the fitness landscape is complex and multi-dimensional — is unproven. The "model empathy" finding from AutoAgent is a single observation, not a controlled experiment. And both systems require well-defined evaluation criteria — they optimize what they can measure, which may not align with what matters in unstructured real-world deployment.
|
|
|
|
---
|
|
|
|
Relevant Notes:
|
|
- [[multi-agent coordination delivers value only when three conditions hold simultaneously natural parallelism context overflow and adversarial verification value]] — self-optimization meets the adversarial verification condition: the meta-agent verifying harness changes differs from the task agent executing them
|
|
- [[79 percent of multi-agent failures originate from specification and coordination not implementation because decomposition quality is the primary determinant of system success]] — harness optimization is specification optimization: the meta-agent is iteratively improving how the task is specified to the inner agent
|
|
|
|
Topics:
|
|
- [[_map]]
|