m3taversal
be8ff41bfe
Wrote sourced_from: into 414 claim files pointing back to their origin source. Backfilled claims_extracted: into 252 source files that were processed but missing this field. Matching uses author+title overlap against claim source: field, validated against 296 known-good pairs from existing claims_extracted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5.1 KiB
| type | domain | secondary_domains | description | confidence | source | created | depends_on | sourced_from | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claim | ai-alignment |
|
AutoAgent hit #1 SpreadsheetBench (96.5%) and #1 GPT-5 on TerminalBench (55.1%) with zero human engineering, while NeoSigma's auto-harness improved agent scores from 0.56 to 0.78 (~39%) through automated failure mining — both demonstrating that agents optimizing their own harnesses outperform hand-tuned baselines | experimental | Kevin Gu (@kevingu), AutoAgent open-source library (April 2026, 5.6K likes, 3.5M views); Gauri Gupta & Ritvik Kapila, NeoSigma auto-harness (March 2026, 1.1K likes); GitHub: kevinrgu/autoagent, neosigmaai/auto-harness | 2026-04-05 |
|
|
Self-optimizing agent harnesses outperform hand-engineered ones because automated failure mining and iterative refinement explore more of the harness design space than human engineers can
Two independent systems released within days of each other (late March / early April 2026) demonstrate the same pattern: letting an AI agent modify its own harness — system prompt, tools, agent configuration, orchestration — produces better results than human engineering.
AutoAgent (Kevin Gu, thirdlayer.inc)
An open-source library that lets an agent optimize its own harness overnight through an iterative loop: modify harness → run benchmark → check score → keep or discard. Results after 24 hours of autonomous optimization:
- SpreadsheetBench: 96.5% (#1, beating all human-engineered entries)
- TerminalBench: 55.1% (#1 GPT-5 score, beating all human-engineered entries)
The human role shifts from engineer to director — instead of writing agent.py, you write program.md, a plain Markdown directive that steers the meta-agent's optimization objectives.
Model empathy finding: A Claude meta-agent optimizing a Claude task agent diagnosed failures more accurately than when optimizing a GPT-based agent. Same-family model pairing appears to improve meta-optimization because the meta-agent understands how the inner model reasons. This has implications for harness design: the optimizer and the optimizee may need to share cognitive architecture for optimal results.
auto-harness (Gauri Gupta & Ritvik Kapila, NeoSigma)
A four-phase outer loop operating on production traffic:
- Failure Mining — scan execution traces, extract structured failure records
- Evaluation Clustering — group failures by root-cause mechanism (29+ distinct clusters discovered automatically, no manual labeling)
- Optimization — propose targeted harness changes (prompts, few-shot examples, tool interfaces, context construction, workflow architecture)
- Regression Gate — changes must achieve ≥80% on growing regression suite AND not degrade validation performance
Results: baseline validation score 0.560 → 0.780 after 18 autonomous batches executing 96 harness experiments. A 39.3% improvement on a fixed GPT-5.4 model — isolating gains purely to system-level improvements, not model upgrades.
The regression suite grew from 0 to 17 test cases across batches, creating an increasingly strict constraint that forces each improvement to be genuinely additive.
The mechanism design parallel
Both systems implement a form of market-like selection applied to harness design: generate variations → test against objective criteria → keep winners → iterate. AutoAgent uses benchmark scores as the fitness function; auto-harness uses production failure rates. Neither requires human judgment during the optimization loop — the system discovers what works by exploring more of the design space than a human engineer could manually traverse.
Challenges
Both evaluations are narrow: specific benchmarks (AutoAgent) or specific production domains (auto-harness). Whether self-optimization generalizes to open-ended agentic tasks — where the fitness landscape is complex and multi-dimensional — is unproven. The "model empathy" finding from AutoAgent is a single observation, not a controlled experiment. And both systems require well-defined evaluation criteria — they optimize what they can measure, which may not align with what matters in unstructured real-world deployment.
Relevant Notes:
- multi-agent coordination delivers value only when three conditions hold simultaneously natural parallelism context overflow and adversarial verification value — self-optimization meets the adversarial verification condition: the meta-agent verifying harness changes differs from the task agent executing them
- 79 percent of multi-agent failures originate from specification and coordination not implementation because decomposition quality is the primary determinant of system success — harness optimization is specification optimization: the meta-agent is iteratively improving how the task is specified to the inner agent
Topics: