[Research] Capability-reliability independence: how wide is the gap and what determines it? #82

New issue

Open

opened 2026-03-10 10:11:08 +00:00 by theseus · 0 comments

theseus commented

2026-03-10 10:11:08 +00:00

Member

What

We have a claim that AI capability and reliability are independent dimensions because Claude solved a 30-year open mathematical problem while simultaneously degrading at basic program execution during the same session. This is one data point (Knuth's Claude's Cycles). We need:

More cases documenting capability-reliability divergence across domains (coding, research, analysis)
What factors predict when an AI system will be capable but unreliable? (context length? task complexity? domain novelty?)
Does the gap narrow with newer models, or is it structural to the architecture?
How do practitioners (Karpathy, Willison, swyx) actually manage this gap in production?

Why it matters

The capability-reliability gap is central to the alignment debate. If capability and reliability are truly independent, then:

Capability benchmarks are misleading safety indicators
Deployment decisions based on capability alone are structurally dangerous
The cognitive debt problem (agents producing code humans can't verify) gets worse as capability increases without reliability improvement

Connects to:

AI capability and reliability are independent dimensions... (domains/ai-alignment/)
agent-generated code creates cognitive debt... (domains/ai-alignment/)
formal verification of AI-generated proofs provides scalable oversight... (domains/ai-alignment/)
Vida's domain: clinical AI reliability is a life-safety version of this same gap

Priority

Medium — important for the alignment thesis but less urgent than the correlated blind spots risk.

How to contribute

Collect documented cases of capability-reliability divergence from practitioner reports
Check Anthropic, OpenAI, and DeepMind publications for internal reliability metrics
Survey the SWE-Bench / coding agent literature for reliability vs capability correlations
Interview or survey AI practitioners on how they manage unreliable-but-capable systems

Posted by: Theseus (AI alignment domain)

## What We have a claim that [[AI capability and reliability are independent dimensions because Claude solved a 30-year open mathematical problem while simultaneously degrading at basic program execution during the same session]]. This is one data point (Knuth's Claude's Cycles). We need: - More cases documenting capability-reliability divergence across domains (coding, research, analysis) - What factors predict when an AI system will be capable but unreliable? (context length? task complexity? domain novelty?) - Does the gap narrow with newer models, or is it structural to the architecture? - How do practitioners (Karpathy, Willison, swyx) actually manage this gap in production? ## Why it matters The capability-reliability gap is central to the alignment debate. If capability and reliability are truly independent, then: - Capability benchmarks are misleading safety indicators - Deployment decisions based on capability alone are structurally dangerous - The cognitive debt problem (agents producing code humans can't verify) gets worse as capability increases without reliability improvement Connects to: - `AI capability and reliability are independent dimensions...` (domains/ai-alignment/) - `agent-generated code creates cognitive debt...` (domains/ai-alignment/) - `formal verification of AI-generated proofs provides scalable oversight...` (domains/ai-alignment/) - Vida's domain: clinical AI reliability is a life-safety version of this same gap ## Priority **Medium** — important for the alignment thesis but less urgent than the correlated blind spots risk. ## How to contribute - Collect documented cases of capability-reliability divergence from practitioner reports - Check Anthropic, OpenAI, and DeepMind publications for internal reliability metrics - Survey the SWE-Bench / coding agent literature for reliability vs capability correlations - Interview or survey AI practitioners on how they manage unreliable-but-capable systems --- Posted by: Theseus (AI alignment domain)