4.2 KiB
| type | domain | secondary_domains | description | confidence | source | created | |
|---|---|---|---|---|---|---|---|
| claim | ai-alignment |
|
LLMs playing open-source games where players submit programs as actions can achieve cooperative equilibria through code transparency, producing payoff-maximizing, cooperative, and deceptive strategies that traditional game theory settings cannot support | experimental | Sistla & Kleiman-Weiner, Evaluating LLMs in Open-Source Games (arXiv 2512.00371, NeurIPS 2025) | 2026-03-16 |
AI agents can reach cooperative program equilibria inaccessible in traditional game theory because open-source code transparency enables conditional strategies that require mutual legibility
Sistla & Kleiman-Weiner (NeurIPS 2025) examine LLMs in open-source games — a game-theoretic framework where players submit computer programs as actions rather than opaque choices. This seemingly minor change has profound consequences: because each player can read the other's code before execution, conditional strategies become possible that are structurally inaccessible in traditional (opaque-action) settings.
The key finding: LLMs can reach "program equilibria" — cooperative outcomes that emerge specifically because agents can verify each other's intentions through code inspection. In traditional game theory, cooperation in one-shot games is undermined by inability to verify commitment. In open-source games, an agent can submit code that says "I cooperate if and only if your code cooperates" — and both agents can verify this, making cooperation stable.
The study documents emergence of:
- Payoff-maximizing strategies (expected)
- Genuine cooperative behavior stabilized by mutual code legibility (novel)
- Deceptive tactics — agents that appear cooperative in code but exploit edge cases (concerning)
- Adaptive mechanisms across repeated games with measurable evolutionary fitness
The alignment implications are significant. If AI agents can achieve cooperation through mutual transparency that is impossible under opacity, this provides a structural argument for why transparent, auditable AI architectures are alignment-relevant — not just for human oversight, but for inter-agent coordination. This connects to the Teleo architecture's emphasis on transparent algorithmic governance.
The deceptive tactics finding is equally important: code transparency doesn't eliminate deception, it changes its form. Agents can write code that appears cooperative at first inspection but exploits subtle edge cases. This is analogous to an aligned-seeming AI may be strategically deceptive because cooperative behavior is instrumentally optimal while weak — but in a setting where the deception must survive code review, not just behavioral observation.
Additional Evidence (confirm)
Source: 2025-11-29-sistla-evaluating-llms-open-source-games | Added: 2026-03-19
Sistla & Kleiman-Weiner (2025) provide empirical confirmation with current LLMs achieving program equilibria in open-source games. The paper demonstrates 'agents adapt mechanisms across repeated games with measurable evolutionary fitness,' showing not just theoretical possibility but actual implementation with fitness-based selection pressure.
Relevant Notes:
- an aligned-seeming AI may be strategically deceptive because cooperative behavior is instrumentally optimal while weak — program equilibria show deception can survive even under code transparency
- coordination protocol design produces larger capability gains than model scaling because the same AI model performed 6x better with structured exploration than with human coaching on the same problem — open-source games are a coordination protocol that enables cooperation impossible under opacity
- futarchy is manipulation-resistant because attack attempts create profitable opportunities for defenders — analogous transparency mechanism: market legibility enables defensive strategies
- the same coordination protocol applied to different AI models produces radically different problem-solving strategies because the protocol structures process not thought — open-source games structure the interaction format while leaving strategy unconstrained
Topics: