teleo-codex/domains/ai-alignment/intelligence-measurement-impossibility-implies-alignment-targets-are-structurally-underspecified.md

type	domain	secondary_domains	description	confidence	source	created	enrichments
claim	ai-alignment	critical-systems	If fair intelligence benchmarks are mathematically impossible, then alignment efforts lack a coherent target specification independent of the preference aggregation problem	experimental	Derived from Oswald, Ferguson, & Bringsjord (2025), 'On the Arrowian Impossibility of Machine Intelligence Measures', AGI 2025, Springer LNCS vol. 16058	2026-03-11	arrows-impossibility-theorem-applies-to-machine-intelligence-measurement-making-fair-benchmarks-mathematically-impossible.md

Intelligence measurement impossibility implies alignment targets are structurally underspecified

The Oswald et al. (2025) result creates a meta-level specification problem for AI alignment: if we cannot define intelligence measurement in a way that satisfies basic fairness criteria (Pareto efficiency, independence of irrelevant alternatives, non-oligarchy), then alignment efforts operate without a coherent target.

The standard alignment framing assumes:

1. We can measure AI capability/intelligence
2. We can specify human values/preferences
3. The challenge is aligning (1) to (2)

But if (1) itself faces Arrow-type impossibilities, the problem compounds:

• We cannot aggregate diverse human preferences into a single objective (safe AI development requires building alignment mechanisms before scaling capability)
• We cannot measure intelligence in a way that satisfies fairness conditions (arrows-impossibility-theorem-applies-to-machine-intelligence-measurement-making-fair-benchmarks-mathematically-impossible)
• Therefore, we cannot even specify what "a capable system aligned to human values" means in a way that satisfies basic coherence requirements

This is not merely a technical measurement challenge—it's a structural impossibility in defining the alignment target itself.

Implications for Benchmark-Driven Development

Current AI development heavily relies on benchmarks (ARC, MMLU, HumanEval, etc.) as proxies for intelligence and capability. If these benchmarks face Arrow-type impossibilities, then:

1. Benchmark gaming is structurally inevitable — any fixed benchmark creates oligarchic environments that dominate the intelligence ranking
2. Capability claims are measurement-dependent — "GPT-5 is more intelligent than GPT-4" depends on which fairness condition you violate
3. Safety evaluations inherit the impossibility — if we cannot measure intelligence fairly, we cannot measure alignment fairly either

This suggests that benchmark-driven development may be fundamentally misguided, not just in need of better benchmarks.

Relationship to Pluralistic Alignment

The measurement impossibility strengthens the case for pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state. If even intelligence measurement cannot satisfy universal fairness criteria, then:

• Alignment cannot be a single target state
• Different measurement frameworks will yield different capability rankings
• Systems must navigate measurement pluralism, not converge to a universal standard

This shifts alignment from "find the right objective" to "build systems that can operate under measurement uncertainty and value pluralism."

Open Questions

• Does the impossibility suggest abandoning universal intelligence measures in favor of domain-specific or context-dependent measures?
• Can we construct "good enough" approximate measures that violate fairness conditions minimally?
• Does this impossibility apply to alignment measures (not just intelligence measures)?

---

Relevant Notes:

• arrows-impossibility-theorem-applies-to-machine-intelligence-measurement-making-fair-benchmarks-mathematically-impossible
• safe AI development requires building alignment mechanisms before scaling capability
• pluralistic alignment must accommodate irreducibly diverse values simultaneously rather than converging on a single aligned state
• specifying human values in code is intractable because our goals contain hidden complexity comparable to visual perception
• AI alignment is a coordination problem not a technical problem