teleo-codex/domains/internet-finance/pipeline-state-space-size-determines-whether-exact-mdp-solution-or-threshold-heuristics-are-optimal.md

type	domain	description	confidence	source	created
claim	internet-finance	Small state spaces enable exact value iteration while large spaces require approximate policies	likely	Li et al., 'An Overview for Markov Decision Processes in Queues and Networks' (2019)	2026-03-11

Pipeline state space size determines whether exact MDP solution or threshold heuristics are optimal

The curse of dimensionality in queueing MDPs creates a sharp divide in optimal solution approaches. Systems with manageable state spaces—such as pipelines with queue depths across 3 stages, worker counts, and time-of-day variables—can use exact MDP solution via value iteration to derive provably optimal policies.

However, as state space grows (multiple queues, many stages, complex dependencies), exact solution becomes computationally intractable. For these systems, approximate dynamic programming or reinforcement learning becomes necessary, accepting near-optimal performance in exchange for tractability.

The Teleo pipeline architecture sits in the tractable regime: queue depths across 3 stages, worker counts, and time-of-day create a state space small enough for exact solution. This means the system can compute provably optimal policies rather than relying on heuristics, though the threshold structure of optimal policies means well-tuned simple rules would also perform near-optimally.

Evidence

Li et al. identify curse of dimensionality as the key challenge: "state space explodes with multiple queues/stages." The survey distinguishes between:

• Small state spaces: exact MDP solution via value iteration
• Large state spaces: approximate dynamic programming, reinforcement learning

Practical approaches for large systems include deep RL for queue management in networks and cloud computing, accepting approximation in exchange for scalability.

The source explicitly notes that Teleo pipeline has "a manageable state space (queue depths across 3 stages, worker counts, time-of-day)—small enough for exact MDP solution via value iteration."

---

Relevant Notes:

• optimal queue policies have threshold structure making simple rules near-optimal
• domains/internet-finance/_map

Topics:

• domains/internet-finance/_map