---
type: claim
domain: internet-finance
description: "Classic MDP results show admission and service policies converge on simple threshold rules rather than complex state-dependent strategies"
confidence: proven
source: "Li et al., 'An Overview for Markov Decision Processes in Queues and Networks' (2019), surveying 60+ years of queueing MDP research"
created: 2026-03-11
---

# Optimal queue control policies have threshold structure for single-server systems

For single-server queueing systems modeled as continuous-time MDPs, optimal admission and service policies typically take the form of simple threshold rules: "serve if queue > K, idle if queue < K" or "admit if queue < N, reject if queue ≥ N." This threshold structure emerges from dynamic programming solutions across a wide range of cost functions and arrival/service distributions.

The threshold property means that despite the theoretical complexity of MDP state spaces, optimal policies can be characterized by a small number of parameters. This has two practical implications:

1. **Policy search simplification**: Instead of solving the full MDP via value iteration across all states, practitioners can search over threshold parameters, reducing computational complexity from exponential to linear in state space size.

2. **Near-optimality of simple heuristics**: Well-tuned threshold policies achieve near-optimal performance even when derived heuristically rather than through formal optimization, because the structure of the optimal solution is known in advance.

For multi-server systems, the threshold structure extends to policies like "join-shortest-queue" with queue-length-dependent routing thresholds. The survey covers applications from the 1960s (admission control in telephone networks) through modern cloud computing resource allocation.

## Evidence

Li et al. (2019) provide a comprehensive 42-page survey of MDP applications in queueing theory, synthesizing results from 60+ years of operations research. The paper documents that across diverse queueing models—M/M/1, M/G/1, networks of queues—optimal policies derived through dynamic programming consistently exhibit threshold structure when the objective is cost minimization or throughput maximization.

Key theoretical results cited:
- **Monotone policies**: When the cost function is convex in queue length, optimal policies are monotone increasing (serve more aggressively as queue grows)
- **Structural results**: For admission control with holding costs, the optimal policy is a threshold on queue length
- **Multi-server extensions**: Join-shortest-queue is optimal for homogeneous servers; threshold-based routing is optimal for heterogeneous servers

The survey notes that these structural results break down in two cases:
1. **High-dimensional state spaces**: Networks with many queues suffer curse of dimensionality, requiring approximate methods
2. **Non-convex costs**: When costs have discontinuities or non-monotone structure, threshold policies may be suboptimal

## Relevance to Teleo Pipeline

The Teleo pipeline architecture has a manageable state space: queue depths across 3 stages (inbox, analysis, synthesis), worker counts per stage, and time-of-day context. This falls well within the regime where exact MDP solutions are tractable via value iteration.

The threshold structure result means that even without solving the full MDP, a heuristic policy of the form "if inbox_queue > X and active_workers < Y, spawn worker" will be near-optimal if X and Y are well-tuned. The survey confirms this approach is theoretically grounded for systems of this scale.

For future work: if the pipeline scales to 10+ stages or incorporates complex dependencies between stages, approximate dynamic programming or reinforcement learning methods (covered in the survey's final section) become necessary.

---

Relevant Notes:
- core/mechanisms/_map

Topics:
- domains/internet-finance/_map