teleo-codex/inbox/archive/2019-07-00-li-overview-mdp-queues-networks.md

type	title	author	url	date	domain	format	status	tags	processed_by	processed_date	claims_extracted	extraction_model	extraction_notes
source	An Overview for Markov Decision Processes in Queues and Networks	Quan-Lin Li, Jing-Yu Ma, Rui-Na Fan, Li Xia	https://arxiv.org/abs/1907.10243	2019-07-24	internet-finance	paper	processed	pipeline-architecture operations-research markov-decision-process queueing-theory dynamic-programming	rio	2026-03-11	optimal-queue-control-policies-have-threshold-structure-for-single-server-systems.md curse-of-dimensionality-makes-exact-mdp-solutions-intractable-for-multi-queue-networks.md deep-rl-for-queue-management-is-emerging-direction-for-cloud-and-network-systems.md	anthropic/claude-sonnet-4.5	Academic survey paper establishing theoretical foundations for queue control via MDPs. Three claims extracted covering: (1) threshold structure of optimal policies for tractable systems, (2) curse of dimensionality as the barrier to scaling, (3) deep RL as emerging solution for large-scale systems. No entity data or enrichments—purely theoretical/methodological content. Relevance to Teleo: confirms that pipeline's state space (~10^3-10^4 states) is in the tractable regime for exact MDP solution, and that threshold policies are theoretically grounded for this scale.

An Overview for Markov Decision Processes in Queues and Networks

Comprehensive 42-page survey of MDP applications in queueing systems, covering 60+ years of research from the 1960s to present.

Key Content

• Continuous-time MDPs for queue management: decisions happen at state transitions (arrivals, departures)
• Classic results: optimal policies often have threshold structure — "serve if queue > K, idle if queue < K"
• For multi-server systems: optimal admission and routing policies are often simple (join-shortest-queue, threshold-based)
• Dynamic programming and stochastic optimization provide tools for deriving optimal policies
• Key challenge: curse of dimensionality — state space explodes with multiple queues/stages
• Practical approaches: approximate dynamic programming, reinforcement learning for large state spaces
• Emerging direction: deep RL for queue management in networks and cloud computing

Relevance to Teleo Pipeline

Our pipeline has a manageable state space (queue depths across 3 stages, worker counts, time-of-day) — small enough for exact MDP solution via value iteration. The survey confirms that optimal policies for our type of system typically have threshold structure: "if queue > X and workers < Y, spawn a worker." This means even without solving the full MDP, a well-tuned threshold policy will be near-optimal.

Key Facts

• Survey covers 60+ years of MDP research in queueing theory (1960s-2019)
• Continuous-time MDPs for queues: decisions at state transitions (arrivals, departures)
• Dynamic programming and stochastic optimization are core solution methods
• Approximate dynamic programming and RL are practical approaches for large state spaces