51a2ed39fc
Some checks are pending
Sync Graph Data to teleo-app / sync (push) Waiting to run
Pentagon-Agent: Ganymede <F99EBFA6-547B-4096-BEEA-1D59C3E4028A>
44 lines
3 KiB
Markdown
44 lines
3 KiB
Markdown
---
|
|
type: source
|
|
title: "An Overview for Markov Decision Processes in Queues and Networks"
|
|
author: "Quan-Lin Li, Jing-Yu Ma, Rui-Na Fan, Li Xia"
|
|
url: https://arxiv.org/abs/1907.10243
|
|
date: 2019-07-24
|
|
domain: internet-finance
|
|
format: paper
|
|
status: processed
|
|
tags: [pipeline-architecture, operations-research, markov-decision-process, queueing-theory, dynamic-programming]
|
|
processed_by: rio
|
|
processed_date: 2026-03-11
|
|
claims_extracted: ["optimal-queue-policies-have-threshold-structure-making-simple-rules-near-optimal.md", "pipeline-state-space-size-determines-whether-exact-mdp-solution-or-threshold-heuristics-are-optimal.md"]
|
|
extraction_model: "anthropic/claude-sonnet-4.5"
|
|
extraction_notes: "Academic survey of MDP applications to queueing theory. Extracted two claims about optimal policy structure and state space tractability. No entities (academic paper, no companies/products). No enrichments (claims are foundational operations research results, not directly connected to existing futarchy/capital formation claims in KB)."
|
|
---
|
|
|
|
# An Overview for Markov Decision Processes in Queues and Networks
|
|
|
|
Comprehensive 42-page survey of MDP applications in queueing systems, covering 60+ years of research from the 1960s to present.
|
|
|
|
## Key Content
|
|
|
|
- Continuous-time MDPs for queue management: decisions happen at state transitions (arrivals, departures)
|
|
- Classic results: optimal policies often have threshold structure — "serve if queue > K, idle if queue < K"
|
|
- For multi-server systems: optimal admission and routing policies are often simple (join-shortest-queue, threshold-based)
|
|
- Dynamic programming and stochastic optimization provide tools for deriving optimal policies
|
|
- Key challenge: curse of dimensionality — state space explodes with multiple queues/stages
|
|
- Practical approaches: approximate dynamic programming, reinforcement learning for large state spaces
|
|
- Emerging direction: deep RL for queue management in networks and cloud computing
|
|
|
|
## Relevance to Teleo Pipeline
|
|
|
|
Our pipeline has a manageable state space (queue depths across 3 stages, worker counts, time-of-day) — small enough for exact MDP solution via value iteration. The survey confirms that optimal policies for our type of system typically have threshold structure: "if queue > X and workers < Y, spawn a worker." This means even without solving the full MDP, a well-tuned threshold policy will be near-optimal.
|
|
|
|
|
|
## Key Facts
|
|
- Li et al. survey covers 60+ years of MDP research in queueing systems (1960s-2019)
|
|
- Continuous-time MDPs for queues: decisions happen at state transitions (arrivals, departures)
|
|
- Classic optimal policies: threshold structure (serve if queue > K, idle if queue < K)
|
|
- Multi-server optimal policies: join-shortest-queue, threshold-based admission
|
|
- Key challenge: curse of dimensionality with multiple queues/stages
|
|
- Practical approaches: approximate dynamic programming, reinforcement learning for large state spaces
|
|
- Emerging direction: deep RL for queue management in networks and cloud computing
|