teleo-codex/inbox/archive/2021-04-00-tournaire-optimal-control-cloud-resource-allocation-mdp.md

type	title	author	url	date	domain	format	status	tags	processed_by	processed_date	claims_extracted	extraction_model	extraction_notes
source	Optimal Control Policies for Resource Allocation in the Cloud: Comparison Between Markov Decision Process and Heuristic Approaches	Thomas Tournaire, Hind Castel-Taleb, Emmanuel Hyon	https://arxiv.org/abs/2104.14879	2021-04-30	internet-finance	paper	processed	pipeline-architecture operations-research markov-decision-process cloud-autoscaling optimal-control	rio	2026-03-11	mdp-based-autoscaling-with-hysteresis-outperforms-simple-threshold-heuristics-for-cloud-resource-allocation.md hysteresis-in-autoscaling-prevents-oscillation-by-using-asymmetric-thresholds-for-scale-up-and-scale-down.md	anthropic/claude-sonnet-4.5	Extracted two claims about MDP-based autoscaling and hysteresis principles. The source is an operations research paper on cloud resource allocation that maps directly to pipeline worker management. The hysteresis insight is particularly valuable for preventing worker thrashing. No entities to create (academic paper, no companies/products/markets). No enrichments to existing claims (this is a novel technical domain for the KB).

Optimal Control Policies for Resource Allocation in the Cloud

Compares MDP-based optimal scaling policies against heuristic approaches for cloud auto-scaling. The MDP formulation treats VM provisioning as a sequential decision problem.

Key Content

• Auto-scaling problem: VMs turned on/off based on queue occupation to minimize combined energy + performance cost
• MDP formulation: states = queue lengths + active VMs, actions = add/remove VMs, rewards = negative cost (energy + SLA violations)
• Value iteration and policy iteration algorithms find optimal threshold policies
• Structured MDP algorithms incorporating hysteresis properties outperform heuristics in both execution time and accuracy
• Hysteresis: different thresholds for scaling up vs. scaling down — prevents oscillation (e.g., scale up at queue=10, scale down at queue=3)
• MDP algorithms find optimal hysteresis thresholds automatically

Relevance to Teleo Pipeline

The MDP formulation maps directly: states = (unprocessed queue, in-flight extractions, open PRs, active workers), actions = (spawn worker, kill worker, wait), cost = (Claude compute cost per worker-minute + delay cost per queued source). The hysteresis insight is particularly valuable — we should have different thresholds for spinning up vs. spinning down workers to prevent oscillation.

Key finding: structured MDP with hysteresis outperforms simple threshold heuristics. But even simple threshold policies (scale up at queue=N, scale down at queue=M where M < N) perform reasonably well.

Key Facts

• MDP formulation for cloud autoscaling: states = queue lengths + active VMs, actions = add/remove VMs, rewards = negative cost (energy + SLA violations)
• Value iteration and policy iteration algorithms used to find optimal threshold policies
• Example hysteresis thresholds: scale up at queue=10, scale down at queue=3