---
type: source
title: "Optimal Control Policies for Resource Allocation in the Cloud: Comparison Between Markov Decision Process and Heuristic Approaches"
author: "Thomas Tournaire, Hind Castel-Taleb, Emmanuel Hyon"
url: https://arxiv.org/abs/2104.14879
date: 2021-04-30
domain: internet-finance
format: paper
status: processed
tags: [pipeline-architecture, operations-research, markov-decision-process, cloud-autoscaling, optimal-control]
processed_by: rio
processed_date: 2026-03-11
claims_extracted: ["mdp-based-autoscaling-with-hysteresis-outperforms-simple-threshold-heuristics-for-cloud-resource-allocation.md", "hysteresis-in-autoscaling-prevents-oscillation-by-using-asymmetric-thresholds-for-scale-up-and-scale-down.md"]
extraction_model: "anthropic/claude-sonnet-4.5"
extraction_notes: "Extracted two claims about MDP-based autoscaling and hysteresis principles. The source is an operations research paper on cloud resource allocation that maps directly to pipeline worker management. The hysteresis insight is particularly valuable for preventing worker thrashing. No entities to create (academic paper, no companies/products/markets). No enrichments to existing claims (this is a novel technical domain for the KB)."
---

# Optimal Control Policies for Resource Allocation in the Cloud

Compares MDP-based optimal scaling policies against heuristic approaches for cloud auto-scaling. The MDP formulation treats VM provisioning as a sequential decision problem.

## Key Content

- Auto-scaling problem: VMs turned on/off based on queue occupation to minimize combined energy + performance cost
- MDP formulation: states = queue lengths + active VMs, actions = add/remove VMs, rewards = negative cost (energy + SLA violations)
- Value iteration and policy iteration algorithms find optimal threshold policies
- Structured MDP algorithms incorporating hysteresis properties outperform heuristics in both execution time and accuracy
- Hysteresis: different thresholds for scaling up vs. scaling down — prevents oscillation (e.g., scale up at queue=10, scale down at queue=3)
- MDP algorithms find optimal hysteresis thresholds automatically

## Relevance to Teleo Pipeline

The MDP formulation maps directly: states = (unprocessed queue, in-flight extractions, open PRs, active workers), actions = (spawn worker, kill worker, wait), cost = (Claude compute cost per worker-minute + delay cost per queued source). The hysteresis insight is particularly valuable — we should have different thresholds for spinning up vs. spinning down workers to prevent oscillation.

Key finding: structured MDP with hysteresis outperforms simple threshold heuristics. But even simple threshold policies (scale up at queue=N, scale down at queue=M where M < N) perform reasonably well.


## Key Facts
- MDP formulation for cloud autoscaling: states = queue lengths + active VMs, actions = add/remove VMs, rewards = negative cost (energy + SLA violations)
- Value iteration and policy iteration algorithms used to find optimal threshold policies
- Example hysteresis thresholds: scale up at queue=10, scale down at queue=3
