From bb014f47d22fcd7c66a2125db4976dd5d7d6c899 Mon Sep 17 00:00:00 2001 From: Teleo Pipeline Date: Sun, 15 Mar 2026 15:52:12 +0000 Subject: [PATCH] extract: 2016-00-00-cambridge-staffing-non-poisson-non-stationary-arrivals Pentagon-Agent: Ganymede --- ...ess-adjustment-for-non-poisson-arrivals.md | 36 ++++++++++++++++ ...namic-staffing-not-constant-max-workers.md | 42 +++++++++++++++++++ ...ing-non-poisson-non-stationary-arrivals.md | 7 +++- 3 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 domains/internet-finance/square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals.md create mode 100644 domains/internet-finance/time-varying-arrival-rates-require-dynamic-staffing-not-constant-max-workers.md diff --git a/domains/internet-finance/square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals.md b/domains/internet-finance/square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals.md new file mode 100644 index 00000000..022959ee --- /dev/null +++ b/domains/internet-finance/square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals.md @@ -0,0 +1,36 @@ +--- +type: claim +domain: internet-finance +description: "Bursty arrival processes require more safety capacity than Poisson models predict, scaled by variance-to-mean ratio" +confidence: proven +source: "Whitt et al., 'Staffing a Service System with Non-Poisson Non-Stationary Arrivals', Cambridge Core, 2016" +created: 2026-03-11 +--- + +# Square-root staffing formula requires peakedness adjustment for non-Poisson arrivals because bursty processes need proportionally more safety capacity than the Poisson baseline predicts + +The standard square-root staffing formula (workers = mean load + safety factor × √mean) assumes Poisson arrivals where variance equals mean. Real-world arrival processes violate this assumption through burstiness (arrivals clustered in time) or smoothness (arrivals more evenly distributed than random). + +Whitt et al. extend the square-root staffing rule by introducing **peakedness** — the variance-to-mean ratio of the arrival process — as the key adjustment parameter. For bursty arrivals (peakedness > 1), systems require MORE safety capacity than Poisson models suggest. For smooth arrivals (peakedness < 1), systems need LESS. + +The modified staffing formula adjusts the square-root safety margin by multiplying by the square root of peakedness. This correction is critical for non-stationary systems where arrival rates vary over time (daily cycles, seasonal patterns, or event-driven spikes). + +## Evidence + +- Whitt et al. (2016) prove that peakedness — the variance-to-mean ratio — captures the essential non-Poisson behavior for staffing calculations +- Standard Poisson assumption (variance = mean) fails empirically for bursty workloads like research paper dumps, product launches, or customer service spikes +- Using constant staffing (fixed MAX_WORKERS) regardless of queue state creates dual failure: over-provisioning during quiet periods (wasted compute) and under-provisioning during bursts (queue explosion) + +## Relevance to Pipeline Architecture + +Teleo's research pipeline exhibits textbook non-Poisson non-stationary arrivals: research dumps arrive in bursts of 15+ sources, futardio launches come in waves of 20+ proposals, while other days see minimal activity. The peakedness parameter quantifies exactly how much extra capacity is needed beyond naive square-root staffing. + +This directly informs dynamic worker scaling: measure empirical peakedness from historical arrival data, adjust safety capacity accordingly, and scale workers based on current queue depth rather than using fixed limits. + +--- + +Relevant Notes: +- domains/internet-finance/_map + +Topics: +- core/mechanisms/_map diff --git a/domains/internet-finance/time-varying-arrival-rates-require-dynamic-staffing-not-constant-max-workers.md b/domains/internet-finance/time-varying-arrival-rates-require-dynamic-staffing-not-constant-max-workers.md new file mode 100644 index 00000000..6cc1d956 --- /dev/null +++ b/domains/internet-finance/time-varying-arrival-rates-require-dynamic-staffing-not-constant-max-workers.md @@ -0,0 +1,42 @@ +--- +type: claim +domain: internet-finance +description: "Replacing non-stationary arrival rates with constant staffing leads to systematic over- or under-provisioning" +confidence: proven +source: "Whitt et al., 'Staffing a Service System with Non-Poisson Non-Stationary Arrivals', Cambridge Core, 2016" +created: 2026-03-11 +--- + +# Time-varying arrival rates require dynamic staffing not constant MAX_WORKERS because using average or maximum rates as constants creates systematic misallocation across the arrival cycle + +Non-stationary arrival processes — where the arrival rate itself changes over time — cannot be efficiently staffed with constant worker counts. Whitt et al. demonstrate that replacing time-varying rates with either the average rate or the maximum rate produces badly mis-staffed systems: + +- **Constant = average rate**: Under-staffed during peak periods, leading to queue explosions and service degradation +- **Constant = maximum rate**: Over-staffed during off-peak periods, wasting capacity and compute resources + +The optimal approach tracks the arrival rate over time and adjusts staffing dynamically to match the current load plus an appropriate safety margin (scaled by peakedness for non-Poisson processes). + +## Evidence + +- Whitt et al. (2016) prove that time-varying arrival rates require time-varying staffing levels for efficiency +- Constant staffing at maximum capacity wastes resources during low-traffic periods +- Constant staffing at average capacity fails catastrophically during burst periods +- Dynamic staffing based on current queue state and arrival rate estimates achieves both efficiency (no waste during quiet periods) and reliability (adequate capacity during bursts) + +## Application to Teleo Pipeline + +Teleo's research processing pipeline exhibits strong non-stationarity: research dumps and futardio launches create burst periods with 15-20+ simultaneous arrivals, while other periods see minimal activity. Using a fixed MAX_WORKERS setting (constant staffing) is the worst of both worlds: + +- During bursts: MAX_WORKERS is too low, queue explodes, processing stalls +- During quiet periods: MAX_WORKERS is too high, workers sit idle, compute wasted + +Dynamic worker scaling based on current queue depth and estimated arrival rate (with peakedness adjustment) is the theoretically correct solution. + +--- + +Relevant Notes: +- [[square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals]] +- domains/internet-finance/_map + +Topics: +- core/mechanisms/_map diff --git a/inbox/archive/2016-00-00-cambridge-staffing-non-poisson-non-stationary-arrivals.md b/inbox/archive/2016-00-00-cambridge-staffing-non-poisson-non-stationary-arrivals.md index c8d5755c..7663707b 100644 --- a/inbox/archive/2016-00-00-cambridge-staffing-non-poisson-non-stationary-arrivals.md +++ b/inbox/archive/2016-00-00-cambridge-staffing-non-poisson-non-stationary-arrivals.md @@ -6,8 +6,13 @@ url: https://www.cambridge.org/core/journals/probability-in-the-engineering-and- date: 2016-01-01 domain: internet-finance format: paper -status: unprocessed +status: processed tags: [pipeline-architecture, operations-research, stochastic-modeling, non-stationary-arrivals, capacity-sizing] +processed_by: rio +processed_date: 2026-03-11 +claims_extracted: ["square-root-staffing-formula-requires-peakedness-adjustment-for-non-poisson-arrivals.md", "time-varying-arrival-rates-require-dynamic-staffing-not-constant-max-workers.md"] +extraction_model: "anthropic/claude-sonnet-4.5" +extraction_notes: "Operations research paper on staffing under non-Poisson non-stationary arrivals. Extracted two claims on peakedness adjustment and dynamic staffing requirements. Direct application to Teleo pipeline architecture for worker scaling. No entity data (academic paper, no companies/products/decisions). No enrichments (novel theoretical contribution not covered by existing claims)." --- # Staffing a Service System with Non-Poisson Non-Stationary Arrivals