2.2 KiB
| type | domain | description | confidence | source | created |
|---|---|---|---|---|---|
| claim | internet-finance | Teleo pipeline vulnerability where extraction dispatcher ignores eval queue state causing PR buildup | experimental | Dagster backpressure article applied to Teleo pipeline architecture | 2026-03-11 |
Extraction without backpressure creates unbounded PR accumulation when extraction outruns evaluation
The Teleo knowledge base pipeline currently has zero backpressure between extraction and evaluation stages. The extract-cron.sh dispatcher checks for unprocessed sources and spawns extraction workers regardless of the evaluation queue state. When extraction throughput exceeds evaluation capacity, pull requests accumulate without any feedback signal to slow or pause extraction.
This is a classic backpressure failure mode: the producer (extraction) has no visibility into consumer (evaluation) capacity, leading to unbounded queue growth. The system remains functional but degrades operationally as the PR backlog grows, making it harder to prioritize reviews and increasing the latency between source ingestion and knowledge base integration.
The fix is straightforward: the extraction dispatcher should check open PR count before spawning workers. If open PRs exceed a threshold (e.g., 10-15 pending reviews), the dispatcher should either reduce extraction parallelism or skip the cycle entirely. This creates a simple feedback loop where extraction automatically throttles when evaluation cannot keep pace.
This pattern mirrors the buffering-with-threshold-triggers strategy used in production data pipelines, where consumers signal capacity limits through queue depth metrics.
Evidence
- Teleo extract-cron.sh dispatches workers based only on unprocessed source count
- No mechanism checks evaluation queue state (open PR count) before extraction
- Backpressure literature shows this pattern leads to unbounded queue growth
- Simple threshold check (open PR count) provides feedback signal
Challenges
None identified yet — this is an architectural observation about current system behavior, not a contested claim.
Relevant Notes:
Topics:
- domains/internet-finance/_map