3.1 KiB
| type | domain | description | confidence | source | created |
|---|---|---|---|---|---|
| claim | internet-finance | Flow control mechanism that signals producers to slow down when consumers reach capacity limits | proven | Dagster, What Is Backpressure glossary entry, 2024 | 2026-03-11 |
Backpressure prevents pipeline failure by creating feedback loop between consumer capacity and producer rate
Backpressure is a flow control mechanism where data consumers signal producers about their capacity limits, preventing system overload. Without backpressure controls, pipelines experience data loss, crashes, and resource exhaustion when producers overwhelm consumers.
The mechanism operates through several implementation strategies:
- Buffering with threshold triggers — queues that signal when capacity approaches limits
- Rate limiting — explicit caps on production speed
- Dynamic adjustment — real-time scaling based on consumer state
- Acknowledgment-based flow — producers wait for consumer confirmation before sending more data
Major distributed systems implement backpressure as core architecture: Apache Kafka uses pull-based consumption where consumers control their own rate, while Flink, Spark Streaming, Akka Streams, and Project Reactor all build backpressure into their execution models.
The tradeoff is explicit: backpressure introduces latency (producers must wait for consumer signals) but prevents catastrophic failure modes. This makes backpressure a design-time decision, not a retrofit — systems must incorporate feedback channels from the start.
Evidence
- Dagster documentation identifies backpressure as standard pattern across Apache Kafka, Flink, Spark Streaming, Akka Streams, Project Reactor
- Implementation strategies documented: buffering, rate limiting, dynamic adjustment, acknowledgment-based flow
- Failure modes without backpressure: data loss, crashes, resource exhaustion
Relevance to Teleo
The Teleo pipeline currently has zero backpressure. The extract-cron.sh dispatcher checks for unprocessed sources and launches workers without checking eval queue state. If extraction outruns evaluation, PRs accumulate with no feedback signal to slow extraction.
Simple implementation: extraction dispatcher should check open PR count before dispatching. If open PRs exceed threshold, reduce extraction parallelism or skip the cycle entirely. This creates the feedback loop that prevents eval queue overload.
Additional Evidence (extend)
Source: 2025-12-00-javacodegeeks-reactive-programming-backpressure-stream-processing | Added: 2026-03-16
Reactive Streams specification implements backpressure through Publisher/Subscriber/Subscription interfaces where Subscriber requests N items and Publisher delivers at most N, creating demand-based flow control. Four standard strategies exist: Buffer (accumulate with threshold triggers, risk unbounded memory), Drop (discard excess), Latest (keep only most recent), and Error (signal failure on overflow). Key architectural insight: backpressure must be designed into systems from the start—retrofitting it is much harder.
Relevant Notes:
- domains/internet-finance/_map
Topics:
- core/mechanisms/_map