40 lines
2.5 KiB
Markdown
40 lines
2.5 KiB
Markdown
---
|
|
type: source
|
|
title: "What Is Backpressure"
|
|
author: "Dagster"
|
|
url: https://dagster.io/glossary/data-backpressure
|
|
date: 2024-01-01
|
|
domain: internet-finance
|
|
format: essay
|
|
status: processed
|
|
tags: [pipeline-architecture, backpressure, data-pipelines, flow-control]
|
|
processed_by: rio
|
|
processed_date: 2026-03-11
|
|
claims_extracted: ["backpressure-prevents-pipeline-failure-by-signaling-consumer-capacity-limits-to-producers.md", "extraction-without-backpressure-creates-unbounded-pr-accumulation-when-extraction-outruns-evaluation.md"]
|
|
extraction_model: "anthropic/claude-sonnet-4.5"
|
|
extraction_notes: "Extracted two claims: one general claim about backpressure as a proven pattern in data systems, one experimental claim about Teleo pipeline architecture. The source is a practitioner glossary entry, not academic research, but describes widely-deployed production patterns. Second claim applies backpressure concept to Teleo's own pipeline based on curator's relevance note. No entities to extract — this is architectural pattern documentation, not company/product/market data."
|
|
---
|
|
|
|
# What Is Backpressure (Dagster)
|
|
|
|
Dagster's practical guide to backpressure in data pipelines. Written for practitioners building real data processing systems.
|
|
|
|
## Key Content
|
|
|
|
- Backpressure: feedback mechanism preventing data producers from overwhelming consumers
|
|
- Without backpressure controls: data loss, crashes, resource exhaustion
|
|
- Consumer signals producer about capacity limits
|
|
- Implementation strategies: buffering (with threshold triggers), rate limiting, dynamic adjustment, acknowledgment-based flow
|
|
- Systems using backpressure: Apache Kafka (pull-based consumption), Flink, Spark Streaming, Akka Streams, Project Reactor
|
|
- Tradeoff: backpressure introduces latency but prevents catastrophic failure
|
|
- Key principle: design backpressure into the system from the start
|
|
|
|
## Relevance to Teleo Pipeline
|
|
|
|
Our pipeline has zero backpressure today. The extract-cron.sh checks for unprocessed sources and dispatches workers regardless of eval queue state. If extraction outruns evaluation, PRs accumulate with no feedback signal. Simple fix: extraction dispatcher should check open PR count before dispatching. If open PRs > threshold, reduce extraction parallelism or skip the cycle.
|
|
|
|
|
|
## Key Facts
|
|
- Apache Kafka implements backpressure through pull-based consumption model
|
|
- Flink, Spark Streaming, Akka Streams, Project Reactor all use backpressure as core pattern
|
|
- Backpressure implementation strategies: buffering with thresholds, rate limiting, dynamic adjustment, acknowledgment-based flow
|