teleo-codex/inbox/archive/2024-00-00-dagster-data-backpressure.md

type	title	author	url	date	domain	format	status	tags	processed_by	processed_date	claims_extracted	extraction_model	extraction_notes
source	What Is Backpressure	Dagster	https://dagster.io/glossary/data-backpressure	2024-01-01	internet-finance	essay	processed	pipeline-architecture backpressure data-pipelines flow-control	rio	2026-03-11	backpressure-prevents-pipeline-failure-by-signaling-consumer-capacity-limits-to-producers.md extraction-without-backpressure-creates-unbounded-pr-accumulation-when-extraction-outruns-evaluation.md	anthropic/claude-sonnet-4.5	Extracted two claims: one general claim about backpressure as a proven pattern in data systems, one experimental claim about Teleo pipeline architecture. The source is a practitioner glossary entry, not academic research, but describes widely-deployed production patterns. Second claim applies backpressure concept to Teleo's own pipeline based on curator's relevance note. No entities to extract — this is architectural pattern documentation, not company/product/market data.

What Is Backpressure (Dagster)

Dagster's practical guide to backpressure in data pipelines. Written for practitioners building real data processing systems.

Key Content

• Backpressure: feedback mechanism preventing data producers from overwhelming consumers
• Without backpressure controls: data loss, crashes, resource exhaustion
• Consumer signals producer about capacity limits
• Implementation strategies: buffering (with threshold triggers), rate limiting, dynamic adjustment, acknowledgment-based flow
• Systems using backpressure: Apache Kafka (pull-based consumption), Flink, Spark Streaming, Akka Streams, Project Reactor
• Tradeoff: backpressure introduces latency but prevents catastrophic failure
• Key principle: design backpressure into the system from the start

Relevance to Teleo Pipeline

Our pipeline has zero backpressure today. The extract-cron.sh checks for unprocessed sources and dispatches workers regardless of eval queue state. If extraction outruns evaluation, PRs accumulate with no feedback signal. Simple fix: extraction dispatcher should check open PR count before dispatching. If open PRs > threshold, reduce extraction parallelism or skip the cycle.

Key Facts

• Apache Kafka implements backpressure through pull-based consumption model
• Flink, Spark Streaming, Akka Streams, Project Reactor all use backpressure as core pattern
• Backpressure implementation strategies: buffering with thresholds, rate limiting, dynamic adjustment, acknowledgment-based flow