---
type: source
title: "What Is Backpressure"
author: "Dagster"
url: https://dagster.io/glossary/data-backpressure
date: 2024-01-01
domain: internet-finance
format: essay
status: processed
tags: [pipeline-architecture, backpressure, data-pipelines, flow-control]
processed_by: rio
processed_date: 2026-03-11
claims_extracted: ["backpressure-prevents-pipeline-failure-by-signaling-consumer-capacity-limits-to-producers.md", "extraction-without-backpressure-creates-unbounded-pr-accumulation-when-extraction-outruns-evaluation.md"]
extraction_model: "anthropic/claude-sonnet-4.5"
extraction_notes: "Extracted two claims: one general claim about backpressure as a proven pattern in data systems, one experimental claim about Teleo pipeline architecture. The source is a practitioner glossary entry, not academic research, but describes widely-deployed production patterns. Second claim applies backpressure concept to Teleo's own pipeline based on curator's relevance note. No entities to extract — this is architectural pattern documentation, not company/product/market data."
---

# What Is Backpressure (Dagster)

Dagster's practical guide to backpressure in data pipelines. Written for practitioners building real data processing systems.

## Key Content

- Backpressure: feedback mechanism preventing data producers from overwhelming consumers
- Without backpressure controls: data loss, crashes, resource exhaustion
- Consumer signals producer about capacity limits
- Implementation strategies: buffering (with threshold triggers), rate limiting, dynamic adjustment, acknowledgment-based flow
- Systems using backpressure: Apache Kafka (pull-based consumption), Flink, Spark Streaming, Akka Streams, Project Reactor
- Tradeoff: backpressure introduces latency but prevents catastrophic failure
- Key principle: design backpressure into the system from the start

## Relevance to Teleo Pipeline

Our pipeline has zero backpressure today. The extract-cron.sh checks for unprocessed sources and dispatches workers regardless of eval queue state. If extraction outruns evaluation, PRs accumulate with no feedback signal. Simple fix: extraction dispatcher should check open PR count before dispatching. If open PRs > threshold, reduce extraction parallelism or skip the cycle.


## Key Facts
- Apache Kafka implements backpressure through pull-based consumption model
- Flink, Spark Streaming, Akka Streams, Project Reactor all use backpressure as core pattern
- Backpressure implementation strategies: buffering with thresholds, rate limiting, dynamic adjustment, acknowledgment-based flow