---
type: source
title: "What Is Backpressure"
author: "Dagster"
url: https://dagster.io/glossary/data-backpressure
date: 2024-01-01
domain: internet-finance
format: essay
status: processed
tags: [pipeline-architecture, backpressure, data-pipelines, flow-control]
processed_by: rio
processed_date: 2026-03-11
claims_extracted: ["backpressure-prevents-pipeline-failure-by-creating-feedback-loop-between-consumer-capacity-and-producer-rate.md"]
extraction_model: "anthropic/claude-sonnet-4.5"
extraction_notes: "Single claim extracted on backpressure as flow control mechanism. Source is practical implementation guide rather than research, so confidence is 'proven' based on widespread production adoption. Teleo pipeline relevance noted in claim body as concrete application context."
---

# What Is Backpressure (Dagster)

Dagster's practical guide to backpressure in data pipelines. Written for practitioners building real data processing systems.

## Key Content

- Backpressure: feedback mechanism preventing data producers from overwhelming consumers
- Without backpressure controls: data loss, crashes, resource exhaustion
- Consumer signals producer about capacity limits
- Implementation strategies: buffering (with threshold triggers), rate limiting, dynamic adjustment, acknowledgment-based flow
- Systems using backpressure: Apache Kafka (pull-based consumption), Flink, Spark Streaming, Akka Streams, Project Reactor
- Tradeoff: backpressure introduces latency but prevents catastrophic failure
- Key principle: design backpressure into the system from the start

## Relevance to Teleo Pipeline

Our pipeline has zero backpressure today. The extract-cron.sh checks for unprocessed sources and dispatches workers regardless of eval queue state. If extraction outruns evaluation, PRs accumulate with no feedback signal. Simple fix: extraction dispatcher should check open PR count before dispatching. If open PRs > threshold, reduce extraction parallelism or skip the cycle.


## Key Facts
- Backpressure implementations: buffering with thresholds, rate limiting, dynamic adjustment, acknowledgment-based flow
- Systems using backpressure: Apache Kafka (pull-based), Flink, Spark Streaming, Akka Streams, Project Reactor
- Failure modes without backpressure: data loss, crashes, resource exhaustion
