extract: 2025-12-00-javacodegeeks-reactive-programming-backpressure-stream-processing
Pentagon-Agent: Ganymede <F99EBFA6-547B-4096-BEEA-1D59C3E4028A>
This commit is contained in:
parent
be04ae7054
commit
743ecd8d50
2 changed files with 17 additions and 1 deletions
|
|
@ -32,6 +32,12 @@ The Teleo pipeline currently has zero backpressure. The extract-cron.sh dispatch
|
|||
|
||||
Simple implementation: extraction dispatcher should check open PR count before dispatching. If open PRs exceed threshold, reduce extraction parallelism or skip the cycle entirely. This creates the feedback loop that prevents eval queue overload.
|
||||
|
||||
|
||||
### Additional Evidence (extend)
|
||||
*Source: [[2025-12-00-javacodegeeks-reactive-programming-backpressure-stream-processing]] | Added: 2026-03-16*
|
||||
|
||||
Reactive Streams specification implements backpressure through Publisher/Subscriber/Subscription interfaces where Subscriber requests N items and Publisher delivers at most N, creating demand-based flow control. Four standard strategies exist: Buffer (accumulate with threshold triggers, risk unbounded memory), Drop (discard excess), Latest (keep only most recent), and Error (signal failure on overflow). Key architectural insight: backpressure must be designed into systems from the start—retrofitting it is much harder.
|
||||
|
||||
---
|
||||
|
||||
Relevant Notes:
|
||||
|
|
|
|||
|
|
@ -6,8 +6,12 @@ url: https://www.javacodegeeks.com/2025/12/reactive-programming-paradigms-master
|
|||
date: 2025-12-01
|
||||
domain: internet-finance
|
||||
format: essay
|
||||
status: unprocessed
|
||||
status: enrichment
|
||||
tags: [pipeline-architecture, backpressure, reactive-streams, flow-control, producer-consumer]
|
||||
processed_by: rio
|
||||
processed_date: 2026-03-16
|
||||
enrichments_applied: ["backpressure-prevents-pipeline-failure-by-creating-feedback-loop-between-consumer-capacity-and-producer-rate.md"]
|
||||
extraction_model: "anthropic/claude-sonnet-4.5"
|
||||
---
|
||||
|
||||
# Reactive Programming Paradigms: Mastering Backpressure and Stream Processing
|
||||
|
|
@ -29,3 +33,9 @@ Practitioner guide to implementing backpressure in reactive stream processing sy
|
|||
## Relevance to Teleo Pipeline
|
||||
|
||||
Our pipeline currently has NO backpressure. Extract produces PRs that accumulate in eval's queue without any feedback mechanism. If research dumps 20 sources, extraction creates 20 PRs, and eval drowns trying to process them all. We need a "buffer + rate limit" strategy: extraction should check eval queue depth before starting new work, and slow down or pause when eval is backlogged.
|
||||
|
||||
|
||||
## Key Facts
|
||||
- Reactive Streams standard defines Publisher/Subscriber/Subscription interfaces for demand-based flow control
|
||||
- Four backpressure strategies: Buffer, Drop, Latest, Error
|
||||
- Practical implementations include Project Reactor (Spring WebFlux), Akka Streams, RxJava
|
||||
|
|
|
|||
Loading…
Reference in a new issue