feat(filter): resume IRR iterations within streaming responses by leseb · Pull Request #988 · praxis-proxy/praxis

leseb · 2026-08-15T13:15:48Z

What does this PR do?

Extends iterative_request_router so a single committed downstream response can span multiple IRR steps.

The first streamed step can complete, evaluate its existing on_result transitions, start another model or tool step, and continue writing through the same client response without recommitting headers.

Key additions:

An internal IrrStepRunner shared by buffered and streaming execution
An IrrStreamingSession that owns state and resumes steps after streamed completion
Header-safe failover before a step exposes body bytes
Completion transitions after response-body filters and completion hooks run
Bounded, provider-neutral local chunk emission between upstream streams
Logical response byte, retained-state, iteration, per-step, idle, and overall deadline enforcement
Typed post-commit termination handling and downstream cancellation cleanup
Runtime-resource and listener-limit propagation into nested step pipelines
End-to-end HTTP/1.1 and HTTP/2 coverage for ordering, backpressure, continuation, failure, suppression, framing, and cancellation

The existing on_result configuration continues to drive transitions. No provider-specific SSE parsing, tool policy, or new transition YAML is added to Praxis.

This builds on the streaming transport and terminal-response foundation merged through #951, #952, and #953. Those changes are dependencies; they do not implement the resumed IRR session loop added here.

Which issue(s) does this relate to?

Fixes #984

Parent epic: #877

Checklist

Signed off all commits (git commit -s)
Unit tests added or updated
Integration tests added or updated
make lint passes locally
make doc passes locally

Does this introduce a breaking change?

No. Existing buffered and terminal-only streaming behavior remains supported. The new streaming-session capabilities and limits are additive.

Keep the downstream response open while the iterative request router runs successive model and tool steps. Stream each step incrementally, then use completion transitions to resume the next step without committing a new response. Add bounded iteration state, logical stream limits, deadline handling, header-safe failover, and typed termination handling. Propagate runtime resources and listener limits into nested step pipelines, and preserve correct lifecycle ordering across normal completion, errors, and client disconnects. Cover the behavior with unit and end-to-end tests for streaming order, multi-step resumption, suppression, limits, failover, and cleanup. Fixes praxis-proxy#984 Signed-off-by: Sébastien Han <seb@redhat.com>

praxis-bot-app · 2026-08-15T13:15:58Z

PR too large: 2175 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add skip/pr-conventions label to override.

leseb · 2026-08-15T16:37:51Z

Validated on praxis-proxy/ai#756

praxis-bot

PR Review: feat(core): add streaming SubRequestClient response transport

Adds IrrStreamingSession for multi-step streaming within a committed downstream response, refactors step execution into a reusable IrrStepRunner, and introduces typed abnormal stream termination (StreamTermination) with filter-driven completion hooks.

Overall Assessment

This is a large, well-structured PR with thorough test coverage (85+ streaming tests). The streaming session design is sound: header-safe failovers run before body exposure, completion-dependent transitions run after clean EOF, and the bounded pending-chunk emission model keeps memory under control. The nested pipeline propagation via visit_nested_pipelines is a necessary architectural fix.

Three findings, all Medium severity.

Severity	File	Finding
Medium	`runner.rs`	Missing per-step tracing span (observability regression)
Medium	`mod.rs`, `streaming.rs`	`into_completion()` error paths drop parent extensions
Medium	`request_filter/mod.rs`	`next_chunk()` cancellation safety undocumented in `tokio::select!`

Fixes a testing flake when the metrics wouldn't respond quickly enough. Signed-off-by: Shane Utt <shaneutt@linux.com>

Restore the per-step tracing span around nested filter and transport execution so logs retain their step and iteration context. Return completion failures with recoverable parent extensions and restore them across buffered and committed-streaming error paths. Add regression coverage for both lifecycle shapes. Document the RAII cleanup contract when downstream disconnect races an in-flight streaming step. Signed-off-by: Sébastien Han <seb@redhat.com>

…-984

The merge with upstream/main incorrectly kept the old multi-line wait_for_metric call that upstream replaced with thread::sleep, causing a nightly rustfmt check failure in CI. Signed-off-by: Sébastien Han <seb@redhat.com>

shaneutt

Did an initial pass: putting a hold for me to do a thorough review.

praxis-bot

Re-review

All three findings from the initial review are addressed by the fix commit (75ba179):

Per-step tracing span: restored via .instrument(step_span) in open_step().
into_completion() extension recovery: StepCompletionError carries parent extensions; four regression tests confirm restoration across buffered, streaming, and guard paths.
Cancellation safety: documented with a // Cancellation contract: comment at the tokio::select! site.

One new finding from the refactored code.

…-984

The streaming transport path logged failures with warn! but the buffered path silently discarded both build_peer and execute errors. Add matching warn! calls so production debugging has parity across both transport modes. Signed-off-by: Sébastien Han <seb@redhat.com>

Upstream renamed `has_body_dependent_transitions` to `streaming_transition_order_is_valid` with new semantics (interleaved transition phases) and shortened the StreamBuffer error message. Update test names, YAML configs, and assert strings accordingly. Signed-off-by: Sébastien Han <seb@redhat.com>

leseb · 2026-08-25T16:57:18Z

Thank you for your contribution! This is already in main via #978, see: 71ab252. Closing as superseded.

Sorry the PR metadata was incorrect, this PR is still valid and the actual head implements #984.

…-984 Resolve three conflicts from upstream's perf/refactor batch: - filter/src/pipeline/build.rs: adopt upstream's body_filter_indices() and request/response_body_filter_indices fields while keeping our id_generator/time_source construction path. - filter/.../iterative_request_router/config.rs: keep both the max_stream_response_bytes validation (ours) and the max_response_bytes and step_timeout_ms validations (upstream); they are additive. - filter/.../iterative_request_router/mod.rs: keep our runner.open_step() refactor and drop upstream's superseded inline step-execution block; the extracted logic lives in runner.rs. Signed-off-by: Sébastien Han <seb@redhat.com>

leseb requested a review from a team August 15, 2026 13:15

leseb requested a review from shaneutt as a code owner August 15, 2026 13:15

leseb added the skip/pr-conventions Skip conventions checks for PRs label Aug 15, 2026

leseb assigned shaneutt Aug 15, 2026

leseb mentioned this pull request Aug 15, 2026

feat(openai): stream Responses across IRR agentic rounds praxis-proxy/ai#756

Draft

11 tasks

praxis-bot reviewed Aug 17, 2026

View reviewed changes

Comment thread filter/src/builtins/http/traffic_management/iterative_request_router/runner.rs

Comment thread filter/src/builtins/http/traffic_management/iterative_request_router/mod.rs Outdated

Comment thread protocol/src/http/pingora/handler/request_filter/mod.rs

shaneutt and others added 4 commits August 17, 2026 15:14

fix(Tests): wait for metrics

322c67e

Fixes a testing flake when the metrics wouldn't respond quickly enough. Signed-off-by: Shane Utt <shaneutt@linux.com>

Merge remote-tracking branch 'upstream/main' into leseb/irr-streaming…

8c54342

…-984

fix(tests): adopt upstream wait_for_metric removal

8ab2e36

The merge with upstream/main incorrectly kept the old multi-line wait_for_metric call that upstream replaced with thread::sleep, causing a nightly rustfmt check failure in CI. Signed-off-by: Sébastien Han <seb@redhat.com>

shaneutt force-pushed the main branch from 322c67e to f872432 Compare August 17, 2026 20:40

shaneutt added this to AI Gateway - Model Serving Aug 18, 2026

shaneutt moved this to Review in AI Gateway - Model Serving Aug 18, 2026

shaneutt added this to the v0.5.3 milestone Aug 18, 2026

shaneutt requested changes Aug 18, 2026

View reviewed changes

shaneutt modified the milestones: v0.5.3, v0.5.4 Aug 19, 2026

praxis-bot reviewed Aug 20, 2026

View reviewed changes

Comment thread filter/src/builtins/http/traffic_management/iterative_request_router/runner.rs Outdated

shaneutt force-pushed the main branch from ed2eaca to dab4acd Compare August 20, 2026 19:07

leseb added 3 commits August 21, 2026 16:40

Merge remote-tracking branch 'upstream/main' into leseb/irr-streaming…

0ff0b22

…-984

leseb requested a review from shaneutt August 25, 2026 11:27

shaneutt closed this Aug 25, 2026

github-project-automation Bot moved this from Review to Done in AI Gateway - Model Serving Aug 25, 2026

leseb reopened this Aug 25, 2026

leseb changed the title ~~feat(core): add streaming SubRequestClient response transport~~ feat(filter): resume IRR iterations within streaming responses Aug 25, 2026

leseb mentioned this pull request Aug 31, 2026

Errors after response.completed leave the response persistable as successful praxis-proxy/ai#841

Open

shaneutt approved these changes Aug 31, 2026

View reviewed changes

shaneutt merged commit 47ba400 into praxis-proxy:main Aug 31, 2026
26 of 27 checks passed

github-project-automation Bot moved this from Review to Done in AI Gateway - Model Serving Aug 31, 2026

leseb deleted the leseb/irr-streaming-984 branch September 1, 2026 06:41

leseb mentioned this pull request Sep 2, 2026

stream_events: synthesize SSE for locally executed tools praxis-proxy/ai#276

Open

Navigation Menu

Uh oh!

feat(filter): resume IRR iterations within streaming responses - #988

feat(filter): resume IRR iterations within streaming responses#988
shaneutt merged 9 commits into
praxis-proxy:mainfrom
leseb:leseb/irr-streaming-984

leseb commented Aug 15, 2026 •

edited

Loading

Uh oh!

praxis-bot-app Bot commented Aug 15, 2026

Uh oh!

leseb commented Aug 15, 2026

Uh oh!

praxis-bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shaneutt left a comment

Uh oh!

praxis-bot left a comment

Uh oh!

Uh oh!

leseb commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

leseb commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Which issue(s) does this relate to?

Checklist

Does this introduce a breaking change?

Uh oh!

praxis-bot-app Bot commented Aug 15, 2026

Uh oh!

leseb commented Aug 15, 2026

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

PR Review: feat(core): add streaming SubRequestClient response transport

Overall Assessment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shaneutt left a comment

Choose a reason for hiding this comment

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

Re-review

Uh oh!

Uh oh!

leseb commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leseb commented Aug 15, 2026 •

edited

Loading