Repository navigation
gateway: a total deadline per request; up-front body reads time out on h2c (D264, D265) - #126
Merged
Merged
Conversation
jaredLunde
enabled auto-merge (squash)
October 4, 2026 18:04
…p front times out (D264, D265)
No request had an absolute deadline: every bound was per read or per write,
and long_running (-pro) rows had no read timeout at all, so a stream kept
moving by its own bytes or kept alive by HTTP/2 PINGs held its tenant slot,
the in-flight gauge and body budget for as long as the client did not hang up.
request_max_secs (default 4h: the catalog's largest max output, 384k tokens,
at ~27 tok/s; 0 disables) fixes a deadline in request_filter (FullBody re-runs
inherit it). upstream_peer caps each attempt's read timeout at the time left
(read_capped), so a silent request ends exactly at the deadline; a moving one
is cut per chunk, both ways, against a coarse clock (deadline.rs: one relaxed
atomic load, one ticker thread per process). Before the head: a 504 "request
exceeded the gateway's maximum duration"; after: a cut stream (cut_short,
estimated). A policy limit: no breaker failure, no resend/failover, no key
cooled; ai_rejections_total{reason="request_deadline"}.
An h2c client that stalled while peek_body_model read its body up front had
no timeout at all (pingora's HTTP/2 server sets none) and held its tenant slot
forever; on HTTP/1.1 pingora's 60s timeout fired but, untagged, was answered
504 "upstream timed out". client_read_timeout_secs (default 60, pingora's own
HTTP/1.1 value; 0 disables) is set as the session read timeout (HTTP/1.1) and
wraps each up-front read where the session has none (HTTP/2); the peek's I/O
errors are tagged downstream, so a stall is a 408 with no provider contacted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
… stall) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
…nd on h1 and h2c (D265) Kills the two changed-lines survivors: a 0 s bound applied when disabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
…ond, same instant) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
jaredLunde
force-pushed
the
request-deadlines
branch
from
October 4, 2026 18:39
bc32582 to
13e1e1d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From the long-run audit: requests that could live forever and hold a tenant slot.
D264: no request had an absolute deadline.
long_runningrows had no read timeout and others only bounded the gap between reads, so a stream kept alive by bytes or h2 PINGs held a tenant slot, the in-flight gauge and its budget indefinitely. Newrequest_max_secs, default 4 h (14400 s; covers the catalog's largest output, 384k tokens, at ~27 tok/s), 0 disables.src/deadline.rs, one ticker thread advancing once a second, started only when the ceiling is on). No syscall or timer per chunk; a cut can land up to 1 s late.cut_short, billed an estimate). Large-body re-runs inherit the parent's deadline.ai_rejections_total{reason="request_deadline"}.D265: an h2c client stalling during the up-front body read had no timeout at all (pingora's h2 body read has none), holding a tenant slot indefinitely. On HTTP/1.1 pingora's 60 s fired but surfaced as 504 "upstream timed out" for a request no provider saw. New
client_read_timeout_secs, default 60 s; applied to h1 via pingora and as a per-read timer on h2 only. Up-front read errors are tagged as the client's: a stall is 408 "request body timed out" (consistent with D260), nothing billed, no provider contacted.Tests (
tests/request_deadlines.rs, 8): endless stream cut and slot released; silent provider 504 on an ordinary and a-prorow with no failover/breaker/key effect; ceiling off at 0; h2c and HTTP/1.1 up-front stalls → 408 with no upstream contact; trickled up-front body; trickled BYO upload; re-run inherits the deadline; late candidate never contacted. 18/18 hand mutations caught. ARCHITECTURE.md, config example updated.🤖 Generated with Claude Code
https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk