fix(pep): detect flow stalls at 3x minRTT and race parallel rescue dials - #92
Merged
Conversation
On a path with heavy upstream loss, flows stalled for tens of seconds with
no recovery: lanes were only declared dead on quic-go's 15s idle timeout,
and downstream keepalives kept connections nominally alive while upstream
throughput collapsed. The rescue that eventually ran was a single
serialized handshake racing a hard 45s server-side grace, so late JOINs
were refused unknown_session and flows failed.
- Add a flow-stall watchdog: no forward progress (acked send offset or
downstream arrivals) for clamp(3x minRTT, 250ms, 2s) while data is
pending demotes the flow's data lanes and signals the lane manager.
Demote, never kill: suspected lanes keep receiving and clear on any ACK
with new delivery information.
- Race up to 3 rescue dial+JOIN attempts, each sprayed onto the next
walked hop port (degenerating to same-port parallel handshakes when
hopping is off); first JOIN wins, losers are cancelled. A healthy spare
lane takes over within one scheduler poll. TCP fallback policy is
unchanged.
- Extend the server's lane-replacement grace when a validated JOIN
arrives during the grace, so an in-progress rescue is not cut off.
- Protect lanes younger than laneDeadPathDetection from lane-ceiling
eviction so a racing loser cannot retire the winner server-side, and
map ResetFlowLimit refusals to a per-attempt errLaneJoinCapacity.
- Stop --handshake-timeout (10s) from silently overriding NewClient's
30s default.
New metrics: queqiao_flow_stalls_detected_total,
queqiao_stall_spare_attaches_total, queqiao_lane_rescue_attempts_total,
queqiao_lane_rescue_wins_total{attempt}, queqiao_lane_grace_extensions_total.
This was referenced Sep 3, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a path with heavy upstream (client→server) packet loss (~30%, min RTT ~200ms — measured in production), flows stalled for tens of seconds and then failed with
peer does not hold this session:unknown_session(36k+ suppressed server-side refusals observed).--handshake-timeout 10ssilently overrodeNewClient's safer 30s default.Fix
clamp(3×minRTT, 250ms, 2s)while data is pending demotes the flow's data lanes and signals the lane manager. Strictly gated on pending data; app-limited idle flows never fire. Demote, never kill: suspected lanes keep receiving and clear on any ACK with new delivery information.laneDeadPathDetectionare now eviction-protected, andResetFlowLimitrefusals map to a per-attempterrLaneJoinCapacity(neverresumeRefused).--handshake-timeoutdefault 10s → 30s, matchingNewClient's default.Metrics / logs
queqiao_flow_stalls_detected_total,queqiao_stall_spare_attaches_total,queqiao_lane_rescue_attempts_total,queqiao_lane_rescue_wins_total{attempt="0|1|2"},queqiao_lane_grace_extensions_total; Info-level stall/rescue log lines.docs/LOGGING.mdupdated; changelog.d entry included.Testing
New
internal/pep/stallwatch_test.go(threshold clamp, gating, demote-not-kill, race first-wins with loser cancellation, refusal-ends-round) plus grace-extension and eviction-protection tests.go build ./...,go vet, and the full test suite pass (internal/pep572s vs 584s baseline at HEAD; new tests pass-race -count=3).Deliberately out of scope
A mid-download receive stall client-side (request acked, response streaming, upstream ACKs lost) is not detected — a quiet-period gate there is indistinguishable from a legitimately paused sender without violating the app-limited rule. The server-side watchdog fires in that case but can only demote/log (servers don't dial). No mux-level packet duplication (rejected in design review). No wire changes.