Skip to content

fix(pep): detect flow stalls at 3x minRTT and race parallel rescue dials - #92

Merged
bojieli merged 1 commit into
mainfrom
fix/flow-stall-parallel-rescue
Sep 3, 2026
Merged

fix(pep): detect flow stalls at 3x minRTT and race parallel rescue dials#92
bojieli merged 1 commit into
mainfrom
fix/flow-stall-parallel-rescue

Conversation

@bojieli

@bojieli bojieli commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Problem

On a path with heavy upstream (client→server) packet loss (~30%, min RTT ~200ms — measured in production), flows stalled for tens of seconds and then failed with peer does not hold this session:

  • Lanes were declared dead only on quic-go's 15s idle timeout of total receive silence. Downstream keepalives (~4% loss direction) kept connections nominally alive while upstream throughput collapsed, so flows sat in dying lanes with zero recovery attempts.
  • When a lane finally died, the rescue was a single serialized QUIC handshake (up to ~22s/attempt) racing a hard 45s server-side grace that in-progress rescues did not extend. Late JOINs were refused unknown_session (36k+ suppressed server-side refusals observed).
  • --handshake-timeout 10s silently overrode NewClient's safer 30s default.

Fix

  • Flow-stall watchdog: no forward progress (protocol acked send offset / downstream arrivals — the signal keepalives can't fake) for clamp(3×minRTT, 250ms, 2s) while data is pending demotes the flow's data lanes and signals the lane manager. Strictly gated on pending data; app-limited idle flows never fire. Demote, never kill: suspected lanes keep receiving and clear on any ACK with new delivery information.
  • Parallel rescue dials: up to 3 concurrent dial+JOIN attempts, each sprayed onto the next walked hop port (degenerating to same-port parallel handshakes when hopping is off — the value is independent handshakes, not port diversity). First JOIN wins, losers cancelled. A healthy spare lane takes over within one scheduler poll. No TCP fallback policy changes.
  • Server grace extension: a validated JOIN arriving during the lane-replacement grace restarts the 45s budget instead of expiring mid-rescue.
  • Eviction hazard fixed: racing same-role JOINs are admitted in arrival order and a late loser could evict the winner's server-side lane at the lane ceiling — lanes younger than laneDeadPathDetection are now eviction-protected, and ResetFlowLimit refusals map to a per-attempt errLaneJoinCapacity (never resumeRefused).
  • --handshake-timeout default 10s → 30s, matching NewClient's default.

Metrics / logs

queqiao_flow_stalls_detected_total, queqiao_stall_spare_attaches_total, queqiao_lane_rescue_attempts_total, queqiao_lane_rescue_wins_total{attempt="0|1|2"}, queqiao_lane_grace_extensions_total; Info-level stall/rescue log lines. docs/LOGGING.md updated; changelog.d entry included.

Testing

New internal/pep/stallwatch_test.go (threshold clamp, gating, demote-not-kill, race first-wins with loser cancellation, refusal-ends-round) plus grace-extension and eviction-protection tests. go build ./..., go vet, and the full test suite pass (internal/pep 572s vs 584s baseline at HEAD; new tests pass -race -count=3).

Deliberately out of scope

A mid-download receive stall client-side (request acked, response streaming, upstream ACKs lost) is not detected — a quiet-period gate there is indistinguishable from a legitimately paused sender without violating the app-limited rule. The server-side watchdog fires in that case but can only demote/log (servers don't dial). No mux-level packet duplication (rejected in design review). No wire changes.

On a path with heavy upstream loss, flows stalled for tens of seconds with
no recovery: lanes were only declared dead on quic-go's 15s idle timeout,
and downstream keepalives kept connections nominally alive while upstream
throughput collapsed. The rescue that eventually ran was a single
serialized handshake racing a hard 45s server-side grace, so late JOINs
were refused unknown_session and flows failed.

- Add a flow-stall watchdog: no forward progress (acked send offset or
  downstream arrivals) for clamp(3x minRTT, 250ms, 2s) while data is
  pending demotes the flow's data lanes and signals the lane manager.
  Demote, never kill: suspected lanes keep receiving and clear on any ACK
  with new delivery information.
- Race up to 3 rescue dial+JOIN attempts, each sprayed onto the next
  walked hop port (degenerating to same-port parallel handshakes when
  hopping is off); first JOIN wins, losers are cancelled. A healthy spare
  lane takes over within one scheduler poll. TCP fallback policy is
  unchanged.
- Extend the server's lane-replacement grace when a validated JOIN
  arrives during the grace, so an in-progress rescue is not cut off.
- Protect lanes younger than laneDeadPathDetection from lane-ceiling
  eviction so a racing loser cannot retire the winner server-side, and
  map ResetFlowLimit refusals to a per-attempt errLaneJoinCapacity.
- Stop --handshake-timeout (10s) from silently overriding NewClient's
  30s default.

New metrics: queqiao_flow_stalls_detected_total,
queqiao_stall_spare_attaches_total, queqiao_lane_rescue_attempts_total,
queqiao_lane_rescue_wins_total{attempt}, queqiao_lane_grace_extensions_total.
@bojieli
bojieli merged commit e5826dd into main Sep 3, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant