Skip to content

fix broker output forwarding under backpressure#205

Merged
almogdepaz merged 3 commits into
mainfrom
fix/take-control-output-lag
Jul 22, 2026
Merged

fix broker output forwarding under backpressure#205
almogdepaz merged 3 commits into
mainfrom
fix/take-control-output-lag

Conversation

@almogdepaz

@almogdepaz almogdepaz commented Jul 22, 2026

Copy link
Copy Markdown
Owner

stacked on

problem

During take-control on high-output TUIs, broker output forwarding can lag its 256-chunk broadcast receiver. The server then gets SubscriptionDropped / replay_truncated, reconnects the browser, and exposes redraw bytes as a visible character stream. The first implementation also allowed output to occupy the shared writer queue and starve control RPCs.

Live repros dropped 203–2,445 chunks per cycle.

change

  • continue draining live broadcast output while its output queue is backpressured
  • use separate per-connection control/event and output queues
  • prioritize control/event frames in the socket writer
  • coalesce contiguous PTY chunks into frames capped at 8 KiB
  • cap local per-subscription buffering at 8 MiB
  • preserve the pre-coalescing ~8 MiB output-queue memory envelope
  • retain existing lag recovery for sustained output beyond the local cap
  • queue subscription_dropped behind accepted output as a replay-ordering barrier
  • define seq as the final PTY-chunk watermark covered by a frame
  • update broker protocol and client/server comments for coalescing and priority semantics

regressions

  • deterministic 2,000-chunk writer-backpressure test: all bytes arrive, no drop, final seq is preserved
  • replay→live seam test: bytes remain ordered and frame watermarks increase through coalescing
  • sustained-output test: crossing the 8 MiB local cap queues output before subscription_dropped
  • real socket-writer test: queued control responses overtake queued output
  • TS client recovery test: the barrier resubscribes from the last delivered chunk seq
  • control-starvation test: saturated output cannot block a control response
  • real-browser, real-broker takeover test with four concurrent high-volume sessions:
    • old broker: red with four subscription forwarder lagged broadcast warnings
    • updated broker: no lag, no reconnect banner, one browser websocket; green 3/3 repeated runs

verification

  • Rust broker: 178 passed, 0 failed
  • Bun: 1,828 passed, 0 failed
  • Playwright: 105 passed, 126 expected skips
  • typecheck
  • rustfmt --check
  • git diff --check

deployment status

The initial broker implementation was deployed as PID 28214, and #196’s server candidate is live as PID 68020. The latest review-fix commit 09c5625 is not deployed; it requires another approved broker restart.

Live #196 assets currently remain:

  • app SHA-256: b2f548b08b293a80be1932659da375c6f05b96b946d09ac8b95103cafd5b2521
  • Ghostty SHA-256: 20a5074d3ef7a4e8aa5555ffc7c2ac936227cd094bd5949a7be08a2711a68f57

release gates

This remains draft pending final repeat differential review and real Android/Gboard verification for #196.

@almogdepaz
almogdepaz marked this pull request as ready for review July 22, 2026 12:31
@almogdepaz
almogdepaz marked this pull request as draft July 22, 2026 14:19
@almogdepaz
almogdepaz changed the base branch from fix/mobile-keyboard-grid-polish to main July 22, 2026 16:42
@almogdepaz
almogdepaz force-pushed the fix/take-control-output-lag branch from 09c5625 to bb3c9e2 Compare July 22, 2026 16:46
@almogdepaz
almogdepaz marked this pull request as ready for review July 22, 2026 16:47
@almogdepaz
almogdepaz merged commit 6ca43d3 into main Jul 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant