Skip to content

Browser proxy - WS tunnel failed at handshake - #78

Merged
bharathm03 merged 1 commit into
developmentfrom
antgrid/ws-tunnel-ordering
Sep 2, 2026
Merged

Browser proxy - WS tunnel failed at handshake#78
bharathm03 merged 1 commit into
developmentfrom
antgrid/ws-tunnel-ordering

Conversation

@bharathm03

Copy link
Copy Markdown
Contributor

A previewed page's WebSocket could lose its handshake and then hang forever. Two independent causes, fixed on both sides plus the direction the first attempt missed.

App → bridge: ordering, and knowing the frames arrived

Browser-to-bridge frames now leave through one FIFO, so async sealing can no longer put the first SignalR frame ahead of its own tunnel:ws-open.

Ordering is only half of it. sendOnStream returns a successfully completed future when no session keys are installed, so a tunnel:ws-open lost to a reconnect was booked as sent and nothing ever recovered the tunnel. The queue now checks isEstablished, bounds itself at 64 frames / 1 MB, times each send out at 10s, and closes the browser socket on any frame it cannot vouch for — the page reconnects instead of holding a socket against a tunnel the bridge never heard of.

The close frame waits on the backlog for a bounded 2s rather than strictly behind it: a wedged queue has already lost the data the close would follow, and the bridge's upstream dev-server socket stays open until it lands.

Bridge → app: the direction that actually reorders

_onRouted fire-and-forgot an async decrypt. The platform AES-GCM implementation dispatches by payload size, so a small frame reliably overtakes a large one — an 11-byte {"type":6} ping ahead of the 30 KB render batch it acknowledges, one terminal:output chunk ahead of another, or a fragment ahead of its predecessor in the reassembler. Now chained per channel in MachineSession; channels stay independent of each other.

Bridge: a lost prefix ends the tunnel

A pre-open buffer that overflows or expires now poisons its tunnelId instead of dropping frames and carrying on. Replaying a stream with a hole in it is worse than the if (!entry) return; this replaced: a dev server handed a spliced message stream believes it holds a valid session and hangs, where a refused tunnel gives the browser the close event its reconnect logic waits for.

  • The tombstone outlives the refusal, so frames still in flight cannot start a second, tail-only buffer for the same id.
  • A bridge-initiated teardown leaves one too. The app answers that close by dropping its own entry and never replies, so onWsClose never runs — trailing frames would otherwise hold one of the 64 slots for a full TTL each.
  • Tombstones are the first eviction candidate when the table fills, which is what stops a dev server in a reconnect loop (a fresh tunnelId per attempt, ~1/s) from starving the live tunnel.
  • Timers capture the tunnelId rather than the frame, so a rejected payload is not pinned for the TTL, and they are unref()'d.
  • Drop paths log once per tunnel or throttled with the id, never once per frame from the data path.

The post-open buffer (entry.pending) gained the same ceilings, and it is the window that needed them more: a port that accepts TCP but stalls the upgrade — a dev server mid-startup, or an https-only port reached as ws:// — holds it open for the OS connect timeout, tens of seconds against the pre-open path's five.

stop() is now terminal (a late frame can no longer re-arm a timer on a torn-down manager) and sends its own tunnel:ws-close before closing each socket. A socket still CONNECTING never fires a close event, so a session deleted mid-handshake left the app believing the tunnel was live. onHttpRequest is deliberately not gated — dropping one costs the app a 30s timeout, and serving it holds nothing open.

Deliberately not in scope

  • Serializing on the transport collapses concurrent AES-GCM sealing to one operation. That is the price of ordering; sealing concurrently and writing in order belongs inside the relay transport, not this queue.
  • sendOnStream's outbound reordering is general to every caller, but serializing it there would put head-of-line blocking on the control channel. Note that hoisting the queue next to AgentTransport is wrong regardless — that package is Apache-2.0 and the boundary is one-way.

Verification

bun run --filter antgrid-bridge test (3234), cd app && flutter test (3137), cd packages/antgrid_relay_client && dart test (214), flutter analyze clean.

New coverage: overflow refusal, tombstone durability across a second open, table starvation under a reconnect loop, stop() mid-connect, and an undeliverable open closing the browser socket. The app-side ordering test now closes the socket while the send gate is held and asserts the close lands after the queued data — asserting only that a close eventually arrives passed on plain fire-and-forget sends.

The app's browser-to-bridge frames now leave in one FIFO, so async sealing can no longer put the first SignalR frame ahead of its tunnel:ws-open. Ordering is only half of it: a send with no session keys installed completes SUCCESSFULLY and delivers nothing, so the queue checks isEstablished, bounds itself at 64 frames / 1 MB, times each send out, and closes the browser socket on any frame it cannot vouch for — the page then reconnects instead of holding a socket against a tunnel the bridge never heard of. The close waits on the backlog for a bounded 2s rather than strictly behind it, since a wedged queue has already lost the data the close would follow and the bridge's upstream dev-server socket stays open until it lands.

Inbound needed the same treatment and is where the reordering actually bites: decrypt is async and the platform AES-GCM implementation dispatches by payload size, so a small frame overtakes a large one — a ping ahead of the 30 KB render batch it acknowledges. Chained per channel in MachineSession; channels stay independent.

On the bridge, a pre-open buffer that overflows or expires now POISONS its tunnelId instead of quietly dropping frames and carrying on. Replaying a stream with a hole in it is worse than the guard this replaced: a dev server handed a spliced message stream believes it holds a valid session and hangs, where a refused tunnel gives the browser the close event its reconnect logic waits for. The tombstone outlives the refusal so frames still in flight cannot start a second, tail-only buffer, and a bridge-initiated teardown leaves one too — the app answers that close by dropping its own entry, so onWsClose never runs and trailing frames would otherwise hold a slot for a full TTL each. Tombstones are the first eviction candidate when the table fills, which is what stops a dev server in a reconnect loop from starving the live tunnel.

The post-open buffer gained the same ceilings, and it is the window that actually needed them: a port that accepts TCP but stalls the upgrade holds it open for the OS connect timeout, tens of seconds against the pre-open path's five. stop() is now terminal and sends its own tunnel:ws-close before closing each socket — a socket still CONNECTING never fires a close event, so a session deleted mid-handshake left the app believing the tunnel was live.
@bharathm03 bharathm03 changed the title A WS tunnel that loses its prefix is refused, not spliced Browser proxy - WS tunnel failed at handshake Sep 2, 2026
@bharathm03
bharathm03 merged commit df96f21 into development Sep 2, 2026
4 checks passed
@bharathm03
bharathm03 deleted the antgrid/ws-tunnel-ordering branch September 2, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant