Terminal attach budget: bound replays, fast reload, pong fix (WI-121)

.github/workflows/e2e.yml

@@ -177,10 +177,12 @@ jobs:
             engine bash /tmp/e2e/scripts/e2e_stack_smoke.sh http://localhost:8910
 
       # The live Playwright project, in-network. A Playwright
-      # container on the compose network runs the SAME gui.spec.ts specs against
+      # container on the compose network runs the `live`-project specs against
       # http://engine:8910 with the real API (no installFixtures), so mock/truth
-      # drift fails a test. The runner cannot reach the stack over host loopback,
-      # so this runs via `docker compose run` on the network like the smoke walk.
+      # drift fails a test. This includes gui.spec.ts and terminalReplay.spec.ts
+      # (H3, WI-124: the terminal-attach-budget symptoms against a real load
+      # session). The runner cannot reach the stack over host loopback, so this
+      # runs via `docker compose run` on the network like the smoke walk.
       #
       # First landing is advisory (continue-on-error): the in-network smoke walk
       # above is the hard end-to-end gate, and this project is being brought up

docs/ENGINE.md

@@ -623,8 +623,9 @@ token can list sessions and read a scrollback snapshot over
 
 The attach sequence is ordered:
 
-1. client sends auth control frame, optionally with its last applied cursor:
-   `{"type":"auth","token":"...","resume_from":123}`
+1. client sends auth control frame, optionally with its last applied cursor
+   and always with its replay budget:
+   `{"type":"auth","token":"...","resume_from":123,"snapshot_tail_bytes":1048576}`
 2. server sends `snapshot-start`
 3. server sends zero or more binary scrollback chunks
 4. server sends `snapshot-done`
@@ -634,20 +635,45 @@ Client text control frames:
 
 ```json
 {"type":"resize","cols":120,"rows":40}
-{"type":"ping"}
+{"type":"ping","id":1}
 ```
 
 Server text control frames:
 
 ```json
 {"type":"snapshot-start","session_id":"uuid","scrollback_bytes":0,"scrollback_pos":0,"reset":true}
 {"type":"snapshot-done"}
+{"type":"pong","id":1,"pos":123}
 {"type":"lag","note":"client too slow; reattach"}
 ```
 
-When `resume_from` is still retained, `snapshot-start.reset` is false and the
-binary snapshot contains only newer bytes. Otherwise the server returns a full
-snapshot with `reset` true.
+A `pong` echoes the `ping.id` and carries `pos`: the absolute byte offset the
+server has **actually streamed to that socket**, not `total_written`. The pong
+is formed and sent by the same outbound task that streams output, after
+flushing anything already queued, so it is ordered on the wire after those
+chunks and its `pos` can never exceed what the client has received. A client
+uses it purely as a liveness probe — a `pos` ahead of what it has rendered is a
+suspect to confirm with a second probe, not an immediate reconnect.
+
+`snapshot_tail_bytes` is the replay **budget**, and it bounds every reply — a
+client's terminal keeps only a fixed scrollback, so the server never ships more
+than the client can hold:
+
+- **Cold attach** (no `resume_from`): a ground-state-aligned tail of at most the
+  budget, `reset` true.
+- **Warm reattach whose cursor is retained and whose delta fits the budget:**
+  `reset` false and the binary snapshot contains only the newer bytes,
+  byte-for-byte — an ordinary switch-away/switch-back appends without a clear.
+- **Warm reattach whose cursor aged out of the ring, or whose delta exceeds the
+  budget:** a ground-state-aligned tail of at most the budget, `reset` true.
+  `reset: true` may therefore follow a `resume_from`. The client discards its
+  stale cursor and re-anchors to `scrollback_pos - scrollback_bytes` (the start
+  of the tail), so after replaying the tail its position is `scrollback_pos`
+  again and live traffic resumes with no gap.
+
+The returned `scrollback_pos` is always the absolute end position, unaffected by
+trimming the front. Omitting `snapshot_tail_bytes` (the in-band lag resync,
+which carries its own cursor) leaves a retained delta unbounded and byte-exact.
 
 A text frame that does not parse as a control message is treated as raw input,
 because some tools send keystrokes as text. Snapshot chunks are capped at

docs/ROADMAP.md

@@ -0,0 +1,111 @@
+# Roadmap and design notes
+
+Forward-looking design decisions that are not yet work in flight. Each entry is
+a recommendation with enough reasoning and numbers to act on — or to decide
+"not now" without re-deriving the analysis.
+
+## Engine-side headless VT per session (D6, WI-130)
+
+*Spike, 2026-09-09. Part of the terminal-attach-budget initiative (WI-121).*
+
+### Question
+
+The engine keeps only **raw PTY bytes** in a per-session ring
+(`scrollback.rs`), and every client rebuilds a terminal by re-parsing those
+bytes into xterm.js on attach. Should the engine instead run a **headless
+terminal emulator per session**, so a cold attach ships *screen state*
+(tmux-style) — a serialized screen plus N lines of scrollback — instead of byte
+history?
+
+### What the budget work already fixed
+
+Before committing to a VT, weigh it against what shipped in this initiative:
+
+- **F1 (WI-125)** bounds *every* replay to the tail budget — the 4 MiB aged-out
+  flood is gone; a cold or stale attach now ships at most the budget
+  (default 1 MiB), ground-state aligned.
+- **F3 (WI-127)** stops retained tabs from time-slicing the active pane's
+  parser on reload.
+- **F5 (WI-129)** persists a serialized xterm screen client-side, so a *reload*
+  restores in one write with no raw re-parse, and (once F4 lands) a *switch*
+  keeps the socket open and never re-streams.
+
+So the two operator symptoms (switch re-stream, slow fresh open) are addressed
+by bounding and by client-side serialization, **without** an engine VT. The VT
+is only worth its cost if a residual problem remains after these ship and are
+measured on prod — see "When to revisit".
+
+### Crate options
+
+| Crate | What it is | Fit for a screen-snapshot server VT |
+|---|---|---|
+| `vt100` | A minimal pure-Rust parser that maintains a screen grid + scrollback, no rendering. | **Best fit.** Smallest surface and memory; exposes the cell grid directly, which is all a snapshot needs. Serializing its screen to an xterm-compatible escape stream is a bounded amount of new code. |
+| `alacritty_terminal` | The terminal model behind Alacritty. | Heavier per-cell model and an API shaped around a GPU renderer's needs; more memory and more moving parts than a snapshot server wants, and version churn tied to Alacritty. |
+| `termwiz` | WezTerm's terminal library. | Full-featured (its own line/cell/attribute model, image protocols); the largest surface of the three. Overkill for "hold a screen and emit a snapshot". |
+
+### Cost estimate (per session, at 8 busy sessions)
+
+No live measurement was taken (an agent session cannot reach a prod stack; use
+H1's load session — `load_session_command` in the engine integration tests — to
+measure before adopting). Grounded estimate for `vt100`:
+
+- **Memory.** A cell is a codepoint + attributes ≈ 8–16 B. A 200-col grid with
+  50 visible rows + 1000 scrollback lines ≈ 210 000 cells ≈ **1.7–3.4 MB per
+  session**, i.e. **~14–27 MB across 8 sessions** — *on top of* the existing
+  4 MiB raw ring per session unless the ring is then shrunk. The ring cannot be
+  dropped entirely: warm `resume_from` deltas and the history archive still read
+  raw bytes.
+- **CPU.** The VT must parse every byte the PTY produces, on the hot reader
+  path. The client already parses at ~5–6 MB/s into xterm; a Rust grid VT is
+  faster (tens of MB/s) but it is now paid **once per session, always**, not
+  once per attach. For a chatty agent session this is continuous cost the
+  raw-ring design does not have.
+
+### Wire shape (if adopted)
+
+Add a snapshot kind rather than changing the existing one:
+
+```json
+{"type":"snapshot-start","kind":"screen","session_id":"…","scrollback_pos":N,"reset":true}
+```
+
+followed by a server-produced, xterm-compatible escape stream that redraws the
+screen and the last N scrollback lines, then live bytes as today. `resume_from`
+still selects the byte-delta path (unchanged); `kind:"screen"` is the cold /
+aged-out path only. The history archive is untouched (it keeps raw bytes).
+
+### Fidelity
+
+A server VT and the client xterm are two independent emulators; they can
+disagree (edge cases in wide chars, unusual SGR, DEC private modes). The
+snapshot is only correct if the server's redraw reproduces on xterm what the
+program intended. The H2 corpus + fidelity harness is the right place to prove
+this: parse each corpus in the candidate crate, emit the screen escape stream,
+write it into headless xterm, and compare `serialize()` against xterm fed the
+raw bytes. Adopt only if that comparison is clean across the corpus.
+
+### Mobile
+
+The prize on mobile would be dropping the IndexedDB cache entirely: if a cold
+attach always ships a small screen snapshot, the phone need not persist raw
+scrollback. But F5 already gives mobile a one-write restore from a *bounded*
+serialized cache, so the marginal win is "no client cache at all" versus "a
+small, capped client cache" — modest.
+
+### Recommendation: **not now**
+
+Ship and measure F1–F5 (and F4) on prod first. The engine VT doubles per-session
+memory and adds continuous CPU on the PTY hot path to solve a problem the
+bounding + client serialization work already targets. Revisit **only** if, with
+those shipped and measured:
+
+1. a first cold attach on mobile is still too slow *and* the ≤1 MiB tail is the
+   bottleneck (not network), or
+2. keeping any client-side scrollback cache proves untenable (storage, privacy),
+   or
+3. a new requirement wants server-authoritative screen state (e.g. server-side
+   search over live screens, or thumbnails).
+
+If revisited, the first step is a `vt100`-based measurement against H1's load
+session at 8 sessions, then the H2 fidelity comparison above — no new work items
+are filed now.

engine/server/src/pty.rs

@@ -130,13 +130,25 @@ impl Session {
     /// Snapshot only output newer than a client cursor. The boolean tells the
     /// client whether its existing terminal state must be reset.
     ///
-    /// `tail_bytes` caps a **cold** attach only: a fresh client with no
-    /// cache sends no `resume_from`, and without a cap the full snapshot is the
-    /// entire scrollback ring. When present it bounds that full snapshot to at
-    /// most that many trailing bytes. A warm reattach (`resume_from` present)
-    /// ignores it entirely, so its `reset: false` delta stays byte-for-byte
-    /// unchanged; the returned position (`total_written`) is unaffected by
-    /// trimming the front, so the live stream still resumes with no gap.
+    /// `tail_bytes` is the replay **budget**, and it now bounds *every* path,
+    /// not just a cold attach (F1, WI-125). A client's xterm keeps only a fixed
+    /// scrollback (5000 lines); shipping more than the budget is wasted parse
+    /// on the client's single message thread, and on a stale reattach it is
+    /// parsed *above* a screen the client already has out of date. So:
+    ///
+    /// - Cold attach (no `resume_from`): bound the full snapshot to the budget.
+    /// - Warm reattach whose cursor is still retained and whose delta fits the
+    ///   budget: the delta is sent byte-for-byte with `reset: false`, so a
+    ///   switch-away/switch-back that produced little still appends cleanly.
+    /// - Warm reattach whose cursor aged out of the ring, **or** whose delta
+    ///   exceeds the budget: a ground-state-aligned tail of at most the budget,
+    ///   with `reset: true`. Never the whole untrimmed ring — that was the
+    ///   4 MiB flood, 4x worse than a cold attach.
+    ///
+    /// The returned position is always `total_written`, unaffected by trimming
+    /// the front, so the live stream resumes with no gap. When `tail_bytes` is
+    /// `None` (the in-band resync path, which carries its own cursor) nothing
+    /// is bounded and a retained delta stays byte-exact.
     pub fn snapshot_for_attach(
         &self,
         resume_from: Option<u64>,
@@ -146,12 +158,25 @@ impl Session {
         let pos = sb.total_written();
         if let Some(cursor) = resume_from {
             if let Some(delta) = sb.snapshot_since(cursor) {
-                return (delta, pos, false);
+                // A retained delta within budget is byte-exact and appends
+                // (reset:false). An oversized delta is capped to a bounded tail
+                // and reset:true — the client's xterm cannot keep more than its
+                // scrollback anyway, so an over-budget delta would only waste
+                // parse above an already-stale screen.
+                match tail_bytes {
+                    Some(limit) if delta.len() > limit => {
+                        return (sb.snapshot_tail(limit), pos, true);
+                    }
+                    _ => return (delta, pos, false),
+                }
             }
-            // Cursor aged out of the ring: a full reset snapshot, untrimmed —
-            // the tail cap is a cold-attach affordance and a warm reattach is
-            // never quietly narrowed.
-            return (sb.snapshot(), pos, true);
+            // Cursor aged out of the ring: a bounded, ground-state-aligned tail
+            // and a reset — never the whole untrimmed ring.
+            let snapshot = match tail_bytes {
+                Some(limit) => sb.snapshot_tail(limit),
+                None => sb.snapshot(),
+            };
+            return (snapshot, pos, true);
         }
         // Cold attach: bound the snapshot to the client's tail hint so first
         // open never ships the whole ring buffer.

engine/server/src/ws.rs

@@ -99,6 +99,47 @@ fn coalesce(snap_pos: u64, chunks: &[OutputChunk]) -> (Vec<u8>, u64) {
     (out, end)
 }
 
+/// Send whatever broadcast chunks are already queued for this socket, without
+/// blocking, advancing `sent_pos` past them. Used before answering a liveness
+/// probe so the pong's position reflects output the client has actually been
+/// sent. A `Lagged`/`Closed` on the non-blocking drain is left for the main
+/// `rx.recv()` loop to handle (resync or teardown); flushing what was already
+/// pulled is still correct and gap-free.
+async fn flush_available<S>(
+    sink: &mut S,
+    rx: &mut tokio::sync::broadcast::Receiver<OutputChunk>,
+    snap_pos: u64,
+    sent_pos: &mut u64,
+) -> Result<(), ()>
+where
+    S: SinkExt<Message> + Unpin,
+{
+    let mut drained: Vec<OutputChunk> = Vec::new();
+    let mut queued = 0usize;
+    while queued < OUTBOUND_COALESCE_CAP {
+        match rx.try_recv() {
+            Ok(next) => {
+                queued += next.data.len();
+                drained.push(next);
+            }
+            Err(_) => break,
+        }
+    }
+    if drained.is_empty() {
+        return Ok(());
+    }
+    let (frame, end) = coalesce(snap_pos, &drained);
+    if end > *sent_pos {
+        *sent_pos = end;
+    }
+    if !frame.is_empty() {
+        sink.send(Message::Binary(frame.into()))
+            .await
+            .map_err(|_| ())?;
+    }
+    Ok(())
+}
+
 /// Re-synchronise a lagging client in-band, on the same socket, instead of
 /// dropping it and forcing a reconnect + replay (a cascade under load).
 ///
@@ -315,7 +356,11 @@ async fn handle_socket(
     };
 
     let (mut sink, mut stream) = socket.split();
-    let (control_tx, mut control_rx) = mpsc::unbounded_channel::<ServerControl>();
+    // Carries liveness-probe ids from the inbound task to the outbound task.
+    // The pong is *formed* by the outbound task with the position it has
+    // actually streamed, not by the inbound task with `total_written` — see the
+    // ping branch of the outbound select (WI-126).
+    let (ping_tx, mut ping_rx) = mpsc::unbounded_channel::<u64>();
 
     // Subscribe BEFORE snapshotting so no broadcast chunks are missed in the gap.
     let mut rx = session.subscribe();
@@ -386,10 +431,11 @@ async fn handle_socket(
                             let _ = writer_session.resize(cols, rows);
                         }
                         Ok(ClientControl::Ping { id }) => {
-                            let _ = control_tx.send(ServerControl::Pong {
-                                id,
-                                pos: writer_session.scrollback_position(),
-                            });
+                            // Hand the probe to the outbound task; it answers
+                            // with the position it has actually sent this
+                            // socket, after flushing queued output, so the pong
+                            // can never report the server ahead of the client.
+                            let _ = ping_tx.send(id);
                         }
                         Ok(ClientControl::Auth { .. }) => {
                             // Already authenticated; ignore further auth frames.
@@ -427,9 +473,23 @@ async fn handle_socket(
 
         loop {
             tokio::select! {
-                Some(control) = control_rx.recv() => {
+                Some(ping_id) = ping_rx.recv() => {
+                    // Flush anything already queued for this socket, then answer
+                    // the probe with `sent_pos` — the byte offset actually
+                    // written here. Because the pong is ordered AFTER those
+                    // chunks on the wire and carries only what has been sent, it
+                    // can never report the server ahead of what the client has
+                    // received, so a bursty session no longer trips a spurious
+                    // recycle (WI-126).
+                    if flush_available(&mut sink, &mut rx, snap_pos, &mut sent_pos)
+                        .await
+                        .is_err()
+                    {
+                        break;
+                    }
+                    let pong = ServerControl::Pong { id: ping_id, pos: sent_pos };
                     if sink
-                        .send(Message::Text(serde_json::to_string(&control).unwrap().into()))
+                        .send(Message::Text(serde_json::to_string(&pong).unwrap().into()))
                         .await
                         .is_err()
                     {