Skip to content

CLI: drive attach automatically resumes after broker or PTY restart #1419

Description

@willwashburn

Problem

A long-lived agent-relay ... attach --mode drive session does not survive a broker or PTY transport restart. The client reports a closed drive input stream (observed as drive input stream is closed) and exits or becomes unable to forward input; the human must manually attach again.

This makes broker recovery incomplete: the worker process or PTY may return, but its human/lead drive session does not, queued messages remain undelivered until another action repairs the state, and the session may stay idle until manually woken.

Reported by Will, relayed through voice. A chief-side shell auto-reattach loop in chief#6 is an interim operational workaround; this issue tracks the durable platform behavior.

Verified current boundary

No existing issue matched a search for drive attach restart/stream-close/resume behavior.

The current CLI establishes a single attach lifecycle in packages/cli/src/cli/lib/attach-drive.ts:

  • runDriveSession resolves the broker connection once, switches inbound delivery to auto_inject, seeds pending/replay state, and enters one runDriveSessionLoop.
  • The loop opens one event WebSocket and one SDK PTY input stream.
  • An event WebSocket error or abnormal close calls finish(1); even normal close settles the session instead of attempting recovery.
  • PTY input-stream send rejection is logged and predictive echo is rolled back, but the stream is not reopened.
  • Final teardown restores the worker's prior inbound-delivery mode, which is correct for intentional detach but wrong as the only response to a transient broker/PTY restart.

The broker already has useful repair primitives: durable event replay/cutoffs, pending-message inspection and flush, inbound-delivery mode CAS/revision handling, PTY snapshot/resize, and flush_injections support while an interactive hold is active. The missing piece is a resumable attach lifecycle that composes them after transport replacement.

Exit-contract defect

Installed Agent Relay 11.3.1 source at dist/cli/lib/attach-drive.js confirms two non-detach failures are indistinguishable from a clean detach to a supervisor:

  • Ctrl+C (0x03) intentionally calls finish(0).
  • Abnormal event-WebSocket close such as 1006 calls finish(1).
  • Event-WebSocket close 1000 or 1005 calls finish(0) even when no user requested detach. A broker restart can therefore look like clean success instead of a resumable transport interruption.
  • If only the PTY input stream dies while the event WebSocket remains healthy, keystrokes reject with input stream send failed: PTY input stream is closed. The client logs and rolls back predictive echo but stays in a degraded session until the human presses Ctrl+C, after which the process exits 0.

This is why the chief#6 supervision loop cannot cover the exact symptom Will reported: the wrapper can restart a non-zero child, but the CLI either reports success or never exits until a human converts the degraded state into a clean detach.

The durable contract must distinguish user intent from transport state. A clean WebSocket close without an explicit detach is resumable. Loss of the input stream under an active drive session is a liveness failure that must trigger recovery immediately; if recovery is exhausted, the CLI must exit non-zero. Exit 0 remains reserved for a genuine clean/user-requested end of the logical drive session.

Expected behavior

When the event or PTY input stream closes for a reason other than an intentional user detach, the drive client should remain alive and automatically establish a new attach session against the restarted broker/PTY. After reconnect it must repair the seat before returning control:

  1. re-read connection state so a restarted broker can change URL, port, or API key;
  2. reopen the event stream and PTY input stream with bounded backoff and cancellation;
  3. resynchronize terminal state from a fresh snapshot and reassert resize ownership;
  4. resume durable events from a safe cutoff without duplicating already-rendered output or pending counts;
  5. reassert the intended inbound-delivery mode, deliver messages queued during the outage, and wake the recovered session so work continues without a human keystroke;
  6. keep the local terminal usable throughout recovery and clearly show reconnecting/recovered state.

An explicit detach, terminating signal, agent release/exit, or unrecoverable authorization/identity error must stop the loop. Transient transport loss must not be treated as user intent, and prior delivery mode should be restored only when the logical drive session actually ends.

Acceptance criteria

  • Event-WebSocket close caused by broker restart automatically reconnects and resumes output/input, including close codes 1000/1005 when no explicit detach occurred.
  • PTY input-stream close or send failure is detected as a session-liveness event and automatically replaces that stream; subsequent human input reaches the recovered PTY.
  • If automatic recovery from either stream failure exhausts its policy, the drive command exits non-zero so an external supervisor can act.
  • Explicit user detach remains a clean exit and never reconnects.
  • Broker connection-file changes are picked up during recovery.
  • Messages queued during the interruption are delivered exactly once after resume, and the session receives the required wake/repair signal.
  • Durable replay, snapshot reconciliation, predictive echo, status rendering, and resize ownership do not duplicate output or leave stale listeners/timers across retries.
  • Reconnect uses bounded exponential backoff with visible status and remains cancellable.
  • Tests cover broker restart, WebSocket 1000/1005 without detach, PTY-only restart, queued-message repair, connection rotation, duplicate prevention, user detach, recovery exhaustion/non-zero exit, and unrecoverable failure.

Priority and sequencing

Normal priority. This is queued behind the current credential-safety amendment/review train. It does not authorize a release; publishing remains human-gated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions