Broker deadlock is undetected and unrecoverable: fleet sidecar hello-timeout storm, then full HTTP hang requiring manual SIGKILL

[khaliqgant]

Summary

The native agent-relay-broker (9.2.1) deadlocked in stages on 2026-07-02, and Pear had no detection or recovery path. The local-fleet sidecar (#390) amplified the wedge into a permanent reconnect storm. Recovery required a human to SIGKILL the broker pid.

A backoff fix for the storm is already on fix/integration-symlink-stale-mirror (reconnectDelayMs in src/main/pear-fleet-node.ts). This issue tracks everything else in one place: the underlying broker deadlock (Pear ships the binary) and the durability gap — Pear cannot detect or self-heal a wedged broker.

Symptom (user-visible)

Console spam repeating every ~5s, indefinitely:

[broker] Local fleet node registration still pending for project 3f644965-…: Local fleet node registration timed out after 2000ms
[broker] Pear fleet sidecar disconnected: Pear fleet sidecar hello request timed out after 5000ms; reconnecting
[broker] Pear fleet sidecar disconnected: Pear fleet sidecar hello request timed out after 5000ms; reconnecting
…

Later, after the wedge progressed, app restart itself broke:

[broker] Failed to start for project 3f644965-…: [DOMException [TimeoutError]: The operation was aborted due to timeout]
Error occurred in handler for 'broker:start': [DOMException [TimeoutError]: The operation was aborted due to timeout]

Diagnostics collected

Environment: macOS, Pear dev build, bundled broker node_modules/@agent-relay/broker-darwin-arm64/bin/agent-relay-broker → agent-relay-broker 9.2.1, project 3f644965-b3bf-4fc2-bab3-eea899e35288 (cwd …/AgentWorkforce/cloud), broker pid 94250 on 127.0.0.1:61886.

1. Direct WS probe of /api/fleet/ws reproduced the hang deterministically:

• Workspace key (rk_live_…) → 401 at upgrade (expected; fleet WS wants the broker instance key).
• Broker instance key (br_… from <cwd>/.agentworkforce/relay/connection.json, field api_key) → upgrade succeeds, hello frame sent, zero bytes ever come back. Exactly matches the 5s client timeout.

12754 OPEN
12755 SENT hello probe_…_1
19752 TIMEOUT-EXIT   ← no reply in 7s

Pear's client (pear-fleet-node.ts) is behaviorally identical to the reference fleet-sidecar.js shipped in the agent-relay CLI — this is not a Pear protocol bug.

2. Broker log showed the fleet node-control slot had worked earlier the same day, then wedged:

• Regular relay_broker::runtime::fleet: fleet node control connected; node delivery active / fleet node control disconnected cycles from 03:44 → 15:56 (last success), then nothing but client-side timeouts.
• All-day WARN flood, hours before the hang (possible lead):

relay_broker::node_control: invalid fleet node ws frame … error=unknown variant `context.update`,
expected one of `deliver`, `action.invoke`, `ping`, `reply`, `error` at line 1 column 30

Something is sending fleet frames the 9.2.1 broker can't parse; the wedge followed hours of that churn.

3. Staged degradation. By the time the app was restarted, the wedge had spread beyond the fleet slot — every HTTP endpoint hung (/api/status, /api/spawned: 5s timeout, zero bytes). That's why broker:start failed: the reuse path probes the existing broker over HTTP and hung.

4. Full deadlock. The process ignored SIGTERM for 8+ seconds and required SIGKILL — the runtime was deadlocked, not just slow.

5. Socket leak from the storm. The pre-fix 5s reconnect ceiling left 32 half-closed (CLOSED) sockets piled on the broker process (lsof -iTCP:61886), on top of the log spam.

6. Recovery worked first try once the pid was killed — which is what makes the missing watchdog the whole gap:

[broker] Existing broker connection is not reusable …: Stale broker connection file (…/connection.json) points to dead pid 94250.
[broker] Started successfully for project: 3f644965-…
[broker] Pear fleet node "pear-3f644965-…-local-fleet" registered with 9 capabilities.

Fresh broker, agents respawned from persisted state, fleet handshake completed in one shot, zero hello timeouts since.

Root cause

1. Relay broker deadlock (raised here, see section below): the bundled agent-relay-broker 9.2.1 deadlocked — fleet node-control slot first, then the entire HTTP API, then SIGTERM-immune.
2. Pear amplifier (fixed): runPearFleetSidecarLoop reconnected on a 5s ceiling even when it had never once registered → permanent storm against a wedged broker. Fixed by reconnectDelayMs(attempt, registeredOnce): 60s ceiling until first successful registration, 5s ceiling after.
3. Pear durability gap: no wedge detection, no auto-restart. The broker was known-sick for hours (never-registered sidecar reconnecting) before the full hang, and recovery still required a human.

Relay broker deadlock (the underlying bug, tracked in this issue)

Pear ships and owns this broker binary (node_modules/@agent-relay/broker-darwin-arm64, pinned via @agent-relay/* in package.json), so the deadlock is raised here rather than split into another tracker. Evidence for whoever picks this up (and for deciding when to bump the pinned broker):

• Trigger lead: all-day WARN flood of invalid fleet node ws frame … unknown variant context.update`` in relay_broker::node_control — some fleet client is sending a frame type the 9.2.1 broker doesn't know. The wedge followed hours of that churn. Identify who sends `context.update` (newer `@agent-relay/fleet` client? relayfile-mount?) and whether the broker leaks per-rejected-frame resources.
• Failure shape: staged, not instant — fleet node-control slot went silent first (accepts WS upgrade, never answers hello), hours later every HTTP endpoint hung (/api/status, /api/spawned: 5s timeout, zero bytes), and the process ignored SIGTERM for 8+ s (required SIGKILL). Points at a poisoned lock / blocked executor in the fleet path that eventually starves the whole runtime.
• Single node-control slot: the broker holds one fleet control slot; log shows connected → disconnected churn all day before the wedge. Slot lifecycle under repeated reconnects is the suspect area.
• Broker log for the incident: ~/Library/Logs/agentworkforce/relay/pear-3f644965-….log.2026-07-02 (03:44 first fleet connect, 15:56 last successful, then silence).

Pear-side acceptance for this part: bump the pinned @agent-relay/* once a fixed broker ships, and add a regression check that the fleet handshake completes against the bundled binary.

Proposed fixes (Pear)

1. Broker watchdog with auto-restart in BrokerManager (src/main/broker.ts):
   • Probe /api/status every ~30s with a 5s timeout.
   • 3 consecutive failures → declare session wedged → SIGTERM, short grace, SIGKILL → let the existing start path recover (stale-pid detection → fresh broker → agent respawn already works, proven above).
   • Restart budget (e.g. 3/hour); when exceeded, stop looping and surface loudly in the UI.
2. Feed the fleet sidecar's never-registered state into the watchdog as an early-warning signal instead of only logging it — it flagged the wedge hours before the HTTP API died.
3. Surface broker/fleet health in the UI (status badge) so a wedged broker isn't just console noise.

Notes

• The 2000ms LOCAL_FLEET_REGISTRATION_TIMEOUT_MS in broker.ts is shorter than the 5000ms hello timeout, so a single "registration still pending" WARN can appear on healthy-but-slow startups. Benign (sidecar keeps registering in the background), but worth knowing when reading logs.
• Backoff fix + unit tests: src/main/pear-fleet-node.ts / src/main/pear-fleet-node.test.ts on fix/integration-symlink-stale-mirror (7/7 passing).

🤖 Generated with Claude Code