Draft: buzz-acp: channel wedges permanently when a session's agent child dies (-32603 retried into the same dead session until dead-letter)
Component: buzz-acp (crates/buzz-acp), observed at desktop-v0.5.3
Severity: channel outage until process restart, silent message loss
Summary
If a session's underlying agent child dies out-of-band (in our case: an operator killed claude session processes), the next session/prompt returns JSON-RPC -32603 "Internal error". buzz-acp classifies every AgentError as "application error — pipe intact", returns the agent to the pool, and crucially leaves SessionState.sessions[channel_id] pointing at the dead session:
pool.rs run_prompt_task error arm: if !matches!(e, AcpError::AgentError { .. }) { agent.state.invalidate(&source); } — the healthy-session exemption also covers dead sessions.
- The requeued batch (
MAX_RETRIES = 10, exponential backoff) re-enters run_prompt_task, hits the session cache (sessions.get(cid)), and re-prompts the same dead session every attempt until the batch dead-letters.
- There is no per-session liveness probe and no
session/load, so nothing ever heals the map short of restarting the whole process.
Real-world impact for us: two channels deaf for ~4 hours, 52 owner events dead-lettered.
Suggested fix (minimal)
Treat -32603 as session-invalidating: matches!(e, AcpError::AgentError { code, .. } if *code == -32603) → agent.state.invalidate(&source). The already-existing requeue path then misses the session cache and create_session_and_apply_model builds a fresh session on the next attempt. False positives are cheap (one extra session/new); false negatives wedge the channel. The proactive max_turns_per_session rotation already proves the invalidate-and-recreate idiom.
We are running this patch locally and can PR it if you'd take it.
Draft: buzz-acp: channel wedges permanently when a session's agent child dies (-32603 retried into the same dead session until dead-letter)
Component: buzz-acp (crates/buzz-acp), observed at desktop-v0.5.3
Severity: channel outage until process restart, silent message loss
Summary
If a session's underlying agent child dies out-of-band (in our case: an operator killed claude session processes), the next
session/promptreturns JSON-RPC-32603 "Internal error". buzz-acp classifies everyAgentErroras "application error — pipe intact", returns the agent to the pool, and crucially leavesSessionState.sessions[channel_id]pointing at the dead session:pool.rsrun_prompt_taskerror arm:if !matches!(e, AcpError::AgentError { .. }) { agent.state.invalidate(&source); }— the healthy-session exemption also covers dead sessions.MAX_RETRIES = 10, exponential backoff) re-entersrun_prompt_task, hits the session cache (sessions.get(cid)), and re-prompts the same dead session every attempt until the batch dead-letters.session/load, so nothing ever heals the map short of restarting the whole process.Real-world impact for us: two channels deaf for ~4 hours, 52 owner events dead-lettered.
Suggested fix (minimal)
Treat
-32603as session-invalidating:matches!(e, AcpError::AgentError { code, .. } if *code == -32603)→agent.state.invalidate(&source). The already-existing requeue path then misses the session cache andcreate_session_and_apply_modelbuilds a fresh session on the next attempt. False positives are cheap (one extrasession/new); false negatives wedge the channel. The proactivemax_turns_per_sessionrotation already proves the invalidate-and-recreate idiom.We are running this patch locally and can PR it if you'd take it.