node up: transient Relaycast DNS failure exits resident nodes fleet-wide

[willwashburn]

Impact

A transient DNS/network failure during agent-relay node up can terminate every resident node started during the outage instead of tolerating or retrying the control-plane connection. On Friday 2026-07-31 at about 17:02Z, all 15 resident nodes on one machine exited and stayed down for roughly 42 hours until a manual kickstart.

This is a fleet-wide availability defect and belongs in the next patch-release discussion.

Machine evidence

All 15 ~/Library/Logs/*-node.log files contain the same failure sequence:

1. The broker startup handshake fails while registering the agent at https://cast.agentrelay.com/v1/agents.
2. The CLI reports Broker process exited with code 1 during initial handshake.
3. The telemetry flush also fails during the same network event with getaddrinfo ENOTFOUND i.agentrelay.com through PostHog fetchWithRetry -> sendBatch -> _flushRoute -> _flush.

The affected logs are agentrelay-com, burn, chief-app, chief, cloud, cmo, cpo, cso, hoopsheet, ladd, mobile, relay, relaycron-cloud, relayfile, and scout. Each has one failed Relaycast registration and the PostHog DNS signature. Live credentials are intentionally omitted here; the full stacks remain in the machine-local logs.

Observed with installed agent-relay 11.3.1 / posthog-node 5.47.3. Current source inspected at origin/main 41475b03be533d2c87b3a8e1147b553a2db04131.

Current failure path

The initial diagnosis was that the PostHog rejection itself killed the process. The artifacts do not support that as the primary exit path:

• crates/broker/src/runtime/session.rs retries only elapsed handshake timeouts. Any Ok(Err(...)) returned by the auth call immediately becomes failed to initialize relaycast session and exits the loop.
• crates/broker/src/relaycast/auth.rs returns non-auth/non-rate-limit registration errors immediately as failed registering agent with ... workspace key.
• crates/broker/src/runtime/init.rs awaits connect_relay(...).await?, so that transient transport error aborts broker initialization.
• The installed PostHog flushBackground() attaches a rejection handler that logs Error while flushing PostHog, and Relay's packages/cli/src/cli/telemetry/client.ts::shutdown() also catches shutdown failure. The PostHog stack is a simultaneous secondary failure and should remain best-effort, but the log sequence identifies Relaycast registration as the process-killing error.

The practical bug is therefore broader and more important: a transient control-plane DNS failure is treated like a definite registration rejection, and the persistent node has no supervisor/reconnect path that restores it when DNS returns.

Expected behavior

Telemetry is best-effort, and transient Relaycast transport failures must not cause a fleet-wide permanent outage.

Acceptance criteria

• Classify DNS failures, connection resets/refusals, and equivalent retryable transport errors separately from definite HTTP/auth failures.
• Retry transient registration/connect failures with bounded exponential backoff and jitter; do not blindly retry 4xx authentication, validation, or strict-name errors.
• Make retries safe against duplicate workspace/agent creation. Registration retries must reuse stable identity/idempotency semantics or reconcile the result before creating anything new.
• A persistent/background node either remains alive while the control plane is unavailable and reconnects when service returns, or is supervised/restarted automatically; it must not require a human kickstart after a short DNS blip.
• Preserve a useful degraded/readiness signal so node up --background and node status distinguish local API readiness from Relaycast connectivity.
• Add deterministic tests that inject a retryable DNS/transport failure followed by recovery and assert one eventual registration, no duplicate resources, and a live node process.
• Add a separate telemetry regression proving periodic and shutdown flush failures are caught and cannot change the command/node exit code.
• Add a concise [Unreleased - Patch] changelog entry describing the fleet-availability fix.

Release note

Treat this as a patch-release blocker/candidate: the failure affects all long-lived resident nodes on a machine at once and has already produced a multi-day fleet outage.