RFC: fail-closed provider reconnects and durable replay across hosted NodeDO

[willwashburn]

Status

Proposed. This is the design gate for AgentWorkforce/relay#1386. No production access or data mutation is authorized by this RFC.

Problem

A broker reconnect can leave hosted delivery deaf while the node and broker both report connected. A deterministic workerd reproduction against relaycast-cloud origin/main (e69994e, engine 6.3.0) passes 15/15 assertions across five consecutive runs and establishes four composing defects:

1. NodeDO treats a half-open socket as live indefinitely: upgrade no longer supersedes sockets globally (correct for multi-provider), but the alarm has no per-socket reaper and currently refreshes D1 liveness merely because any socket has readyState === OPEN.
2. A new broker instance receives provider_instance_conflict; the broker registers once, keeps the socket open, continues heartbeating, and reports connected before registration succeeds.
3. Provider-scoped frames are accepted on an unbound socket. agent.register therefore falls back to provider default, permanently routing that agent away from the actual broker provider.
4. relaycast-cloud does not mirror the engine's provider delivery-readiness contract, so relay:delivery-cursor-v1 is rejected and the broker cannot seed its durable delivery cursor before replay.

A socket write to the stale incumbent returns success and the row becomes delivered, even though the live broker never saw it. delivered remains replayable while seq > delivery_ack_seq, but only after a correctly bound reconnect; poisoned provider_name='default' rows are excluded from broker replay indefinitely.

Required invariant

A provider connection is not deliverable, live, or allowed to mutate provider/agent state until node.register has succeeded for that exact connection. Durable delivery may begin for an agent only after the accepted provider connection has received the agent's authoritative delivery_ack_seq and declared that identity ready.

WebSocket.readyState === OPEN is transport state, not liveness or delivery acknowledgement.

Proposed design

1. Engine: reject unbound provider traffic

For registries that expose connection-scoped provider binding, accept only node.register until providerNameForConnection(connectionId) resolves.

• Reject node.heartbeat, agent.register, inventory.sync, delivery.ack, action results, spawn, and deregistration with a stable provider_not_registered error.
• Do not write node liveness, agent rows, provider rows, inventory, acknowledgements, or invocation results for a rejected frame.
• Preserve legacy/default-provider compatibility only for adapters that do not implement connection-scoped provider binding; do not let that compatibility path weaken the hosted provider-aware path.
• Include retry guidance on provider_instance_conflict (a bounded retry_after_ms, or an equivalent stable contract) so clients do not guess the arbitration window.

This prevents future poisoned rows even if a client fails to gate its own traffic.

2. relaycast-cloud NodeDO: make socket state truthful

• Reap individual socket attachments whose lastSeen exceeds the provider/node liveness bound; close them and run the existing provider-scoped disconnect path. Do not refresh nodes.last_heartbeat_at because an open socket exists.
• Keep multi-provider semantics: only the stale attachment for that provider is removed; distinct provider names remain independent.
• Persist the engine's three delivery-readiness states in each socket attachment: unknown/not registered, immediate legacy readiness, or an agent-scoped ready set.
• Mirror setProviderDeliveryReadiness, markProviderAgentsDeliveryReady, and isProviderAgentDeliveryReady in the connection-scoped registry.
• Gate /send inside the DO using the frame's agent_id; return delivered:false unless that exact provider socket is ready for that identity. This keeps the cross-DO adapter async-safe without adding a per-delivery edge→DO readiness round trip before the existing /send call.
• A new provider instance starts with an empty ready set. Same-connection/same-instance re-registration may preserve it, matching the in-process adapter.

3. Relay broker: explicit registration state machine

The broker side is implemented in AgentWorkforce/relay but shares this protocol contract.

• Model Disconnected → TransportConnected → Registering → Ready; expose connected=true only in Ready after an accepted node.register reply.
• On provider_instance_conflict, retry registration with bounded backoff/retry guidance. Do not send heartbeat, inventory, agent registration, or delivery acknowledgement while unregistered.
• Clear readiness and health on any registration error or transport replacement; surface the current registration error and next retry time in health/status.
• Negotiate relay:delivery-cursor-v1, seed each agent from the authoritative delivery_ack_seq, and only then accept delivery for that identity.
• Log/metric every registration rejection and transition so an operator can distinguish socket connectivity from delivery readiness.

4. Existing poisoned data: human-run repair, not agent production access

Ship an auditable runbook with the code change. It must include:

• a dry-run query for agents whose provider_name='default', whose location node has an active broker provider, and for which no legitimate default provider owns the agent;
• counts and agent/node identifiers only—no tokens or message bodies;
• a backup/snapshot table or export sufficient to restore every changed row;
• a narrowly constrained update to provider_name='broker' and a post-check that queued/delivered rows above delivery_ack_seq become eligible for replay;
• rollback instructions and an explicit human approval/execution step.

Agents must not inspect or mutate production while preparing or validating this runbook.

Test plan / acceptance

Adopt the self-contained workerd reproduction from relay#1386 into relaycast-cloud and extend it so the final behavior proves:

• clean reconnect remains healthy;
• half-open incumbent is reaped or superseded after bounded arbitration, and the new provider becomes ready without operator restart;
• unbound heartbeat/agent/inventory frames are rejected and make no D1 mutations;
• no agent can be stamped provider_name='default' from a rejected broker socket;
• hosted node.register accepts the cursor capability;
• no deliver frame is sent before the agent's cursor-bearing reply and readiness transition;
• a frame written to a stale/unready socket is not reported delivered;
• delivered but unacked rows replay exactly once after readiness, in ascending sequence;
• two different provider names on one node remain concurrently valid;
• broker health is false between TCP connect and accepted registration, and registration conflict retry converges deterministically.

Run engine unit/integration tests, relaycast-cloud typechecks + unit + workerd integration suites, and Relay broker Rust tests. Each implementation PR requires two recorded current-head reviews under the organization policy.

Rollout order

1. Engine protocol hardening and tests; publish by a human.
2. relaycast-cloud engine bump + NodeDO readiness/reaper + adopted workerd regression; deploy by a human.
3. Relay broker registration retry/health truthfulness + cursor tests; release by a human.
4. Human dry-run and repair of confirmed poisoned rows, followed by read-only verification.

Roll forward in that order so old brokers fail safely under the server guard and new brokers do not advertise readiness against an old hosted path.