[P0] ACKing volatile manual-flush messages causes data loss on broker restart

[willwashburn]

Description

When an inbound Relaycast delivery reaches a worker in manual_flush mode, the broker stores it only in the in-memory pending queue but immediately sends a cumulative delivery.ack.

If the broker exits before the queue is flushed, the pending message disappears. Relaycast has already terminalized the delivery and marked it read, so it is excluded from reconnect replay. The message is silently lost.

This crosses the durability boundary too early: accepted into a volatile local queue is currently treated as durably surfaced.

Steps to Reproduce

1. Connect a Relay broker to Relaycast and register a worker.
2. Set the worker to manual_flush through PUT /api/spawned/{name}/delivery-mode.
3. Send the worker a message through Relaycast.
4. Confirm GET /api/spawned/{name}/pending contains the message.
5. Observe that the broker has already emitted delivery.ack { up_to_seq: N }.
6. Terminate the broker before calling /flush.
7. Restart and reconnect the broker with the same agent identity.
8. Observe that the local pending queue is empty and Relaycast does not replay sequence N because it is already terminal and read.

A deterministic integration test can reproduce this by recreating broker state after step 5 and invoking Relaycast reconnect replay.

Expected Behavior

A Relaycast delivery is not cumulatively acknowledged until either:

• it has been successfully injected into the worker, or
• it has been committed to broker-owned storage that survives process failure and is restored before reconnect.

Actual Behavior

The delivery is acknowledged while its only actionable copy is in broker memory. A broker crash, restart, or queue eviction can permanently discard it.

Impact

This is silent message loss in a mode specifically intended to hold messages for later delivery. Relaycast also marks the message read, so normal inbox and replay mechanisms cannot repair the loss.

Evidence

handle_fleet_deliver treats any successful surface_fleet_deliver result as delivered, commits the local cursor, and emits the cumulative ACK:

• handle_fleet_deliver

For manual_flush, surface_fleet_deliver returns Ok(()) immediately after appending to delivery_states; no PTY injection or durable write has occurred:

• surface_fleet_deliver

The broker documents that delivery mode and its pending queue have no disk persistence:

• WorkerDeliveryState
• Runtime initialization

On receipt of the ACK, Relaycast advances the persistent cursor, changes the rows to acked, and marks their messages read. Reconnect replay then excludes the delivery:

• ackDeliveriesUpToSeq
• deliverPendingToNode

Environment

• Relay main at 58ec87cc2699877fe5bf3b8c54a38be13504e0be
• Relaycast main at e905f3acfb790279ee2eb964cf711f29de87f718
• Introduced by the Relaycast-only delivery path in fix(broker): route all message delivery through Relaycast, no local bypass #1219

Acceptance Criteria

• A message held by manual_flush survives broker restart or remains replayable from Relaycast.
• The cumulative ACK cursor never advances past a delivery whose only copy is in the volatile pending queue.
• A successful explicit flush advances the ACK only after the message crosses the chosen durability or injection boundary.
• Injection or persistence failure withholds the ACK and leaves the delivery retryable.
• Queue overflow cannot discard a delivery that Relaycast has already terminalized.
• Auto-inject behavior remains unchanged: successful injection is acknowledged and failed injection is not.
• Add an end-to-end test covering deliver -> manual queue -> broker restart -> reconnect/replay -> flush.
• Add a multi-sequence test proving a cumulative ACK cannot skip a held delivery.

Related

• Design RFC: Fleet Delivery — Agents, Nodes, and Reliable Messaging #1056
• broker: unify inbound dispatch behind always-queue, mode = drain policy #885
• broker: new / view / drive / relay CLI primitives for the broker-owned agent model #864
• fix(broker): route all message delivery through Relaycast, no local bypass #1219