Skip to content

fix(protocol): a retry that the direct carriers refuse is never offered to the mesh #541

Closed

Description

@bahdotsh

What happens

A direct message that a carrier accepted and then lost is never handed to the mesh. Every retry goes to direct carriers only, so the message waits for a direct link to its recipient even when a neighbour could carry it.

Found running examples/offline-first (#539). C was taken off the network A shared with it before C's service stopped, so A held a half-open peer stream to C for about 30 s until keepalive ended it. A's send went down that stream and registered a pending acknowledgement. Once the stream was gone, every retry was refused by the direct carriers and queued again. Minutes later, A showed pending_ack_count 1, retry_queue_size 1 and mesh transmissions 0, while B (which reaches both A and C) was up the whole time.

Where

process_retry_queue's Err(e) arm (crates/offline-protocol/src/protocol/mod.rs, around line 3896) re-enqueues the message and emits message_retrying. It never calls offer_to_mesh. Every mesh offer is in send.rs: handle_send_success when !can_reach_recipient, handle_send_failure, the park path, the handshake path, and the ACK path. All of these fire on the first send or on a park, never on a retry that fails.

Expected

A retry that no carrier takes is offered to the mesh, as the first send's failure is (handle_send_failure). The same holds for a retry a carrier accepts for a recipient it cannot reach, as handle_send_success does. Offering is additive, and the recipient's deduplicator absorbs a second copy.

How to reproduce

examples/offline-first, scenario 4, with DockerRunner.part_a_and_c changed to disconnect C from net-ab before stopping it. The documented workaround in the script stops C first, so A sees the stream close.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions