fix(sync): retry missing content downloads on every successful sync

[vladimirlogachev]

Description

A record can arrive from a peer that does not have the record's content: the sender is a relay that received the record but never fetched the bytes (its download policy forbids it, or it simply hasn't downloaded yet). The receiver then parks the content hash in missing_hashes — correctly, since the only known peer provably lacks the bytes. The problem is what unparks it: only a gossip Op::ContentReady broadcast from a neighbor that finished downloading that content. Gossip broadcasts are best-effort, so if that one message is lost — or nobody else ever downloads the content — the hash stays parked forever. Re-running reconciliation does not help: the record is already present, so no new InsertRemote fires and no download is ever triggered. The node keeps completing successful syncs with peers that do have the bytes, and still never asks them.

We hit this downstream at roughly a 0.7% failure rate (1 in 303 runs) in a four-node last-writer-wins scenario, where a record relayed ahead of its content permanently starved on one device.

Changes:

• missing_hashes entries are keyed (NamespaceId, Hash) instead of Hash, so retries are scoped to the namespace being synced and PendingContentReady attribution stays exact when several namespaces want the same hash.
• On every successful sync (on_sync_finished, both origins), parked hashes of that namespace are retried against the just-synced peer via the existing start_download(…, only_if_missing = true) path — the peer is a fresh provider candidate; if it lacks the bytes the download fails and the hash parks again, no worse than before.
• Content already being downloaded when a sync finishes gets the peer registered as a provider too, plus a one-shot retry_after_failure mark: ProviderNodes::find_providers snapshots the provider set when a download starts, so a provider registered mid-flight is invisible to the running attempt. If that attempt fails, the marked hash is re-queued once with the enriched provider set. The mark is consumed on use and cleared on success, so a repeated failure without a new sync cannot loop.
• start_download is split into provider registration plus a node-less queue_download, so the failure-path retry can re-queue without inventing a provider argument.
• Regression test sync_fetches_parked_content_from_later_sync_peer: writer → relay with DownloadPolicy::NothingExcept([]) (record without bytes), writer leaves the document, receiver imports from the relay alone and parks the hash, writer returns, receiver completes an empty sync with it — the content must arrive. Red before this fix (starves for the full timeout), green after (relay leg included: the mid-flight window above is exactly what the test caught in the first version of this fix). The writer's leave/return is what keeps the red arm deterministic — otherwise gossip may connect receiver and writer directly and mask the starvation.

Breaking Changes

None. All touched state is private to the live actor; public APIs and events are unchanged (nodes may now emit ContentReady in situations where they previously starved).

Notes & open questions

• The retry is deliberately bounded: parked hashes are retried once per finished sync, and an in-flight download is re-queued at most once per mark. An alternative would be a live (non-snapshotting) provider stream in find_providers, which would make retry_after_failure unnecessary — larger change, left out of scope.
• The sync-finished retry runs for both Origin::Connect and Origin::Accept: an accepted peer is just as good a provider candidate as one we dialed.

Change checklist

• Self-review.
• Documentation updates following the style guide, if relevant.
• Tests if relevant.
• All breaking changes documented.

[cbenhagen]

This might be related: #88