standalone-indexer: bound, gate and retry /kv_recover downloads

[Shang-Pin]

Stacked on #39 (base is dp-rank-watch); this PR is the one commit 37e1b1a26b.

Why

A listener that detects a gap fetches the worker's dump with a reqwest client whose total timeout was hard-coded to 10 s, and a failed fetch was final: the live batch that revealed the gap then advanced the watermark past it, so the missed range was lost for good.

On DeepSeek-V4.1-Flash engines (DP=2, 28M-token KV pools) a TreeDump is about 70 MB, serialized inside EngineCore's ThreadingHTTPServer. When the three indexer flavors were (re)started against frank's 39 pods, each engine received six concurrent dump requests; about a third ran past 10 s (reality 25/78, routing 5/78, h24 28/78). The indexer logged kv_recover request failed ... error decoding response body, the engines logged BrokenPipeError in kv_events.py do_GET, and the affected listeners carried orphaned chains (ParentBlockNotFound floods) from then on. Recreating one indexer pod at a time halved the contention but still lost 17 recoveries.

What

• --recover-timeout-secs (default 120) replaces the hard-coded 10 s total timeout; connect timeout stays short (5 s).
• --recover-concurrency (default 8): a process-wide semaphore around the download, so a fleet-wide (re)subscription queues rather than stampeding every engine. Live events buffer in the ZMQ SUB socket meanwhile, as they already do during recovery.
• A failed download is retried up to 3 times with 2 s / 4 s backoff before the gap is given up; the final failure keeps the kv_recover request failed substring that ops grep for, with an explicit "giving up, batches lost".
• Unit tests for the backoff schedule and the unconfigured defaults; docs.md paragraph.

Verification

cargo test -p dynamo-kv-router --features kube-discovery standalone_indexer: 51 passed. cargo fmt --check on the crate only flags pre-existing drift in server.rs, untouched here.

🤖 Generated with Claude Code

[Shang-Pin]

Validation (rustcheck:1.96 container, cargo test -p dynamo-kv-router --features kube-discovery standalone_indexer):

• main line @ 37e1b1a: 51 passed, 0 failed (includes new recover_backoff_doubles_and_caps, recover_gate_defaults_when_unconfigured).
• cargo fmt --check: no drift from this change (only pre-existing drift in server.rs).

Image built and pushed: localhost:30500/dynamo-indexer:kvtest-37e1b1a26b. Not rolled out anywhere yet.

[Shang-Pin]

Prod trial on frank/DeepSeek-V4.1-Flash kv-indexer:reality and :routing (image kvtest-37e1b1a26b, 22:14 UTC): both pods healthy, 0 restarts. Indexer coverage of the engine's resident tree on the three (pod, dp_rank) units that were at 8-16% before the roll is now 100% (108/108, 113/113, 111/111 sampled chains fully present). reality_best moved from 0.83 to 0.86 = perfect within 12 min.

Added ae7403c: unbounded RCVHWM on the SUB sockets. The h24 flavor of this change (#42) crash-looped on libzmq 4.3.4's _input_stopped assertion (zeromq/libzmq#3596) once recoveries started succeeding en masse and listeners stalled long enough to HWM-stop their pipes; the main line ran fine but has the same exposure. Tests: 52 passed.