Conversation
A listener that detects a gap fetches the worker's dump with a reqwest client whose total timeout was 10 s, and a failed fetch was final: the live batch that revealed the gap then advanced the watermark past it, so the missed range was lost for good. On DeepSeek-V4.1-Flash engines (DP=2, 28M-token KV pools) a TreeDump is ~70 MB serialized inside EngineCore. When three indexer flavors started against 39 such pods, each engine got six concurrent dump requests, a third of them ran past 10 s, the indexer logged "error decoding response body", the engine logged BrokenPipeError in kv_events.py do_GET, and the affected listeners carried orphaned chains (ParentBlockNotFound floods) from then on. - --recover-timeout-secs (default 120) replaces the hard-coded 10 s; connect timeout stays short at 5 s. - --recover-concurrency (default 8): a process-wide semaphore around the download, so a fleet-wide (re)subscription queues instead of stampeding every engine. - A failed download is retried up to 3 times with 2 s / 4 s backoff before the gap is given up; the final failure keeps the "kv_recover request failed" substring ops already grep for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 37e1b1a)
github-actions
Bot
added
the
documentation
|
Validation (rustcheck:1.96 container,
Image built and pushed: |
|
Prod trial on frank/DeepSeek-V4.1-Flash kv-indexer:h24 (image kvtest-cbe6247d92, 2026-09-17 22:14 UTC): CrashLoopBackOff, 6 restarts in 10 min, exit 139, ~20 s after start while startup TreeDump recoveries were being applied (33 of 66 done, all successful, 3-26 s each). Last log line each time: This is libzmq 4.3.4 zeromq/libzmq#3596 / ai-dynamo#3937: a slow receiver that hits its receive HWM while heartbeating is enabled. Our SUB sockets have heartbeats on and the default RCVHWM (1000). In h24 mode every listener applies its 270k-block dump under the single Rolled h24 back to kvtest-d57bab4a29. Follow-up commit: unbounded RCVHWM on the SUB socket so input is never stopped. |
A listener blocked in /kv_recover (now up to 120 s with the gate) can let its SUB pipe hit the default RCVHWM of 1000. With heartbeats enabled libzmq 4.3.4 then aborts on `Assertion failed: _input_stopped` (zeromq/libzmq#3596, ai-dynamo#3937). Hit in prod on frank/DeepSeek-V4.1-Flash kv-indexer:h24: 6 restarts in 10 min while 66 startup TreeDumps were applied under the single H24Indexer mutex. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GitHub Actions
Failure
Two concurrent build-indexer-image.sh runs on different branches share the /cargo-target cache mount. git archive gives every file the commit mtime, so the second build saw a kv-router rlib the first build had just written, judged it fresh, and failed compiling the bindings against a struct from the other branch. Touch the tree after extraction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GitHub Actions
Failure
|
Re-rolled frank/DeepSeek-V4.1-Flash kv-indexer:h24 on kvtest-05fee893c6 (this branch @ 05fee89, includes the RCVHWM=0 fix) at 22:48 UTC: 0 restarts, no assertion, 68/68 startup recoveries completed as TreeDumps, 0 retries, 0 parent misses. The earlier image crashed ~20 s after start every time. Also in this push: |
1 participant
Cherry-pick of #41 onto the h24 line (stacked on #40, base
dp-rank-watch-h24). Two trivial both-sides-added conflicts inIndexerConfigand its CLI construction resolved by keeping both. Same change and tests; see #41 for the analysis and measurements.🤖 Generated with Claude Code