Rate-limit topo bootstrap repair heads - #5161
Open
pwojcikdev wants to merge 1 commit into
Open
Conversation
Repair heads re-scan the discovered range indefinitely and re-fire as soon as a page makes progress, so on a synced node they sweep the whole ledger at line rate, bounded only by the pre-check throughput. Without the topo index the pre-check probes every entry by hash, turning the sweep into a perpetual random-read storm that pegs the disk. Add `topo_scan.repair_rate_limit` (20 req/s) with a dedicated limiter in `topo_strategy`, applied via the repair gate so a throttled repair head never stalls the spearhead. Repair heads are also disabled entirely when the ledger has no topo index. Co-Authored-By: Claude Code <noreply@anthropic.com>
Test Results for Commit b9a9811Pull Request 5161: Results Test Case Results
Last updated: 2026-09-03 05:00:58 UTC |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With topo bootstrap enabled, a synced node ramps to 100% disk read time after a few minutes and stays there (observed on Windows + RocksDB).
Repair heads re-scan the already-discovered topo range indefinitely, and
topo_scan::maybe_advancezeroes a head's timestamp whenever a page makes progress, so the head re-fires as soon as the reply lands. On a synced node every repair page "makes progress", so thecooldownnever engages; when a band is exhausted the head is immediately re-armed on the same band. The only limits are the sharedtopo_rate_limit(500 req/s) and the 64-page pre-check queue, so the sweep runs as fast as the pre-check can read the ledger, i.e. it pegs the disk by construction.If the ledger has no topo index (
--populate_topo_indexnever run on an existing ledger),topo_strategy::precheckfalls back to probing every entry by hash, which turns that sweep into ~1.6k random block reads per page withfill_cache = false.Changes
bootstrap.topo_scan.repair_rate_limit(default 20 req/s,0= unlimited) and a dedicatednano::rate_limiterintopo_strategy. It is applied throughhead_gates.include_repair, so a throttled repair head never stalls the spearhead on the shared scan thread; the spearhead keeps its current pacing.ledger.flags.topo_indexis off. The spearhead still works there (its pages are bounded), but a perpetual sweep needs the sequential topology crawler to be sane.Testing
bootstrap.*,bootstrap_topo_scan.*,bootstrap_topo_blocks.*,toml.*pass. Not reproduced locally; the fix follows from the code path above and should be verified on the affected Windows/RocksDB setup.🤖 Generated with Claude Code