Summary
On a full-history index, any confirmed slot gap that occurs before startup finishes causes a hard panic! in self_healing::fill_gaps, which kills the indexer and triggers a full snapshot re-bootstrap. Because the bootstrap itself takes several hours on a large database, the probability of at least one gap occurring during that window is effectively 100%, so the node never completes startup — it crash-loops, re-downloading the full snapshot every cycle.
Environment
- cloudbreak v0.1.2
- Solana mainnet, full index (all programs), ~1.13B snapshot accounts, ~108 GB full snapshot
- Postgres 16, ZFS, dedicated host
- Yellowstone gRPC geyser source
Observed behavior
After a clean bootstrap that ran for ~6.5h and successfully passed every post-snapshot phase (snapshot processed → cluster → dedup → clean_up_closed_accounts (12.7M) → create_database_indexes), a single one-slot confirmed gap aborted everything during the index-build phase:
ERROR self_healing: Confirmed slot gap: last received 428826599 - new slot 428826601 - parent_new_slot 428826600 - queuing 1 slots for repair
WARN finalizer: Finalization paused (gap fill in progress)
WARN service_health: Service marked UNHEALTHY (reason added: GapFill)
ERROR cloudbreak_index::modules::self_healing: Confirmed slot gap detected before startup finished; cannot repair gaps during startup
thread 'tokio-runtime-worker' (18) panicked at crates/index/src/modules/self_healing.rs:199:21:
Confirmed slot gap detected before startup finished; cannot repair gaps during startup
index-1 exited with code 1 (restarting)
On restart the process re-downloads the full ~108 GB snapshot from scratch, re-runs the multi-hour post-processing, and — if another gap lands before it finishes — panics again. This is an unrecoverable loop.
Root cause
This is by design, per the code’s own comments:
- check_slot_gap (crates/index/src/modules/self_healing.rs) pauses the finalizer the moment a confirmed gap is detected, noting: “a gap discovered during startup pauses the very worker that completes startup; that is intentionally unsupported and fill_gaps fails fast in that case.”
- fill_gaps (lines ~186-202) then checks SnapshotProcessingState == FinishedAndCleanedUp; if startup is not finished it logs and panic!s, since the paused finalizer can never complete startup, so the gap can never be repaired.
The implicit assumption is that startup completes faster than any gap can occur. That holds for small databases, but on a full-history index the post-snapshot pipeline (dedup, clean_up_closed_accounts, create_database_indexes) is a multi-hour serial phase. Throughout it, finalization is paused and the finalize-slot-buffer-size buffer stays pinned at its bound (we observed it stuck at ~1031 entries for ~3h). A full buffer back-pressures the gRPC stream; the upstream geyser then drops a slot, which surfaces as a confirmed gap — and the panic fires. So on a large DB the gap is not an edge case, it is the expected outcome.
Impact
A full-history deployment cannot reach a healthy state at all: it crash-loops on bootstrap, repeatedly re-downloading the full snapshot.
Suggested fix
The design needs to tolerate gaps discovered during startup rather than fail fast. Some options:
- Defer startup gaps instead of pausing/panicking. When a confirmed gap is detected and startup is not yet FinishedAndCleanedUp, queue the gap (and its boundary) but do not pause the finalizer, so startup can complete. In fill_gaps, replace the panic! with a wait/continue until startup finishes, then repair the queued gaps through the normal path.
- Don’t gate finalization on the full post-snapshot pipeline, so the finalize buffer drains during startup and never back-pressures the gRPC stream (removing the most common gap trigger).
Summary
On a full-history index, any confirmed slot gap that occurs before startup finishes causes a hard panic! in self_healing::fill_gaps, which kills the indexer and triggers a full snapshot re-bootstrap. Because the bootstrap itself takes several hours on a large database, the probability of at least one gap occurring during that window is effectively 100%, so the node never completes startup — it crash-loops, re-downloading the full snapshot every cycle.
Environment
Observed behavior
After a clean bootstrap that ran for ~6.5h and successfully passed every post-snapshot phase (snapshot processed → cluster → dedup → clean_up_closed_accounts (12.7M) → create_database_indexes), a single one-slot confirmed gap aborted everything during the index-build phase:
On restart the process re-downloads the full ~108 GB snapshot from scratch, re-runs the multi-hour post-processing, and — if another gap lands before it finishes — panics again. This is an unrecoverable loop.
Root cause
This is by design, per the code’s own comments:
The implicit assumption is that startup completes faster than any gap can occur. That holds for small databases, but on a full-history index the post-snapshot pipeline (dedup, clean_up_closed_accounts, create_database_indexes) is a multi-hour serial phase. Throughout it, finalization is paused and the finalize-slot-buffer-size buffer stays pinned at its bound (we observed it stuck at ~1031 entries for ~3h). A full buffer back-pressures the gRPC stream; the upstream geyser then drops a slot, which surfaces as a confirmed gap — and the panic fires. So on a large DB the gap is not an edge case, it is the expected outcome.
Impact
A full-history deployment cannot reach a healthy state at all: it crash-loops on bootstrap, repeatedly re-downloading the full snapshot.
Suggested fix
The design needs to tolerate gaps discovered during startup rather than fail fast. Some options: