Summary
A node restarted before its first sync pivot is resolved can get permanently stuck in the UpdatingPivot stage. On the unresolvable-header path the forkchoice (finalized/head) hashes delivered by engine_forkchoiceUpdated are never persisted, so after a restart the re-sent FCU never feeds StartingSyncPivotUpdater and the pivot never resolves. Sync wedges silently — no crash, no error log that points at the cause.
Setup / when it happens
- master (found via the post-merge smoke fuzz tests that kill/restart the EL early in sync).
- Most reproducible when the node restarts with zero peers before the first pivot update.
Root cause
BeaconSync.GetFinalizedHash has no fallback to a persisted BlockTree.FinalizedHash. The finalized/head hashes from ForkchoiceUpdated live only in memory while the first pivot is being resolved. If the process restarts before StartingSyncPivotUpdater completes the first pivot:
- The in-memory forkchoice state is lost.
- On restart the CL re-sends FCU, but the unresolvable-header path does not persist those hashes.
StartingSyncPivotUpdater never receives a usable finalized hash, so the pivot is never set and the node sits in UpdatingPivot indefinitely.
Fix (PR linked below)
- Persist forkchoice hashes on the unresolvable-header path so they survive a restart and
StartingSyncPivotUpdater can resume (ForkchoiceUpdatedHandler, BeaconSync, StartingSyncPivotUpdater).
- Distinguish "no FCU data yet" from "unresolvable finalized" in the pivot-wait log so the wedge is diagnosable.
- Regression coverage:
StartingSyncPivotUpdaterTests, EngineModuleTests.Synchronization.
Related (NOT duplicates)
Found during 1.38 smoke-test triage. Must not be allowlisted in the smoke harness — it would hide a real sync-recovery wedge.
Summary
A node restarted before its first sync pivot is resolved can get permanently stuck in the
UpdatingPivotstage. On the unresolvable-header path the forkchoice (finalized/head) hashes delivered byengine_forkchoiceUpdatedare never persisted, so after a restart the re-sent FCU never feedsStartingSyncPivotUpdaterand the pivot never resolves. Sync wedges silently — no crash, no error log that points at the cause.Setup / when it happens
Root cause
BeaconSync.GetFinalizedHashhas no fallback to a persistedBlockTree.FinalizedHash. The finalized/head hashes fromForkchoiceUpdatedlive only in memory while the first pivot is being resolved. If the process restarts beforeStartingSyncPivotUpdatercompletes the first pivot:StartingSyncPivotUpdaternever receives a usable finalized hash, so the pivot is never set and the node sits inUpdatingPivotindefinitely.Fix (PR linked below)
StartingSyncPivotUpdatercan resume (ForkchoiceUpdatedHandler,BeaconSync,StartingSyncPivotUpdater).StartingSyncPivotUpdaterTests,EngineModuleTests.Synchronization.Related (NOT duplicates)
HeadersSyncFeedin the same headers/pivot subsystem, but a crash path, not this silent forkchoice-hash non-persistence wedge.Found during 1.38 smoke-test triage. Must not be allowlisted in the smoke harness — it would hide a real sync-recovery wedge.