Skip to content

Sync wedges in UpdatingPivot after early restart with no peers — forkchoice hashes not persisted on the unresolvable-header path #11992

Description

@stdevMac

Summary

A node restarted before its first sync pivot is resolved can get permanently stuck in the UpdatingPivot stage. On the unresolvable-header path the forkchoice (finalized/head) hashes delivered by engine_forkchoiceUpdated are never persisted, so after a restart the re-sent FCU never feeds StartingSyncPivotUpdater and the pivot never resolves. Sync wedges silently — no crash, no error log that points at the cause.

Setup / when it happens

  • master (found via the post-merge smoke fuzz tests that kill/restart the EL early in sync).
  • Most reproducible when the node restarts with zero peers before the first pivot update.

Root cause

BeaconSync.GetFinalizedHash has no fallback to a persisted BlockTree.FinalizedHash. The finalized/head hashes from ForkchoiceUpdated live only in memory while the first pivot is being resolved. If the process restarts before StartingSyncPivotUpdater completes the first pivot:

  1. The in-memory forkchoice state is lost.
  2. On restart the CL re-sends FCU, but the unresolvable-header path does not persist those hashes.
  3. StartingSyncPivotUpdater never receives a usable finalized hash, so the pivot is never set and the node sits in UpdatingPivot indefinitely.

Fix (PR linked below)

  • Persist forkchoice hashes on the unresolvable-header path so they survive a restart and StartingSyncPivotUpdater can resume (ForkchoiceUpdatedHandler, BeaconSync, StartingSyncPivotUpdater).
  • Distinguish "no FCU data yet" from "unresolvable finalized" in the pivot-wait log so the wedge is diagnosable.
  • Regression coverage: StartingSyncPivotUpdaterTests, EngineModuleTests.Synchronization.

Related (NOT duplicates)

Found during 1.38 smoke-test triage. Must not be allowlisted in the smoke harness — it would hide a real sync-recovery wedge.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions