FlatDb computes a wrong state root and rejects a canonical block (consensus divergence) — chiado, intermittent

[stdevMac]

Summary

With FlatDb.Enabled=true on master, Nethermind intermittently computes a wrong state root and rejects a canonical block with InvalidStateRoot. A non-flat (patricia/HalfPath) control node accepts the same block. This is a consensus divergence specific to the flat-state backend.

Evidence

• Smoke run 27314819957 (chiado, FlatDb.Enabled=true) rejected canonical block 21574961 (0x0ba5905c2295b4b12ed035f8eace55f4ae9c288f55afcdcc7eaf7d6c6b9b9394, 0 txs, 8 non-zero withdrawals): computed root 0x78374a9a… vs canonical 0x12173c61….
• A patricia control node passed the same block.
• Intermittent. An offline full FlatTrieVerifier cross-check on a synced chiado DB (head 21576781, which had processed past the bad block cleanly on a different run) came back clean: 539,247 accounts / 35,390,325 slots, 0 mismatched, 0 missing. A 24h forward-processing soak on two fresh chiado nodes did not re-trigger it. So a synced DB can be divergence-free; the fault must be caught on a fresh sync that hits the right window.

Where it likely is

Flat-backend reads are not hash-verified the way trie-node reads are, so a wrong flat value surfaces only as a silent wrong state root. Static analysis points at the SnapshotCompactor self-destruct / storage-clear merge when folding per-block snapshots into a compacted snapshot:

• the storage-slot clear keys on Address, while the storage-trie-node clear keys on the account-path hash and is gated on !isNewAccount, with asymmetric self-destruct-marker semantics (TryAdd(addr, true) for new-account vs [addr] = false for non-new).
• the under-tested case is self-destruct-then-recreate within a single compaction window (e.g. a CREATE2 redeploy), for which there is no regression test — SnapshotCompactorTests only covers destroy-with-storage and destroy-new-account.

The per-instance persisted compaction offset (#11756) is likely a trigger, not the cause: it makes compaction-window boundaries node-specific, so the buggy merge path is exercised at block alignments that a deterministic schedule / the patricia control never hit — consistent with "one flat node diverged, the control didn't."

How to reproduce / confirm

• chiado forward sync with --FlatDb.Enabled true --FlatDb.VerifyWithTrie true to convert the silent InvalidStateRoot into a precise per-account/per-slot TrieException naming the diverging address/slot — look for FlatStorageTree "Get slot got wrong value … Self destruct it {idx}" and FlatWorldStateScope "Incorrect account …".
• Pin the schedule with --FlatDb.CompactionOffset 0 and run --FlatDb.InlineCompaction true to remove compactor/persist timing as a variable.
• Signature: a slot-level throw naming a recently self-destructed/recreated contract near the diverging block.

Note: --FlatDb.VerifyWithTrie true cannot be used for a post-snap live soak — the live trie-comparison store can't reload the snap-sync state root once evicted and the node wedges on a TrieNodeException (unrelated to this bug). Use it on a forward sync from genesis/early, or rely on the natural InvalidStateRoot on a snap+forward soak.

Related (NOT duplicates)

• FlatDB sync ends up on invalid chain #11353 (closed) — "FlatDB sync ends up on invalid chain": dismissed as testing the wrong schema (HalfPath, not FlatDb), a whole-block import divergence; not this steady-state single-slot merge defect.
• Gnosis archive sync - Invalid block on master #11193 (open) — Gnosis archive InvalidStateRoot: an AuRa archive-sync finalization bug (PR fix: skip AuRa finalization startup walk on post-merge chains #11306); does not enable FlatDb.
• FlatDb 1.37.1: restart mid-snap promotes incomplete state, deletes canonical chain, sync permanently stuck #11457, FlatDB sync does not survive restart of the client #11351, With FlatInTrie, --FlatDb.ImportFromPruningTrieState=true triggers on partway synced DB #11418, With FlatInTrie, a restart during Snap State Ranges greatly increases RAM usage and OOMs #11442, During FlatDB sync, Nethermind logs Unable to find beacon header at height xxx. This is unexpected, forcing a new beacon sync. #11447, testing_commitBlockV1 does not persist committed block state — every commit after the first fails #11979 — distinct FlatDb restart/durability/import/test defects.

Do not allowlist

Real consensus divergence on the flat backend; must be fixed before FlatDb ships. The smoke SyncCNWSF / Sync*Flat tests correctly catch it.

[asdacap]

Do this should any log? "Get slot got wrong value … Self destruct it {idx}" and FlatWorldStateScope "Incorrect account …".

[stdevMac]

@asdacap — re your question about which log lines surface this:

Without --FlatDb.VerifyWithTrie, nothing surfaces it early. Flat account/slot reads aren't hash-verified the way trie-node reads are, so a wrong flat value stays silent and only shows up downstream as the eventual InvalidStateRoot at block processing (computed root ≠ header root). That's why it looks intermittent and "comes from nowhere."

With --FlatDb.VerifyWithTrie=true (the dual-read diagnostic mode), the stale read is caught at the source — it throws at the first mismatching slot/account instead of waiting for the block-level root check. The two throws to grep for are:

• Get slot got wrong value … Self destruct it {idx} — FlatStorageTree.cs:81
• Incorrect account … — FlatWorldStateScope.cs:149

So: run a fresh sync with --FlatDb.VerifyWithTrie=true and the first occurrence will name the exact address/slot, rather than only the failing block. (Caveat from our soaks: VerifyWithTrie=true can wedge a node post-snap with an unrelated TrieNodeException when the comparison store can't reload an evicted snap root — so use it on a forward-from-early sync, not on a node that already finished snap.)

Status from the 1.39.0 validation (smoke run 27590615537): we exercised the flat backend again on 1.39.0-rc+2832ba91 (the _Flat sync lanes, mainnet + gnosis). We did not catch a fresh wrong-root divergence this round — i.e. it remains intermittent and uncaught in forward soak. What those _Flat nodes did hit was a separate, deterministic native SIGSEGV (exit 139) on restart of a fully-synced node — that's a distinct manifestation, filed separately as #12079 so this divergence thread stays focused.

Re-affirming the suspect framing in the issue body: #11739 (orphaned-snapshot pruning on persist) and #11756 (randomized per-instance compaction offset) remain the leading triggers; #11958 (CompactionOffset config) is a no-op at the default offset and not the mechanism. Please keep this not allowlisted — the SyncCNWSF test is doing its job.

[asdacap]

How far after snap sync do this happen?

[stdevMac]

Re: "How far after snap sync does this happen?" — the distance is irrelevant; the trigger is the restart.

We've reproduced this twice on chiado (snap + FlatDb + nimbus). The node syncs and runs completely clean — it only diverges when it's gracefully restarted and reloads the persisted flat DB.

Repro build: 1.39.0-rc + cherry-picks of #11996, #11967, #11997, #12035 (commit 32cfe5ee). Sync.VerifyTrieOnStateSyncFinished=true.

Timeline (latest run, chiado):

00:16:35  Verification complete. Accounts=539839, Slots=35588210,
          MismatchedAccounts=0, MismatchedSlots=0, MissingInFlat=0, MissingInTrie=0
          -> snap state is CLEAN (verified against trie)
00:16-03:03  ~3h clean forward catch-up, followed live head, no errors
03:03:30  Nethermind is shutting down...        (graceful restart)
03:04:08  State backend: flat (existing flat DB detected)
03:04:09  InvalidStateRoot: Expected 0x473c4c57..., got 0x844cda0d...
          (block hash expected 0xc6a6c905..., got 0x1e93e7a6...)

So: clean snap (0 mismatches verified) → clean forward processing → wrong root recomputed only from the persisted flat DB on restart. Steady-state processing never diverges; the on-disk state is what's wrong.

What this rules out:

• Not snap correctness — VerifyTrieOnStateSyncFinished passes with 0 mismatches.
• Not steady-state processing — ran ~3h at the head cleanly.
• Survives Flat: force-persist the head-reachable fork instead of an arbitrary one #11967, fix(flatdb): make snap finalize crash-durable and the restart wipe cheap #11997, Flat - Persistence Cache #12035 (all in the repro build), so it's none of the fork-selection / crash-durability / cache changes.
• fix(flatdb): preserve format markers when clearing the flat DB #11996 holds — no more SIGSEGV (FlatDb: native SIGSEGV (exit 139) on restart of a fully-synced node — no managed exception, not OOM #12079), but the wrong-root underneath it remains. Looks like the same restart-replay family.
• The only Unexpected old snapshot created. Lease count 2 warnings fire at the snap block (21756614), well before divergence — probably unrelated.

Happy to run a FlatDb.VerifyWithTrie=true soak to pin the exact diverging account/slot if that'd help — just let me know. Full EL + test logs available.