Skip to content

Use Pipelined Execution instead of sequential for Full sync #6484

@Arkenan

Description

@Arkenan

Est. +20-40% throughput

Current: add_blocks_in_batch calls execute_block_from_state per block sequentially, then does merkle at the end. The EVM runs single-threaded and merkleization happens only after ALL blocks execute.

What add_block_pipeline does differently (currently only used for CL blocks):

  • 3 concurrent threads: warmer, executor, merkleizer
  • Pipelined merkle: State transitions stream from executor to merkleizer via channel during execution
  • 16 parallel shard workers for trie computation
  • Cache warming: Prefetches state before execution (less useful for full sync since blocks are sequential)

The issue: add_blocks_in_batch uses execute_block_from_state which is a simple sequential path. It deliberately skips pipelining because batch mode collapses all blocks into a single diff layer.

Proposed approach: Instead of running all 1024 blocks sequentially then doing one giant merkle at the end, process blocks in smaller sub-batches (e.g., 64 blocks) using the pipeline. This lets merkleization overlap with execution.

Trade-off: Pipelining means per-sub-batch merkleization instead of one collapsed batch. The collapsed approach is cheaper for merkle (22ms vs 16 × ~20ms = 320ms), but the pipelining overlap should more than compensate.

Effort: High. Needs careful integration with the batch state management.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceBlock execution throughput and performance in generalsyncingSnap sync, Full sync

    Type

    No type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions