Est. +20-40% throughput
Current: add_blocks_in_batch calls execute_block_from_state per block sequentially, then does merkle at the end. The EVM runs single-threaded and merkleization happens only after ALL blocks execute.
What add_block_pipeline does differently (currently only used for CL blocks):
- 3 concurrent threads: warmer, executor, merkleizer
- Pipelined merkle: State transitions stream from executor to merkleizer via channel during execution
- 16 parallel shard workers for trie computation
- Cache warming: Prefetches state before execution (less useful for full sync since blocks are sequential)
The issue: add_blocks_in_batch uses execute_block_from_state which is a simple sequential path. It deliberately skips pipelining because batch mode collapses all blocks into a single diff layer.
Proposed approach: Instead of running all 1024 blocks sequentially then doing one giant merkle at the end, process blocks in smaller sub-batches (e.g., 64 blocks) using the pipeline. This lets merkleization overlap with execution.
Trade-off: Pipelining means per-sub-batch merkleization instead of one collapsed batch. The collapsed approach is cheaper for merkle (22ms vs 16 × ~20ms = 320ms), but the pipelining overlap should more than compensate.
Effort: High. Needs careful integration with the batch state management.
Est. +20-40% throughput
Current:
add_blocks_in_batchcallsexecute_block_from_stateper block sequentially, then does merkle at the end. The EVM runs single-threaded and merkleization happens only after ALL blocks execute.What
add_block_pipelinedoes differently (currently only used for CL blocks):The issue:
add_blocks_in_batchusesexecute_block_from_statewhich is a simple sequential path. It deliberately skips pipelining because batch mode collapses all blocks into a single diff layer.Proposed approach: Instead of running all 1024 blocks sequentially then doing one giant merkle at the end, process blocks in smaller sub-batches (e.g., 64 blocks) using the pipeline. This lets merkleization overlap with execution.
Trade-off: Pipelining means per-sub-batch merkleization instead of one collapsed batch. The collapsed approach is cheaper for merkle (22ms vs 16 × ~20ms = 320ms), but the pipelining overlap should more than compensate.
Effort: High. Needs careful integration with the batch state management.