perf: eliminate double block execution in `engine_newPayloadWithWitness`

[Dyslex7c]

Background

PR #11623 implements the POST /new-payload-with-witness REST endpoint and the engine_newPayloadWithWitness JSON-RPC method as specified in execution-apis#773. The implementation is correct, but has a known performance shortcoming that was explicitly called out in the review and deferred as a follow-up.

Problem

Every call to either the REST endpoint or the JSON-RPC method causes the block to be executed twice:

1. First execution — inside engine_newPayloadV5 via NewPayloadHandler, which enqueues the block into BlockchainProcessor and processes it against the production IWorldState. The block is committed to the chain.
2. Second execution — inside TryGenerateWitness / TryGenerateWitnessForBlock, which calls WitnessCollector.GetWitnessForExistingBlock:

public Witness GetWitnessForExistingBlock(BlockHeader parentHeader, Block block)
{
    using IDisposable? scope = worldState.BeginScope(parentHeader);
    blockProcessor.ProcessOne(block, ProcessingOptions.ReadOnlyChain, ...);
    return worldState.GetWitness(parentHeader);
}

This second ProcessOne call spins up a fresh WitnessGeneratingBlockProcessingEnvScope (new ReadOnlyDbProvider, new WitnessCapturingTrieStore, new Autofac lifetime scope) and re-executes all transactions from scratch. The original spec motivation for execution-apis#773 was to produce the witness as part of the primary execution at ~700ms total, not to add a second full re-execution on top of it.

Impact

• On a heavy mainnet block the second execution can push total latency well past the 8s engine_newPayload timeout budget.
• The stated performance benefit of the combined endpoint over the old two-step engine_newPayload + debug_executionWitness flow is not delivered.
• The extra Autofac scope + ReadOnlyDbProvider construction adds meaningful allocation pressure on every VALID payload, regardless of actual witness use.

Root cause

engine_newPayloadV5 uses the production IWorldState injected into BlockchainProcessor at construction time. WitnessGeneratingWorldState is a separate wrapper type with its own isolated DI scope — it cannot be transparently swapped into the existing pipeline without structural changes to how BranchProcessor and BlockchainProcessor resolve their world state.

Desired outcome

When the caller is engine_newPayloadWithWitness (REST or JSON-RPC), witness collection should be wired into the first ProcessOne call rather than triggering a second one. Concretely, this likely requires one of:

• Passing an optional WitnessGeneratingWorldState (or a post-processing hook) through the NewPayloadHandler → BlockchainProcessor path so the witness is captured during primary execution.
• Introducing a ProcessingOptions flag that causes BranchProcessor to record state access using the witness trie store as part of its existing execution pass.

Either approach needs to avoid allocating the witness infrastructure for normal (non-witness) engine_newPayloadV5 calls.

References

• PR introducing the feature: feat(engine): introduce REST based + SSZ serialized new-payload-with-witness #11623
• Upstream spec: Introduction of REST based + SSZ serialized new-payload-with-witness ethereum/execution-apis#773
• Reviewer comment (round 1): "That's a full re-execution — exactly the cost NDM consumer service refactoring & testing #773 was created to eliminate."
• Relevant files: WitnessCollector.cs, NewPayloadWithWitnessHandler.cs, NewPayloadWithWitnessSszHandler.cs, WitnessGeneratingBlockProcessingEnvFactory.cs