Skip to content

New payload with witness ssz#12048

Open
hudem1 wants to merge 101 commits into
masterfrom
new-payload-with-witness-ssz
Open

New payload with witness ssz#12048
hudem1 wants to merge 101 commits into
masterfrom
new-payload-with-witness-ssz

Conversation

@hudem1

@hudem1 hudem1 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Follow up of PR #11623 as this was from a forked repo

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

  • Yes
  • No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

  • Yes
  • No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.

Dyslex7c and others added 30 commits May 15, 2026 21:02
WitnessCapturingWorldStateProxy implemented only the explicit IWorldState
methods, leaving IsNonZeroAccount, IsStorageEmpty, HasCode, GetNonce, and
IsDelegatedCode to fall through to the default interface implementations.
Those defaults route via TryGetAccount and see only the committed
AccountStruct, while the real WorldState overrides them to consult the
in-flight persistent storage provider. The mismatch broke EIP-7610 checks
on SELFDESTRUCT-then-CREATE within a single block, causing the Pyspec
test_recreate fixtures on Paris and Shanghai to fail with a header
gas-used mismatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restore codeHash-first lookup and the parallel-BAL comment. The proxy
already captures the address via GetCodeHash(Address) earlier in
InternalGetCodeInfo, so the swap is not required for witness capture.

Also drop the verbose comment on the new default-method forwarders and
update the misleading note on GetCode(in ValueHash256).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spec MAX_REQUEST_BODY_SIZE per execution-apis#764 is 16 MiB; revert the
undocumented bump to 64 MiB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Witness owns ArrayPoolList buffers. On the JSON-RPC success path the
result is wrapped in ResultWrapper -> JsonRpcSuccessResponse, whose
Dispose() calls Result.TryDispose(). Implementing IDisposable here
routes that into Witness.Dispose() so the pool buffers are returned
instead of GC'd.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When two engine_newPayloadWithWitness calls land for the same blockHash,
ArmCapture cancels the first TCS. The first caller's await captureTask
then throws OperationCanceledException. The block itself executed
successfully, so return VALID with a null witness rather than letting
the cancellation propagate as a 500.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The INVALID_BLOCK_HASH constant and its SSZ byte mapping (4) were added
but no code path produces them; Nethermind returns INVALID with a
descriptive error for block-hash mismatches. Drop both until/unless we
wire the status through the actual mismatch path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BranchProcessor tracked the witness-capture lifecycle with a pair of
booleans and a manual finally block to disarm on exception. Replace with
a `using` session struct that arms in its factory, drains on success,
and disarms on Dispose if not drained. Removes the inner try/finally
and the ArmWitnessCapture / DrainWitnessCapture helpers; restores the
minor cosmetic edits that crept into the file.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the Func<...> indirection the handler used to break the
construction cycle. Now it takes `Lazy<IEngineRpcModule>` directly and
calls `Value.engine_newPayloadV5(...)`, which simplifies the DI wiring
to a plain type registration. Tests substitute IEngineRpcModule on the
builder.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
newPayloadWithWitness already has both a JSON-RPC and SSZ-REST surface
gated on the same fork, so route it through the same Configure(...) helper
the other Amsterdam entries use rather than two parallel dict writes.

Trim the comment in SszWireTypes that explains why
NewPayloadWithWitnessResponseV1 is hand-rolled in SszCodec — keep the
relevant why, drop the prose.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restore the blank line after the opening brace and the inline body of
engine_getClientVersionV1 — neither was needed for the PR's witness
work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three near-identical tests differing only in the engine module factory
collapse into a single [TestCaseSource]-driven case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WitnessGeneratingWorldState.GetWitness and WitnessCapturingWorldStateProxy.BuildWitness
both ran the same per-(address, slots) AccountProofCollector tree-walk loop. Pull the
loop into WitnessProofCollector.CollectAccountProofs so both call sites share it.

Also:
- Swap Arm() order in the proxy to allocate the tracking dictionaries before flipping
  the armed flag; the prior order was safe only on the single-threaded ProcessOne
  path but inverted is a free correctness improvement.
- Update WitnessCaptureRegistry xmldoc; the registry tolerates multiple concurrent
  armed entries for distinct block hashes (and cancels-and-replaces on duplicates).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dispatch

SszMiddleware.InvokeAsync (versioned) and DispatchWitnessAsync (non-versioned witness
path) duplicated the entire post-route pipeline: PipeReader body read, request-bytes
metric, handler invocation, status-code bucketing, and the 413/400/500 error mapping.
Pull the shared work into DispatchAsync(handler, version, extra) — the witness path now
adds only the path-specific method/content-type/handler-not-null checks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use Utf8JsonReader(ReadOnlySequence<byte>) in NewPayloadWithWitnessSszHandler instead
  of `body.ToArray()` on multi-segment bodies — saves an LOH allocation on every
  request.
- Promote the encode-length Debug.Assert in EncodeNewPayloadWithWitnessResponse to a
  runtime check so a future header-size bug fails fast in Release rather than emitting
  silently-wrong bytes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Claude finished @hudem1's task in 8m 52s —— View job


Claude finished reviewing the updated PR


PR Review: New payload with witness SSZ (updated)

Tasks

  • Load context and rule files
  • Read key changed files
  • Check if previous findings (H1, H2, M1-M4) were addressed
  • Review new architecture (PatriciaTrieWitnessGenerator, updated WitnessGeneratingWorldState)
  • Post inline comments for open findings
  • Post final summary

Architecture changes since previous review

The author's comment explains the key shift: state-node capture was removed from the in-flight path (the deleted WitnessCapturingTrieStore) and replaced with a post-execution trie walk using PatriciaTrieWitnessGenerator (from PR #12029). The new approach walks the pre-state patricia trie after execution, given the set of touched accounts/slots — cleaner than per-trie-node interception and avoids the flat-DB compatibility problem.

The new code is well-structured. WitnessGeneratingWorldState accumulates address/slot/bytecode touches during execution; CollectStateNodes fans those out into trie walks via PatriciaTrieWitnessGenerator; the PatriciaTrieWitnessGenerator itself is a clean BulkSet-mirroring traversal that correctly handles collapse siblings and the order-independence rule. The WitnessGeneratingBlockProcessingEnvFactory pool is idiomatic.


Status of previous findings

ID Finding Status
H1 DI lambda anti-pattern in BlockProcessingModule.cs Still open
H2 fallbackTrieStore not disposed ResolvedWitnessCapturingTrieStore deleted; BeginScope/using pattern properly manages the SnapshotBundle. IReadOnlyTrieStore.Dispose() is technically missing but practically a no-op (see new Low finding below).
M1 LINQ on hot-path (WitnessTrieStoreRecorder) Resolved — recorder deleted; new code uses plain Dictionary/foreach
M2 Fragile cast (WitnessCapturingWorldStateProxy)ctx.Resolve<IWorldState>() Still open — code comment added explaining the constraint, but structural fragility unchanged
M3 Fragile is NewPayloadWithWitnessSszHandler check in SszMiddleware Still open — code comment added, check unchanged
M4 TryArm <remarks> inverts the memory-ordering description Still open
L1 TruncateUtf8 silent empty-string on all-continuation input Still present (low risk)
L2 node.Keccak null guard in WitnessCapturingTrieStore Resolved — file deleted
L3 No timeout before CancelWitnessRequest Still present, acknowledged with code comment

New finding

Low – IReadOnlyTrieStore implements IDisposable but is never disposed

IReadOnlyTrieStore extends ITrieStore : IDisposable. In WitnessCapturingBlockProcessor.ProcessOne (line 94) a trie store is created and passed to WitnessGeneratingWorldState without ever being disposed. The same in WitnessGeneratingBlockProcessingEnvFactory.BuildEntry (line 83). Practical impact is low because the actual snapshot bundle (flat-DB path) is managed by the BeginScope/using in CollectStateNodes, and the patricia store's Dispose is a no-op — but it violates robustness.md dispose discipline. See inline comment for fix options.


Verdict: Not mergeable

H1 (DI lambda anti-pattern in BlockProcessingModule.cs:66–77) is unchanged from the previous review and is the primary blocker. M2, M3, and M4 remain open and have not been explicitly acknowledged with rationale in PR-level comments (code comments partially note the constraints for M2/M3 but not M3/M4).

Suggested next steps:

  1. Fix H1 — change isWitnessExecution: Func<bool>? to witnessExecutionPredicate: WitnessExecutionPredicate? and derive the Func<bool> in the constructor. That lets the single-liner .AddScoped<IBlockAccessListManager, BlockAccessListManager>() work again.
  2. Fix M4 — flip "post-CAS" to "pre-CAS" in the TryArm remarks.
  3. Decide on M2/M3 — either fix them or post a PR comment explicitly accepting the fragility and why it is bounded.

Comment thread src/Nethermind/Nethermind.Init/Modules/BlockProcessingModule.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs Outdated
Comment thread src/Nethermind/Nethermind.Consensus/Stateless/WitnessCaptureSession.cs Outdated
Comment thread src/Nethermind/Nethermind.Consensus/Stateless/WitnessCapturingBlockProcessor.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszCodec.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin.Test/EngineModuleTests.WitnessCapture.cs Outdated
@asdacap

asdacap commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Use a different graph for block processor so that it does not do a closure lookup on all read? #12094

asdacap and others added 2 commits June 23, 2026 18:52
…time switches (#12094)

* refactor(witness): capture via a second BlockProcessor graph, not runtime switches

Replace the in-flight witness capture (per-call session/predicate switches threaded through the main pipeline) with a statically witness-wired second IBlockProcessor graph that a thin selector delegates a witnessed block to. The graph shares the main pipeline's writable IWorldState through a transparent recorder, so the witnessed block stays a real single-execution import.

- WitnessCapturingBlockProcessingEnv (root singleton, lazy): builds the witness graph off the root scope (so it does not inherit the main scope's IBlockProcessor selector decorator -> cycle); the recorder wraps IMainProcessingContext.WorldState; the bundle is resolved from the scope.
- WitnessCapturingBlockProcessor becomes a thin selector delegating a pending-request block to the env, else to the inner processor; TransactionsExecuted passes straight through to inner.
- WitnessCapturingHeaderFinder is now session-free (records into WitnessHeaderRecorder).
- BlockAccessListManager reverts to a static bool witnessMode (the witness env sets it true).
- Delete WitnessCaptureSession, WitnessExecutionPredicate, CodeInfoRepositoryProxy, WitnessCapturingWorldStateProxy. Keep MainProcessingContext.DistinctBy (guards the selector decorator under AuRa's double module tree).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(witness): keep witness bundle in a single Graph

Per review: drop the resolved-Graph + separate Built disposal record and keep everything in one Graph (constructed directly, owning the scope and witness-walk trie store).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: Reset blockhashcache across blocks

* chore: Restore single line BALManager DI registration

* fix: Graph.Dispose always disposes trie store

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Hugo Demeyere <demeyere.hugo@gmail.com>
private readonly FrozenDictionary<string, List<ISszEndpointHandler>>.AlternateLookup<ReadOnlySpan<char>> _postLookup;
private readonly FrozenDictionary<string, List<ISszEndpointHandler>>.AlternateLookup<ReadOnlySpan<char>> _getLookup;

private readonly ISszEndpointHandler? _witnessHandler;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No special case here please. Abstract the path/characteristic behind ISszEndpointHandler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature rpc state+storage taiko related to the taiko alethia rollup trie

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants