Skip to content

perf(op-interop-filter): prefetch blocks during ingestion#20398

Draft
karlfloersch wants to merge 8 commits intodevelopfrom
karlfloersch/op-interop-filter-prefetch-pipeline-v2
Draft

perf(op-interop-filter): prefetch blocks during ingestion#20398
karlfloersch wants to merge 8 commits intodevelopfrom
karlfloersch/op-interop-filter-prefetch-pipeline-v2

Conversation

@karlfloersch
Copy link
Copy Markdown
Contributor

Summary

  • adds configurable --rpc-concurrency and --fetch-concurrency settings for op-interop-filter ingestion
  • adds a bounded block prefetcher that overlaps RPC fetches while returning blocks in number order
  • keeps logs DB writes, parent-hash checks, and executing-message processing on the existing sequential write path
  • validates positive concurrency values and requires fetch-concurrency <= rpc-concurrency

Notes

Validation

  • go test ./op-interop-filter/...
  • go mod tidy -diff
  • ./linter/bin/op-golangci-lint run ./...

@karlfloersch
Copy link
Copy Markdown
Contributor Author

Tested the prefetch branch against mainnet OP + Unichain RPCs with fresh local DBs. RPC URLs intentionally omitted.

Functional smoke test:

  • Filter came up cleanly with --rpc-concurrency 100 --fetch-concurrency 64.
  • Backfilled and served queries for OP Mainnet and Unichain Mainnet.
  • Spammer checks against the filter passed for unsafe and cross-unsafe query paths.
  • No observed 429s, reorg handling errors, panics, failsafe activation, or ingestion failures in the test logs.

10-minute backfill A/B:

  • Same binary, same machine, fresh data dirs per run.
  • Networks: OP Mainnet + Unichain Mainnet.
  • Serial baseline: --fetch-concurrency 1, --rpc-concurrency 100.
  • Concurrent run: --fetch-concurrency 64, --rpc-concurrency 100.

Results:

Mode Fetch concurrency Backfill target Time Throughput
Serial 1 OP 301, Unichain 601 104.966s 8.59 blocks/s
Concurrent 64 OP 301, Unichain 602 3.365s 268.35 blocks/s

Observed speedup: about 31.2x for the backfill phase.

@karlfloersch karlfloersch marked this pull request as ready for review April 28, 2026 17:46
@karlfloersch karlfloersch requested a review from a team as a code owner April 28, 2026 17:46
@karlfloersch karlfloersch force-pushed the karlfloersch/op-interop-filter-prefetch-pipeline-v2 branch from a7c96ea to 42ba28e Compare April 28, 2026 18:35
@karlfloersch karlfloersch marked this pull request as draft April 28, 2026 18:35
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2026

Codecov Report

❌ Patch coverage is 0% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.5%. Comparing base (484be19) to head (4e52cde).
⚠️ Report is 5 commits behind head on develop.

Files with missing lines Patch % Lines
...protocol/derive/src/stages/channel/channel_bank.rs 0.0% 9 Missing ⚠️
rust/kona/crates/protocol/registry/src/lib.rs 0.0% 9 Missing ⚠️
...st/kona/crates/protocol/registry/src/superchain.rs 0.0% 5 Missing ⚠️
...ates/protocol/genesis/src/superchain/chain_list.rs 0.0% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (484be19) and HEAD (4e52cde). Click for more details.

HEAD has 13 uploads less than BASE
Flag BASE (484be19) HEAD (4e52cde)
cannon-go-tests-64 1 0
contracts-bedrock-tests 12 0
Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #20398      +/-   ##
===========================================
- Coverage     76.4%     0.5%   -76.0%     
===========================================
  Files          186      487     +301     
  Lines        10691    62411   +51720     
===========================================
- Hits          8175      317    -7858     
- Misses        2372    62094   +59722     
+ Partials       144        0     -144     
Flag Coverage Δ
cannon-go-tests-64 ?
contracts-bedrock-tests ?
unit 0.5% <0.0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
rust/kona/crates/node/peers/src/nodes.rs 0.0% <ø> (ø)
...ates/protocol/genesis/src/superchain/chain_list.rs 9.6% <0.0%> (ø)
...st/kona/crates/protocol/registry/src/superchain.rs 47.1% <0.0%> (ø)
...protocol/derive/src/stages/channel/channel_bank.rs 0.0% <0.0%> (ø)
rust/kona/crates/protocol/registry/src/lib.rs 38.8% <0.0%> (ø)

... and 668 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant