Skip to content

Serve data_column_sidecar events for gossip publication requests - #249

Merged
Bronek merged 5 commits into
mainfrom
bronek/sse_data_column_sidecar_v2
Sep 14, 2026
Merged

Bronek merged 5 commits into
mainfrom
bronek/sse_data_column_sidecar_v2

Conversation

@Bronek

@Bronek Bronek commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Add data_column_sidecar to /eth/v1/events. The promise is: these events correspond to what we publish on gossip. An event reports silver's publication request, not successful delivery to a peer.

This PR uses Beacon API v5.0.0-alpha.2 as its specification baseline, rather than released v4.0.0.

This deliberately narrows the alpha's validation event. Upstream covers sidecars received through P2P or API after subnet gossip validation. Silver additionally requires publication, intentionally coupling these notifications to relay policy.

Both publication paths follow silver's gossip checks, including KZG. Gossip columns request SendGossip; RPC columns request PublishDataColumn while synced, regardless of custody. Buffered copies remain silent after their missing block or parent arrives. Syncing RPC, held copies, and EL reconstruction also remain silent. A later acquisition or encoding failure does not retract the original request.

event: data_column_sidecar
data: {"block_root":"0x<64 lowercase hex>","index":"<decimal>","slot":"<decimal>"}

Compatibility: clients requiring v4's kzg_commitments field cannot decode this event. Silver follows the alpha's three-field payload, introduced in v5.0.0-alpha.1. #583 removed kzg_commitments on 2026-03-09 after Gloas removed commitments from sidecars; Fulu sidecars still carry them. Released v4.0.0 retains the field. The inspected go-eth2-client checkout (2026-07-16) and Lighthouse v8.2.2 use the v4 event shape. go-eth2-client rejects an empty commitments array; Lighthouse requires both kzg_commitments and versioned_hashes. Consumers using these decoders, including Xatu and ethereum-metrics-exporter, need v5 support to decode this event.

This PR stacks on bronek/sse_block_gossip_v2, as the second PR above main. It has no prerequisite on PR #231.

  • Carry column metadata on publication requests: introduce the metadata enum and enrich both existing requests. Producer tests cover selection, metadata, persistence, and completion; Controller tests verify routing and that conversion adds no second spine publication.
  • Serve data_column_sidecar events over SSE: add the topic, renderer, and boundary handling for both request variants. Socket tests cover all three topics, repeats, late subscribers, and engine saturation.
  • Clean up temporary test directories and shared memory: give fixtures cleanup owners and unlink Flux objects before removing their link files. This includes pre-existing test leaks found while reviewing the stack.
  • Let rust-analyzer see every feature: a root rust-analyzer.toml mirroring just clippy's --all-features, so feature-gated test support stops showing as unresolved imports in editors.

The cleanup commit covers normal teardown and unwinding. Process-global timer objects and cleanup after aborts remain separate work.

Persist, Available, and their consumers remain unchanged. Repeated requests are not deduplicated; late subscribers receive no replay. Additional subscriptions can reach the existing send cap sooner.

The ADR-0004 amendment records the publication contract and deliberate divergence. P2P conformance gaps remain tracked separately.
Message sizes and ring capacities are unchanged.

Coverage limits:

  • Synthetic Fulu fixtures pre-seed the signature cache because their validator registry is empty; parent, proposer, inclusion, and KZG checks still run.
  • Staged-parent assertions require installed EF fixtures; the test returns early when they are absent.

@Bronek
Bronek added this pull request to stack #250 September 11, 2026 15:22
@Bronek
Bronek force-pushed the bronek/sse_data_column_sidecar_v2 branch from 7a44803 to 4380391 Compare September 11, 2026 15:54
@Bronek
Bronek force-pushed the bronek/sse_data_column_sidecar_v2 branch from 4380391 to 9c5e952 Compare September 11, 2026 17:02

let sources = crate::discovery::discover(&tmp, "surfer_test").unwrap();
let sources = crate::discovery::discover(tmp.path(), "surfer_test").unwrap();
let file = sources.counters.iter().find(|f| f.name == "surfer_smoke").unwrap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this change? and given this change doesn't this line above also need to change?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is in 0843f87 to stop silver's unit tests leaking temporary files / shmem . I should have created a separate PR for it - sorry about that. It's pure unit tests cleanup.

Base automatically changed from bronek/sse_block_gossip_v2 to main September 14, 2026 12:34
Replace SendGossip.block with optional GossipMetadata, wrapping the
existing block record and adding GossipDataColumn. PublishDataColumn
carries its column record directly.

Attach slot, block_root, and column_index at both existing publication
sites after KZG verification. Preserve the RPC sync gate and the
absence of publication for buffered, held, and reconstructed columns.

Control carries the column record into its converted SendGossip request,
which stays local to peer-manager. Controller tests observe encoding,
sender exclusion, IWANT service, and no second spine publication.
The converted record's fields have no independent observation because
peer-manager ignores metadata when routing.

The boundary still serves block metadata only. Its existing socket test
also injects both column request variants to protect block subscribers.
Column SSE serving follows in the next commit.

Leave Persist and Available behavior unchanged. A table checks gossip
and following RPC metadata for custody and non-custody columns.
Syncing RPC processes its column without requesting publication.

Focused cases cover Fulu proposer resolution, held copies, buffered
Gloas columns, and EL reconstruction. A mixed valid/invalid KZG case
requires publication of the valid column and silence for the invalid one.

The signed EF staged-parent case checks both arrival orders. Buffered
copies remain silent; fresh copies request relay after their parent
is staged. The fixture loader discovers the sidecar without embedding
its hash and derives metadata from its contents.

The staged-parent case requires installed Fulu EF fixtures. An absent
fixture directory reports unavailable coverage; an incomplete installed
case fails. The final run exercised this case with fixtures present.

One counting blob provides real KZG cells with distinguishable proofs.
The focused Fulu proposer test pre-seeds the signature cache because
its empty validator registry cannot verify signatures.
Column gossip fixtures reuse SSZ as an unread protobuf handle.

Sidecars enter production handlers, and loop_body completes validation.
Publication assertions check metadata and topic without fixing batch
contents or requiring deferred verification. Persistence observations
establish successful processing on otherwise silent paths.

Controller fixtures use synthetic transport payloads. The pre-existing
EL reconstruction reservation-failure bug remains outside this change.

Validation: just fmt-check, just clippy, just nextest, and
git diff --check exited zero. Nextest passed 1,368 tests and skipped five.

Assisted-by: Codex:gpt-6-astra
Add the data_column_sidecar topic, renderer, and BeaconApi publication
method. Emit block_root as lowercase hex and index and slot as decimal
strings.

Publish column metadata from SendGossip and PublishDataColumn through
the existing peer-event consumer, outside the engine capacity gate.
Select metadata or request kind without reading payload bytes or
repeating the producer's topic check.

SSE acknowledges publication requests following silver's gossip checks,
including KZG; it does not guarantee delivery to peers. The ADR amendment
records this narrower validation contract, excluded paths, repeated
requests, and no replay. No additional queue or spine publication is
introduced.

Extend the existing topic parser test, retaining repeated parameters
and URL-encoded commas. The renderer test checks required JSON fields
with distinct slot and index values, allowing additional fields and
different formatting.

Extend three existing socket scenarios through spine injection and
ApplicationBoundaryTile::loop_body. They cover both column request
variants, topic isolation, repeated requests, late subscriptions, and
delivery during engine saturation. Mixed subscriptions preserve each
topic's sequence without requiring an order between topics. Trailing
sentinels keep unexpected frames inside the observations. The saturation
fixture fills the configured pool with its own unanswered FCU requests.

Publication requests carry opaque fixture bytes; these socket tests do
not establish validation or network delivery. Fixtures keep topic and
metadata consistent, so they cannot distinguish an additional topic
guard.

The changed tests passed with reordered JSON keys, extra whitespace,
an additional nested field, and the peer-event drain preceding the
beacon-event drain. Those temporary changes were restored. The full
control runs encountered SIGBUS failures in other tests at default
concurrency; no negative controls or size measurements were repeated.

Final gates: just fmt-check, just clippy, git diff --check, and
NEXTEST_TEST_THREADS=4 just nextest exited 0. The workspace passed
1,369 tests and skipped five.

Assisted-by: Codex:gpt-6-astra
Several fixtures create directories without retaining a cleanup owner.
Others remove directories only after their assertions succeed. Retain
TempDir ownership for storage, configuration, counters, telemetry, and
panic-log fixtures so cleanup also runs during unwinding.

Deleting Flux link files alone leaves their POSIX shared-memory objects
behind. Add ShmemDir under silver_common's test-util feature to call
cleanup_shmem before its TempDir drops. Use it for tile spines, boundary
fixtures, Surfer queues, and the E2E harnesses and examples.

Keep directory owners alive alongside the resources they support.
The beacon-state producer fixture now owns its spine and directory
together. Existing behavior assertions remain unchanged.

One regression test checks that the directory disappears and its shared
memory cannot be reopened after normal return or unwinding. Before the
cleanup call, that test failed because the shared-memory object survived;
the other 1,369 tests passed.

An isolated Controller test previously left 119 shared-memory objects.
It now leaves no fixture objects and 28 process-global timer objects.
Those timers remain outside the private fixture directories. Forgotten
Flux mappings remain live until process exit; this change unlinks their
names without changing the mapping lifetime. Destructor cleanup cannot
cover aborts or kills that skip unwinding.

Final gates: just fmt-check, just clippy, git diff --check, and
NEXTEST_TEST_THREADS=4 just nextest exited 0. The workspace passed
1,370 tests and skipped five.

Assisted-by: Codex:gpt-6-astra
Match the feature selection used by just clippy so analysis includes
feature-gated test support such as silver_common's test-util.

Assisted-by: Claude:claude-fable-5-1
Assisted-by: Codex:gpt-6-astra
Replace SendGossip's per-topic metadata with a decompressed SSZ handle
for every topic. Remove the column metadata from PublishDataColumn and
delete GossipBlock, GossipDataColumn and GossipMetadata. Consumers derive
the fields they need from the referenced bytes.

The application boundary reads gossip and RPC caches to build
block_gossip and data_column_sidecar events. It computes block roots
from block bytes and extracts sidecar identities from Fulu or Gloas
layouts. Cache read failures and unrecognized sidecar layouts produce
warnings and no event. Block roots are computed even without subscribers.
ADR-0004 documents these costs and failure conditions.

Publish the boundary consumers' tails on every loop iteration so idle
consumers can release cache space even when no publication requests arrive.
A boundary test fills both caches and checks that producers can reserve
space again after the idle interval and another tile iteration.

Producer fixtures use distinct protobuf and SSZ payloads to expose
incorrect handle forwarding. Boundary fixtures include a fixed block-body
prefix to exercise fork-specific hashing. Tests cover topic selection,
repeated requests, late subscriptions and engine saturation. Controller
tests check routing and IWANT service with distinct payloads.

Assisted-by: Claude:claude-fable-5-1
Assisted-by: Codex:gpt-6-astra
@Bronek
Bronek force-pushed the bronek/sse_data_column_sidecar_v2 branch from 0833481 to cc9488b Compare September 14, 2026 12:34
@Bronek
Bronek merged commit 1260d01 into main Sep 14, 2026
2 of 3 checks passed
@Bronek
Bronek deleted the bronek/sse_data_column_sidecar_v2 branch September 14, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants