Serve data_column_sidecar events for gossip publication requests - #249
Merged
Merged
Conversation
Bronek
added this pull request to stack #250
September 11, 2026 15:22
Bronek
force-pushed
the
bronek/sse_data_column_sidecar_v2
branch
from
September 11, 2026 15:54
7a44803 to
4380391
Compare
Bronek
force-pushed
the
bronek/sse_data_column_sidecar_v2
branch
from
September 11, 2026 17:02
4380391 to
9c5e952
Compare
vladimir-ea
reviewed
Sep 14, 2026
|
|
||
| let sources = crate::discovery::discover(&tmp, "surfer_test").unwrap(); | ||
| let sources = crate::discovery::discover(tmp.path(), "surfer_test").unwrap(); | ||
| let file = sources.counters.iter().find(|f| f.name == "surfer_smoke").unwrap(); |
Collaborator
There was a problem hiding this comment.
what is this change? and given this change doesn't this line above also need to change?
Collaborator
Author
There was a problem hiding this comment.
This is in 0843f87 to stop silver's unit tests leaking temporary files / shmem . I should have created a separate PR for it - sorry about that. It's pure unit tests cleanup.
vladimir-ea
approved these changes
Sep 14, 2026
Replace SendGossip.block with optional GossipMetadata, wrapping the existing block record and adding GossipDataColumn. PublishDataColumn carries its column record directly. Attach slot, block_root, and column_index at both existing publication sites after KZG verification. Preserve the RPC sync gate and the absence of publication for buffered, held, and reconstructed columns. Control carries the column record into its converted SendGossip request, which stays local to peer-manager. Controller tests observe encoding, sender exclusion, IWANT service, and no second spine publication. The converted record's fields have no independent observation because peer-manager ignores metadata when routing. The boundary still serves block metadata only. Its existing socket test also injects both column request variants to protect block subscribers. Column SSE serving follows in the next commit. Leave Persist and Available behavior unchanged. A table checks gossip and following RPC metadata for custody and non-custody columns. Syncing RPC processes its column without requesting publication. Focused cases cover Fulu proposer resolution, held copies, buffered Gloas columns, and EL reconstruction. A mixed valid/invalid KZG case requires publication of the valid column and silence for the invalid one. The signed EF staged-parent case checks both arrival orders. Buffered copies remain silent; fresh copies request relay after their parent is staged. The fixture loader discovers the sidecar without embedding its hash and derives metadata from its contents. The staged-parent case requires installed Fulu EF fixtures. An absent fixture directory reports unavailable coverage; an incomplete installed case fails. The final run exercised this case with fixtures present. One counting blob provides real KZG cells with distinguishable proofs. The focused Fulu proposer test pre-seeds the signature cache because its empty validator registry cannot verify signatures. Column gossip fixtures reuse SSZ as an unread protobuf handle. Sidecars enter production handlers, and loop_body completes validation. Publication assertions check metadata and topic without fixing batch contents or requiring deferred verification. Persistence observations establish successful processing on otherwise silent paths. Controller fixtures use synthetic transport payloads. The pre-existing EL reconstruction reservation-failure bug remains outside this change. Validation: just fmt-check, just clippy, just nextest, and git diff --check exited zero. Nextest passed 1,368 tests and skipped five. Assisted-by: Codex:gpt-6-astra
Add the data_column_sidecar topic, renderer, and BeaconApi publication method. Emit block_root as lowercase hex and index and slot as decimal strings. Publish column metadata from SendGossip and PublishDataColumn through the existing peer-event consumer, outside the engine capacity gate. Select metadata or request kind without reading payload bytes or repeating the producer's topic check. SSE acknowledges publication requests following silver's gossip checks, including KZG; it does not guarantee delivery to peers. The ADR amendment records this narrower validation contract, excluded paths, repeated requests, and no replay. No additional queue or spine publication is introduced. Extend the existing topic parser test, retaining repeated parameters and URL-encoded commas. The renderer test checks required JSON fields with distinct slot and index values, allowing additional fields and different formatting. Extend three existing socket scenarios through spine injection and ApplicationBoundaryTile::loop_body. They cover both column request variants, topic isolation, repeated requests, late subscriptions, and delivery during engine saturation. Mixed subscriptions preserve each topic's sequence without requiring an order between topics. Trailing sentinels keep unexpected frames inside the observations. The saturation fixture fills the configured pool with its own unanswered FCU requests. Publication requests carry opaque fixture bytes; these socket tests do not establish validation or network delivery. Fixtures keep topic and metadata consistent, so they cannot distinguish an additional topic guard. The changed tests passed with reordered JSON keys, extra whitespace, an additional nested field, and the peer-event drain preceding the beacon-event drain. Those temporary changes were restored. The full control runs encountered SIGBUS failures in other tests at default concurrency; no negative controls or size measurements were repeated. Final gates: just fmt-check, just clippy, git diff --check, and NEXTEST_TEST_THREADS=4 just nextest exited 0. The workspace passed 1,369 tests and skipped five. Assisted-by: Codex:gpt-6-astra
Several fixtures create directories without retaining a cleanup owner. Others remove directories only after their assertions succeed. Retain TempDir ownership for storage, configuration, counters, telemetry, and panic-log fixtures so cleanup also runs during unwinding. Deleting Flux link files alone leaves their POSIX shared-memory objects behind. Add ShmemDir under silver_common's test-util feature to call cleanup_shmem before its TempDir drops. Use it for tile spines, boundary fixtures, Surfer queues, and the E2E harnesses and examples. Keep directory owners alive alongside the resources they support. The beacon-state producer fixture now owns its spine and directory together. Existing behavior assertions remain unchanged. One regression test checks that the directory disappears and its shared memory cannot be reopened after normal return or unwinding. Before the cleanup call, that test failed because the shared-memory object survived; the other 1,369 tests passed. An isolated Controller test previously left 119 shared-memory objects. It now leaves no fixture objects and 28 process-global timer objects. Those timers remain outside the private fixture directories. Forgotten Flux mappings remain live until process exit; this change unlinks their names without changing the mapping lifetime. Destructor cleanup cannot cover aborts or kills that skip unwinding. Final gates: just fmt-check, just clippy, git diff --check, and NEXTEST_TEST_THREADS=4 just nextest exited 0. The workspace passed 1,370 tests and skipped five. Assisted-by: Codex:gpt-6-astra
Match the feature selection used by just clippy so analysis includes feature-gated test support such as silver_common's test-util. Assisted-by: Claude:claude-fable-5-1 Assisted-by: Codex:gpt-6-astra
Replace SendGossip's per-topic metadata with a decompressed SSZ handle for every topic. Remove the column metadata from PublishDataColumn and delete GossipBlock, GossipDataColumn and GossipMetadata. Consumers derive the fields they need from the referenced bytes. The application boundary reads gossip and RPC caches to build block_gossip and data_column_sidecar events. It computes block roots from block bytes and extracts sidecar identities from Fulu or Gloas layouts. Cache read failures and unrecognized sidecar layouts produce warnings and no event. Block roots are computed even without subscribers. ADR-0004 documents these costs and failure conditions. Publish the boundary consumers' tails on every loop iteration so idle consumers can release cache space even when no publication requests arrive. A boundary test fills both caches and checks that producers can reserve space again after the idle interval and another tile iteration. Producer fixtures use distinct protobuf and SSZ payloads to expose incorrect handle forwarding. Boundary fixtures include a fixed block-body prefix to exercise fork-specific hashing. Tests cover topic selection, repeated requests, late subscriptions and engine saturation. Controller tests check routing and IWANT service with distinct payloads. Assisted-by: Claude:claude-fable-5-1 Assisted-by: Codex:gpt-6-astra
Bronek
force-pushed
the
bronek/sse_data_column_sidecar_v2
branch
from
September 14, 2026 12:34
0833481 to
cc9488b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
data_column_sidecarto/eth/v1/events. The promise is: these events correspond to what we publish on gossip. An event reports silver's publication request, not successful delivery to a peer.This PR uses Beacon API v5.0.0-alpha.2 as its specification baseline, rather than released v4.0.0.
This deliberately narrows the alpha's validation event. Upstream covers sidecars received through P2P or API after subnet gossip validation. Silver additionally requires publication, intentionally coupling these notifications to relay policy.
Both publication paths follow silver's gossip checks, including KZG. Gossip columns request
SendGossip; RPC columns requestPublishDataColumnwhile synced, regardless of custody. Buffered copies remain silent after their missing block or parent arrives. Syncing RPC, held copies, and EL reconstruction also remain silent. A later acquisition or encoding failure does not retract the original request.Compatibility: clients requiring v4's
kzg_commitmentsfield cannot decode this event. Silver follows the alpha's three-field payload, introduced inv5.0.0-alpha.1. #583 removedkzg_commitmentson 2026-03-09 after Gloas removed commitments from sidecars; Fulu sidecars still carry them. Releasedv4.0.0retains the field. The inspected go-eth2-client checkout (2026-07-16) and Lighthouse v8.2.2 use the v4 event shape. go-eth2-client rejects an empty commitments array; Lighthouse requires bothkzg_commitmentsandversioned_hashes. Consumers using these decoders, including Xatu and ethereum-metrics-exporter, need v5 support to decode this event.This PR stacks on
bronek/sse_block_gossip_v2, as the second PR abovemain. It has no prerequisite on PR #231.rust-analyzer.tomlmirroringjust clippy's--all-features, so feature-gated test support stops showing as unresolved imports in editors.The cleanup commit covers normal teardown and unwinding. Process-global timer objects and cleanup after aborts remain separate work.
Persist,Available, and their consumers remain unchanged. Repeated requests are not deduplicated; late subscribers receive no replay. Additional subscriptions can reach the existing send cap sooner.The ADR-0004 amendment records the publication contract and deliberate divergence. P2P conformance gaps remain tracked separately.
Message sizes and ring capacities are unchanged.
Coverage limits: