Serve `head` and `head_v2` events over SSE by Bronek · Pull Request #231 · gattaca-com/silver

Bronek · 2026-09-09T09:50:28Z

Adds head and head_v2 to /eth/v1/events, alongside block. Both topics report changes to the selected head or its execution optimism. Validating an optimistic head, for example, produces another event for the same block with execution_optimistic: false.

head_v2 also reports changes in fork choice's empty/full payload selection, including both transition directions. These can occur without changing the head block or its optimism. Its versioned response identifies the configured fork at the head block's slot; selected pre-Gloas blocks report full.

The existing BeaconStateEvent::Status carries the head's state root, both dependent roots and payload resolution as one coherent observation. Additional publications cover head changes outside block import, including execution verdicts. Consumers decide which observations require action; /eth/v1/node/syncing also receives prompt optimism updates.

The application boundary suppresses repeated observations. The first complete observation establishes a baseline, and subscribers receive future changes without an initial snapshot. epoch_transition is true only when the observed head epoch advances. If checkpoint history has overwritten a required dependent root, publication waits for a complete observation and retains the previous baseline.

The checkpoint anchor now advertises its block header's slot. Previously, a checkpoint state advanced through empty slots could advertise the later state slot instead.

Memory cost: the beacon-events ring grows from 3 MiB to 5 MiB, with event size increasing from 144 to 240 bytes and queue slots from 192 to 320 bytes. Retaining the state root adds 32 bytes per fork-choice node. Payload resolution fits the event's padding and adds no further ring growth.

Tests cover head and payload transitions, dependent-root history and checkpoint boundaries, repeated Status consumers, replay gating, fork-version selection, and delivery to individual and mixed-topic subscribers. Slot and PTC tests exercise publication from controlled fork-choice inputs; signed envelope and PTC gossip validation are outside those tests.

Validation: formatting, clippy and diff checks passed; just nextest passed 1,382 tests with five skipped. Subsequent prose edits passed formatting and diff checks.

vladimir-ea · 2026-09-09T12:56:22Z

+        assert_eq!(d.local_enr_raw, raw, "the signed record is byte-identical");
+        assert_eq!(d.fork_digest, digest);
+        assert!(d.previous_fork_digest.is_none(), "nothing was superseded");
+    }


is this related to head and head_v2 somehow?

Yes - for the purpose of head and head_v2 we update Status on events which previously did not generate an update - so I had tests created specifically to ensure the extra updates are not a problem for any downstream consumer of Status.

(specifics below)

The new mechanism publishes at loop end only if (head root, optimism, payload resolution) differs from the last emitted Status. The specific additional cases are:

Input When it now produces Status

EL VALID response, from NewPayload or FCU The selected head becomes validated, changing its optimism.

EL INVALID response Fork choice selects another head, or a Gloas head keeps its block root but changes payload resolution to empty.

Execution-payload envelope received over RPC Verification changes the selected head or its payload resolution. This path previously had no direct Status publication.

Disk replay of blocks or envelopes An iteration ends with a changed head observation. Previously replay published Status on Done; it can now publish intermediate observations too.

Slot processing without state advancement or a head-root change Payload resolution changes. The existing slot-start condition would not publish in this case.

Accepted blocks, accepted gossip envelopes and accepted PTC batches already published Status through on_accept. Ordinary slot advancement and vote-driven head-root changes also already published. Their snapshots are now richer; they do not inherently cause an extra message.

If any existing publication already reported the final observation, the new comparison emits nothing. Unrelated or repeated verdicts likewise add nothing.

i don't think we need to emit events for when we're replaying blocks from disk - nor do we need to spam status for each. it should be sufficient to send status once at the end of replay (successful or not)

how does payload resolution change on slot start without the block? does this have something to do with the proposer boost expiring?

i don't think we need to emit events for when we're replaying blocks from disk - nor do we need to spam status for each. it should be sufficient to send status once at the end of replay (successful or not)

We do it for simplicity. May add an extra check in beacon_api / application_boundary to avoid spamming the subscribers.

how does payload resolution change on slot start without the block? does this have something to do with the proposer boost expiring?

It can change without a new block because payload resolution depends on the current slot and fork choice votes - see resolves_to_full in fork_choice/head.rs. Proposer boost expiration is also an input to the same recalculation, see fork_choice_tick in fork_choice.rs

i don't think we need to emit events for when we're replaying blocks from disk - nor do we need to spam status for each. it should be sufficient to send status once at the end of replay (successful or not)

I am experimenting with two different approaches to withhold head notifications while replaying from disk. Both add replay_pending: bool to BeaconStateTile:

add replay_pending: bool also to BeaconStateEvent::Status (/// True until configured disk replay finishes or is skipped) - we need to set this in all the locations where this event is published so it has poor cohesion, then we give it special handing in ApplicationBoundaryTile to skip the Status update when it's true. See ada7a8a (added 13 lines in production code, 161 in tests)

withhold HeadRoots in head_roots (beacon_state/tile/src/tile.rs) if replay_pending in BeaconStateTile. This means there are now two different situations when HeadRoots is unavailable: during disk replay (very common) and when roots are unavailable because of overwritten history (probably not common). See 4b75ec6 (added 9 lines in production code, 175 in tests)

The latter appears to be brittle, because of the semantic ambiguity "head roots are unavailable, but we do not know why, and in principle they could be available - we just withhold them during disk replay". So, I would slightly prefer the first variant ada7a8a , even though it demonstrates worse cohesion. Your opinion @ninaiiad @vladimir-ea ?

Yet another alternative, similar to first above is to add following: bool to BeaconStateEvent::Status, used by application_boundary to distinguish synchronization progress from a head observation eligible for SSE. See 8010333

Yet another variant, where ObservedHead (structure created specifically to support this head and head_v2 events, in this PR) has a following: bool flag, switched when handling SyncUpdate::Following. I think this is probably the cleanest solution to avoid spamming head and head_v2 during disk replay b6f8a8b

PS. In the above commit, changes in control/src/sync_engine/mod.rs probably deserve more scrutiny.

vladimir-ea

looks good - would be good if @ninaiiad could take a look at the beacon state tile changes

ninaiiad · 2026-09-09T18:31:03Z

 #[repr(C)]
 pub enum BeaconStateEvent {
    ReplayComplete,
+    /// An observation that may repeat unchanged. Consumers decide which


why would it repeat unchanged? must it repeat unchanged?

Publication follows accepted work, even when that work changes none of the Status fields. Two examples:

An accepted sync committee message updates the contribution pool and seen-message tracking. The batch then calls on_accept(None), which unconditionally publishes Status. The head, import progress and other status fields can remain unchanged (see beacon_state/tile/src/tile/gossip.rs:511 and beacon_state/tile/src/tile/gossip.rs:445)

Accepting a parent can recursively import a buffered child. The child’s acceptance publishes its Status; returning to the parent’s acceptance publishes the same final Status again. See on_accept (beacon_state/tile/src/tile/orphan_pool.rs:190)

ninaiiad · 2026-09-09T18:34:07Z

+        assert_eq!(d.local_enr_raw, raw, "the signed record is byte-identical");
+        assert_eq!(d.fork_digest, digest);
+        assert!(d.previous_fork_digest.is_none(), "nothing was superseded");
+    }


i don't think we need to emit events for when we're replaying blocks from disk - nor do we need to spam status for each. it should be sufficient to send status once at the end of replay (successful or not)

ninaiiad · 2026-09-09T18:35:28Z

+        assert_eq!(d.local_enr_raw, raw, "the signed record is byte-identical");
+        assert_eq!(d.fork_digest, digest);
+        assert!(d.previous_fork_digest.is_none(), "nothing was superseded");
+    }


how does payload resolution change on slot start without the block? does this have something to do with the proposer boost expiring?

A checkpoint state can advance through empty slots beyond its latest block. Initializing the fork-choice anchor with the state slot caused P2P Status to advertise that later slot as the head. Use the latest block header's slot for the anchor node. An EF checkpoint fixture covers the case where the state is ahead of its latest block. Assisted-by: Claude Code:claude-opus-5 Assisted-by: Codex:GPT-6

Add the head topic to /eth/v1/events. Extend Status with the selected block's state root and both duty-dependent roots from its fork. Publish Status when the selected head or its execution optimism changes. An end-of-loop comparison covers changes not already reported by existing publication sites. Keep its marker separate from reorg detection. The application boundary filters repeated observations and computes epoch_transition. The first complete snapshot establishes a baseline. If checkpoint history has overwritten a required root, suppress the head event and retain that baseline. Node-status updates continue. Retaining the state root adds 32 bytes per fork-choice node. The larger Status increases the beacon-events ring allocation by 2 MiB. Tests cover snapshot construction, validation and head changes, history bounds, state remapping, baseline filtering, and topic-specific delivery. Fixture tests exercise import and replay metadata; a fork-choice test covers validation propagation through ancestors. Assisted-by: Claude Code:claude-opus-5 Assisted-by: Codex:GPT-6

Cover replay request gating, sync progress, checkpoint scheduling, column-buffer draining, and ENR stability when Status repeats. Verify that a deferred checkpoint remains eligible after catching up, and that a repeated head root drains columns buffered since the previous observation. Repeating an unchanged fork ID preserves the signed ENR and its sequence number. Extract StorageTile::on_status from the loop body to test checkpoint scheduling directly. These tests cover scheduling and buffer removal, not completed checkpoint writes or successful column validation. Assisted-by: Claude Code:claude-opus-5 Assisted-by: Codex:GPT-6

Add head_v2 alongside head and block. Status carries fork choice's empty/full resolution of the selected block's payload, and publishes an updated observation when that resolution changes. The application boundary publishes head_v2 when the root, optimism or payload resolution changes, including both empty-to-full and full-to-empty transitions. Legacy head retains its existing filter. Render the versioned response using the configured fork at the head block's slot. Reuse the existing dependent roots under the v2 field names. The added resolution field fits existing padding, leaving BeaconStateEvent at 240 bytes. Tests cover resolution and validation changes, slot and PTC effects, repeat suppression, fork-version selection, JSON mapping, and topic isolation through socket delivery. The slot test injects votes early to isolate the previous-slot rule; envelope and PTC tests bypass gossip validation. Assisted-by: Claude Code:claude-fable-5-1 Assisted-by: Codex:GPT-6

Assisted-by: Codex:GPT-6

The tile constructor seeds the anchor before any head observation. Make SelectedHead.idx mandatory and replace missing-node fallbacks with an explicit residency check. Keep overwritten-history handling unchanged. Test startup on both forks and return to the anchor after branch invalidation. Extend the finalization test through Status publication after pruning remaps the surviving head. These tests exercise the residency check through production publication paths. Assisted-by: Codex:gpt-6-astra

Assert required JSON fields after reading complete SSE chunks. Allow member reordering, insignificant whitespace, and future fields. Keep topic selection, payload changes, validation, and repeat suppression visible through socket observations. Observe completed producer operations instead of requiring publication at a particular internal step. Retain startup, resident-anchor, pruning, slot-transition, and reorg coverage. Import and replay metadata tests now enter through spine messages and derive slots from their fixtures. Consolidate overlapping consumer scenarios. Replace the columns buffer probe with valid-sidecar persistence after repeated Status observations. Remove redundant ENR coverage and the ineffective store-assignment test. Five renderer and socket tests pass with reordered fields, added spaces, and an extra field. Rebuilt negative controls catch wrong dependent roots and topics, lost payload changes, lost verdicts, and a missing reorg. They also catch premature replay requests, lost deferred checkpoints, and omitted column reconsideration. Synthetic producer setup still bypasses block and envelope validation. Checkpoint scheduling still uses private state with simulated consumption. The server's test marker uses private fan-out only to delimit observations. EF fixtures remain necessary for the import, replay, and sidecar scenarios. just fmt-check, just clippy, just nextest, and git diff --check pass. The workspace run passed 1,407 tests and skipped five. Production code is unchanged. Assisted-by: Codex:gpt-6-astra

Preserve head and head_v2 observations alongside gossip publication events. Keep both event queues active and retain cache-consumer cleanup. Adapt head fixtures to the shared-memory cleanup guard. Combine staged parent publication checks with repeated Status persistence coverage. Verify subscriptions can select all five event topics. Assisted-by: Codex:GPT-6

Publish head and head_v2 only for changes observed while Control reports Following. Disk restoration, the wait for a replay strategy and network catch-up produce no head notifications. The boundary keeps tracking every complete Status as its baseline in every mode, so a following period starts from the head the node already has and its first change is reported. No field is added to Status and beacon-state is unchanged. Status and sync updates travel on separate queues. The boundary reads the mode once per iteration after draining beacon events, so an observation drained in the same iteration as a mode change follows the earlier mode, and one queued between the two drains is reported in the next iteration. The imprecision is bounded by one iteration. The alternative that removes it is the following flag on Status in "Report following mode in Status to gate head events". Extend Control's replay gate to hold Following until replay finishes or is skipped. Previously it held only network requests, and disk replay is chosen exactly when peers look comparable, so Following could be announced during replay. Completion re-evaluates the target. Tests: the observer reports only while following and baselines in every mode; a TCP subscriber sees changes across two following periods while node status follows every observation; Control waits for completion before Following. Amend ADR-0004. just fmt-check, just clippy and git diff --check pass. nextest for silver_application_boundary and silver_control under the CI profile: 111 passed. The workspace suite was not run. Assisted-by: Claude:claude-fable-5-1

Bronek · 2026-09-14T14:09:47Z

i don't think we need to emit events for when we're replaying blocks from disk - nor do we need to spam status for each. it should be sufficient to send status once at the end of replay (successful or not)

Resolved in a28748f (previously detached commit b6f8a8b)

Preserve the anchor block header for Status while selecting the Gloas checkpoint execution hash from the latest payload bid. Repair the following-mode test fixture to use the shared-memory cleanup guard. Assisted-by: Codex:gpt-6-astra

Preserve SSE head exports alongside cluster messaging exports. Assisted-by: Codex:gpt-6-astra

Bronek changed the title ~~Serve head events over SSE~~ Serve head and head_v2 events over SSE Sep 9, 2026

Bronek marked this pull request as ready for review September 9, 2026 12:03

Bronek requested review from ninaiiad and vladimir-ea September 9, 2026 12:04

vladimir-ea reviewed Sep 9, 2026

View reviewed changes

Comment thread crates/common/src/spine/messages.rs

vladimir-ea reviewed Sep 9, 2026

View reviewed changes

Comment thread crates/common/src/spine/messages.rs

vladimir-ea reviewed Sep 9, 2026

View reviewed changes

Comment thread crates/common/src/spine/messages.rs

vladimir-ea reviewed Sep 9, 2026

View reviewed changes

Comment thread crates/common/src/spine/messages.rs Outdated

vladimir-ea approved these changes Sep 9, 2026

View reviewed changes

ninaiiad reviewed Sep 9, 2026

View reviewed changes

This was referenced Sep 11, 2026

Serve block_gossip events for gossip publication requests #248

Merged

Serve data_column_sidecar events for gossip publication requests #249

Merged

Bronek force-pushed the bronek/sse_head branch from 33a8a67 to 1ab497b Compare September 11, 2026 15:53

Bronek added 11 commits September 11, 2026 18:02

Use B256 for HeadRoots fields and the unavailable-root check

1ca62fa

Assisted-by: Codex:GPT-6

Compose SelectedHead from HeadObservation and the node index

f5cc651

Assisted-by: Codex:GPT-6

Rename the head_v2 renderer parameter to fork_name

6ab6775

Assisted-by: Codex:GPT-6

Update comment

f8424f3

Comment update, clarify when will diverge from

a56472e

Assisted-by: Codex:GPT-6

Bronek force-pushed the bronek/sse_head branch from 1ab497b to dd5644f Compare September 11, 2026 17:20

Bronek added 2 commits September 14, 2026 13:48

Merge main into bronek/sse_head

5cd4021

Preserve the anchor block header for Status while selecting the Gloas checkpoint execution hash from the latest payload bid. Repair the following-mode test fixture to use the shared-memory cleanup guard. Assisted-by: Codex:gpt-6-astra

Merge main into bronek/sse_head

bad4fb9

Preserve SSE head exports alongside cluster messaging exports. Assisted-by: Codex:gpt-6-astra

Input	When it now produces Status
EL `VALID` response, from `NewPayload` or FCU	The selected head becomes validated, changing its optimism.
EL `INVALID` response	Fork choice selects another head, or a Gloas head keeps its block root but changes payload resolution to empty.
Execution-payload envelope received over RPC	Verification changes the selected head or its payload resolution. This path previously had no direct Status publication.
Disk replay of blocks or envelopes	An iteration ends with a changed head observation. Previously replay published Status on `Done`; it can now publish intermediate observations too.
Slot processing without state advancement or a head-root change	Payload resolution changes. The existing slot-start condition would not publish in this case.

Uh oh!

Conversation

Bronek commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bronek Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bronek Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bronek Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimir-ea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bronek commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Bronek commented Sep 9, 2026 •

edited

Loading

Bronek Sep 9, 2026 •

edited

Loading

Bronek Sep 10, 2026 •

edited

Loading

Bronek Sep 14, 2026 •

edited

Loading