Skip to content

feat: range sync for gloas#9155

Closed
twoeths wants to merge 10 commits intote/gloas_process_chain_segmentfrom
te/gloas_range_sync
Closed

feat: range sync for gloas#9155
twoeths wants to merge 10 commits intote/gloas_process_chain_segmentfrom
te/gloas_range_sync

Conversation

@twoeths
Copy link
Copy Markdown
Contributor

@twoeths twoeths commented Apr 2, 2026

Motivation

Support range sync for gloas.

Description

  • Range sync — gloas envelope support: Batch state now carries payloadEnvelopes: Map<Slot, PayloadEnvelopeInput> | null across all state transitions (AwaitingDownload, Downloading, AwaitingProcessing, Processing, AwaitingValidation). Batch.getRequests() emits an envelopesRequest for post-gloas forks, independently of the DA retention window.

  • downloadByRange: Extended to request and validate SignedExecutionPayloadEnvelopes. Returns {responses, payloadEnvelopes} instead of bare ValidatedResponses. A new validateEnvelopesByRangeResponse function verifies each envelope's beaconBlockRoot matches the corresponding downloaded block.

  • cacheByRangeResponses: Now accepts downloadedPayloadEnvelopes, existingPayloadEnvelopes, custodyConfig, and seenTimestampSec. Constructs PayloadEnvelopeInput objects for gloas slots, merges in gloas DataColumnSidecars (differentiated from fulu columns by isGloasDataColumnSidecar), and returns {blocks, payloadEnvelopes}.

  • validateColumnsByRangeResponse: Handles both fulu (signedBlockHeader.message.slot) and gloas (column.slot) DataColumnSidecar shapes. Dispatches to validateGloasBlockDataColumnSidecars or validateFuluBlockDataColumnSidecars based on fork.

  • processChainSegment wiring: RangeSync.processChainSegment and SyncChain.processBatch now forward payloadEnvelopes to chain.processChainSegment instead of hard-coding null.

  • Fork-choice — proposer boost fix: Proposer boost (current and previous) is applied only to the EMPTY variant for gloas blocks to prevent a 2x boost from the score being applied to both EMPTY and FULL then back-propagated to PENDING.

  • Fork-choice — getParentNodeIndex fix: Wrapped the gloas traversal path in a try/catch; UNKNOWN_PARENT_BLOCK errors (finalized block boundary) now return undefined instead of throwing.

  • Fork-choice — updateHead slot fix: updateHead() accepts an optional slot parameter. prepareNextSlot passes currentSlot + 1 so the gloas FULL/EMPTY tie-breaker uses the correct upcoming slot.

AI Assistance Disclosure

Used Claude Code to assist with implementation and review.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements support for the Gloas fork, primarily focusing on the integration of execution payload envelopes and data columns within the range sync and fork choice modules. Key updates include extending the sync batch state to manage payload envelopes, adding validation logic for these envelopes during range requests, and refining fork choice head recomputation to support Gloas tie-breaker logic. Review feedback identified critical logic errors in the range sync implementation where partial downloads could lead to data loss by overwriting existing payload containers or discarding columns when envelopes are missing. Suggestions were also provided to improve the accuracy of download progress tracking and ensure type safety for column sidecars.

Comment on lines +229 to +247
for (const [slot, envelope] of downloadedPayloadEnvelopes) {
const blockInput = updatedBatchBlocks.get(slot);
if (blockInput?.hasBlock() && isForkPostGloas(blockInput.forkName)) {
const payloadEnvelopeInput = PayloadEnvelopeInput.createFromBlock({
blockRootHex: blockInput.blockRootHex,
block: blockInput.getBlock() as SignedBeaconBlock<ForkPostGloas>,
forkName: blockInput.forkName,
sampledColumns: custodyConfig.sampledColumns,
custodyColumns: custodyConfig.custodyColumns,
timeCreatedSec: blockInput.timeCreatedSec,
});
payloadEnvelopeInput.addPayloadEnvelope({
envelope,
source: PayloadEnvelopeInputSource.byRange,
seenTimestampSec,
peerIdStr,
});
payloadEnvelopes.set(slot, payloadEnvelopeInput);
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This logic causes data loss during partial downloads. If existingPayloadEnvelopes already contains a PayloadEnvelopeInput for a slot (e.g., with previously downloaded columns), creating a new one via createFromBlock and calling set(slot, ...) will overwrite the existing container and discard the cached columns. It should check for an existing container first.

    for (const [slot, envelope] of downloadedPayloadEnvelopes) {
      const blockInput = updatedBatchBlocks.get(slot);
      if (blockInput?.hasBlock() && isForkPostGloas(blockInput.forkName)) {
        let payloadEnvelopeInput = payloadEnvelopes.get(slot);
        if (!payloadEnvelopeInput) {
          payloadEnvelopeInput = PayloadEnvelopeInput.createFromBlock({
            blockRootHex: blockInput.blockRootHex,
            block: blockInput.getBlock() as SignedBeaconBlock<ForkPostGloas>,
            forkName: blockInput.forkName,
            sampledColumns: custodyConfig.sampledColumns,
            custodyColumns: custodyConfig.custodyColumns,
            timeCreatedSec: blockInput.timeCreatedSec,
          });
          payloadEnvelopes.set(slot, payloadEnvelopeInput);
        }
        payloadEnvelopeInput.addPayloadEnvelope({
          envelope,
          source: PayloadEnvelopeInputSource.byRange,
          seenTimestampSec,
          peerIdStr,
        });
      }
    }

Comment on lines +250 to +263
for (const {columnSidecars} of responses.validatedColumnSidecars ?? []) {
const firstColumn = columnSidecars[0];
if (!firstColumn || !isGloasDataColumnSidecar(firstColumn)) continue;
const payloadEnvelopeInput = payloadEnvelopes.get(firstColumn.slot);
if (!payloadEnvelopeInput) continue;
for (const columnSidecar of columnSidecars as gloas.DataColumnSidecar[]) {
payloadEnvelopeInput.addColumn({
columnSidecar,
source: PayloadEnvelopeInputSource.byRange,
seenTimestampSec,
peerIdStr,
});
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Gloas columns are discarded if they are downloaded for a slot that doesn't have an envelope yet (and didn't just download one in this response), because payloadEnvelopes.get(firstColumn.slot) will return undefined. The container should be created if it doesn't exist, provided the block is available.

    // Add gloas DataColumnSidecars to their corresponding PayloadEnvelopeInput
    for (const {columnSidecars} of responses.validatedColumnSidecars ?? []) {
      const firstColumn = columnSidecars[0];
      if (!firstColumn || !isGloasDataColumnSidecar(firstColumn)) continue;
      const slot = firstColumn.slot;
      let payloadEnvelopeInput = payloadEnvelopes.get(slot);
      if (!payloadEnvelopeInput) {
        const blockInput = updatedBatchBlocks.get(slot);
        if (blockInput?.hasBlock() && isForkPostGloas(blockInput.forkName)) {
          payloadEnvelopeInput = PayloadEnvelopeInput.createFromBlock({
            blockRootHex: blockInput.blockRootHex,
            block: blockInput.getBlock() as SignedBeaconBlock<ForkPostGloas>,
            forkName: blockInput.forkName,
            sampledColumns: custodyConfig.sampledColumns,
            custodyColumns: custodyConfig.custodyColumns,
            timeCreatedSec: blockInput.timeCreatedSec,
          });
          payloadEnvelopes.set(slot, payloadEnvelopeInput);
        }
      }
      if (payloadEnvelopeInput) {
        for (const columnSidecar of columnSidecars as gloas.DataColumnSidecar[]) {
          payloadEnvelopeInput.addColumn({
            columnSidecar,
            source: PayloadEnvelopeInputSource.byRange,
            seenTimestampSec,
            peerIdStr,
          });
        }
      }
    }

if (blockInput.hasBlock() && blockStartSlot === blockSlot) {
blockStartSlot = blockSlot + 1;
}
if (blockInput.hasBlock() && envelopeStartSlot === blockSlot && envelopesBySlot.has(blockSlot)) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The envelopeStartSlot should only advance if the execution payload envelope is actually present in the container, not just because the container exists (as it might only contain columns).

Suggested change
if (blockInput.hasBlock() && envelopeStartSlot === blockSlot && envelopesBySlot.has(blockSlot)) {
if (blockInput.hasBlock() && envelopeStartSlot === blockSlot && envelopesBySlot.get(blockSlot)?.hasPayloadEnvelope()) {

let blocks: undefined | SignedBeaconBlock[];
let blobSidecars: undefined | deneb.BlobSidecars;
let columnSidecars: undefined | fulu.DataColumnSidecar[];
let columnSidecars: undefined | fulu.DataColumnSidecars;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The type for columnSidecars should be the union DataColumnSidecar[] to correctly support both Fulu and Gloas shapes during range requests.

Suggested change
let columnSidecars: undefined | fulu.DataColumnSidecars;
let columnSidecars: undefined | DataColumnSidecar[];

@twoeths
Copy link
Copy Markdown
Contributor Author

twoeths commented Apr 3, 2026

blocked by #9103 #9165 #9164

ensi321 added a commit that referenced this pull request Apr 15, 2026
Range sync now fetches and validates execution payload envelopes alongside
blocks for Gloas forks. With deferred processing, blocks can sync
optimistically without envelopes, but envelopes are fetched in parallel
for fork-choice FULL/EMPTY variant accuracy.

Changes:
- Batch carries payloadEnvelopes across all state transitions
- downloadByRange fetches envelopes via sendExecutionPayloadEnvelopesByRange
- validateEnvelopesByRangeResponse verifies beaconBlockRoot matches blocks
- processChainSegment accepts payloadEnvelopes parameter
- SyncChainFns types updated for envelope data flow
- Add INVALID_ENVELOPE_BEACON_BLOCK_ROOT error code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ensi321 ensi321 mentioned this pull request Apr 15, 2026
5 tasks
ensi321 added a commit that referenced this pull request Apr 16, 2026
Range sync now fetches and validates execution payload envelopes alongside
blocks for Gloas forks. With deferred processing, blocks can sync
optimistically without envelopes, but envelopes are fetched in parallel
for fork-choice FULL/EMPTY variant accuracy.

Changes:
- Batch carries payloadEnvelopes across all state transitions
- downloadByRange fetches envelopes via sendExecutionPayloadEnvelopesByRange
- validateEnvelopesByRangeResponse verifies beaconBlockRoot matches blocks
- processChainSegment accepts payloadEnvelopes parameter
- SyncChainFns types updated for envelope data flow
- Add INVALID_ENVELOPE_BEACON_BLOCK_ROOT error code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ensi321 added a commit that referenced this pull request Apr 20, 2026
Range sync now fetches and validates execution payload envelopes alongside
blocks for Gloas forks. With deferred processing, blocks can sync
optimistically without envelopes, but envelopes are fetched in parallel
for fork-choice FULL/EMPTY variant accuracy.

Changes:
- Batch carries payloadEnvelopes across all state transitions
- downloadByRange fetches envelopes via sendExecutionPayloadEnvelopesByRange
- validateEnvelopesByRangeResponse verifies beaconBlockRoot matches blocks
- processChainSegment accepts payloadEnvelopes parameter
- SyncChainFns types updated for envelope data flow
- Add INVALID_ENVELOPE_BEACON_BLOCK_ROOT error code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@wemeetagain wemeetagain added the spec-gloas Issues targeting the Glamsterdam spec version label Apr 20, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.60%. Comparing base (af5ab38) to head (9a275ca).
⚠️ Report is 23 commits behind head on te/gloas_process_chain_segment.

Additional details and impacted files
@@                        Coverage Diff                         @@
##           te/gloas_process_chain_segment    #9155      +/-   ##
==================================================================
+ Coverage                           52.57%   52.60%   +0.02%     
==================================================================
  Files                                 848      848              
  Lines                               61378    61273     -105     
  Branches                             4528     4524       -4     
==================================================================
- Hits                                32268    32231      -37     
+ Misses                              29045    28977      -68     
  Partials                               65       65              
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@twoeths
Copy link
Copy Markdown
Contributor Author

twoeths commented Apr 21, 2026

this does not work for the latest spec alpha.5
superseded by #9242

@twoeths twoeths closed this Apr 21, 2026
@twoeths twoeths deleted the te/gloas_range_sync branch April 21, 2026 07:22
ensi321 added a commit that referenced this pull request Apr 22, 2026
Range sync now fetches and validates execution payload envelopes alongside
blocks for Gloas forks. With deferred processing, blocks can sync
optimistically without envelopes, but envelopes are fetched in parallel
for fork-choice FULL/EMPTY variant accuracy.

Changes:
- Batch carries payloadEnvelopes across all state transitions
- downloadByRange fetches envelopes via sendExecutionPayloadEnvelopesByRange
- validateEnvelopesByRangeResponse verifies beaconBlockRoot matches blocks
- processChainSegment accepts payloadEnvelopes parameter
- SyncChainFns types updated for envelope data flow
- Add INVALID_ENVELOPE_BEACON_BLOCK_ROOT error code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spec-gloas Issues targeting the Glamsterdam spec version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants