fix(ci): pin consensus-spec-tests fixture to v1.6.0-alpha.6#1900
Closed
pauldelucia wants to merge 1 commit into
Closed
fix(ci): pin consensus-spec-tests fixture to v1.6.0-alpha.6#1900pauldelucia wants to merge 1 commit into
pauldelucia wants to merge 1 commit into
Conversation
The Makefile downloaded `mainnet.tar.gz` from "the latest consensus-spec-tests release" (`head -n 1` of the releases API). `testing/ef-tests/tests/build_proofs.rs` hardcodes expected SSZ proofs over `*/ssz_random/*` fixtures, which are regenerated with fresh randomness on each release — so the unpinned download silently rots the proofs the moment a new release lands. Timeline: * 2025-09-23: PR ethereum#1896 last touched `build_proofs.rs`. CI green — current release was `v1.6.0-alpha.6` (2025-08-30). * 2025-09-24: `consensus-spec-tests v1.6.0-beta.0` shipped, with new randomness in the bellatrix `ssz_random/case_0/value.yaml` fixtures. * Every subsequent CI run (e.g. PR ethereum#1899) downloaded `v1.6.0-beta.0` (or whatever's newest) and failed all four bellatrix proof tests with "left != right" Merkle path mismatches. Fix: introduce `EF_TESTS_VERSION` (defaulted to `v1.6.0-alpha.6`, the release `build_proofs.rs` was last updated against) and construct the download URL from it directly. Drops `ALL_RELEASES_URL` and the curl-grep-head pipeline since they no longer have anything to do. Verified locally: with this pin, all 7 ef-tests pass: $ rm -rf testing/ef-tests/mainnet $ make ef-tests test result: ok. 7 passed; 0 failed; 0 ignored When `consensus-spec-tests` next ships a release with new randomness, the bump should be deliberate: refresh the expected-proof arrays in `build_proofs.rs` against the new fixtures, then bump the version constant in the same commit.
3 tasks
Author
|
Closing — trin is unmaintained per the README banner; master CI isn't being run and won't be. The diagnosis (unpinned |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
`ci/circleci: test` has been silently failing on master for ~7 months
because the Makefile downloads `mainnet.tar.gz` from "the latest
`consensus-spec-tests` release" (`head -n 1` of the releases API),
unpinned. `testing/ef-tests/tests/build_proofs.rs` hardcodes expected
SSZ proofs over `/ssz_random/` fixtures, which are regenerated with
fresh randomness on every release — so the unpinned download rots
the hardcoded proofs the moment a new release ships.
Timeline
The 4 failing tests are all bellatrix random-case fixtures
(`beacon_block_body_root_proof`, `historical_batch_block_root_proof`,
`block_body_execution_payload_proof`,
`execution_payload_block_hash_proof`); the 3 passing tests use
fixtures the new release didn't perturb.
Fix
Introduce `EF_TESTS_VERSION` and construct the download URL directly
instead of scraping the GitHub releases API:
```makefile
EF_TESTS_VERSION = v1.6.0-alpha.6
EF_TESTS_URL = https://github.com/ethereum/consensus-spec-tests/releases/download/\$(EF_TESTS_VERSION)/\$(EF_TESTS_TARGET)
```
`v1.6.0-alpha.6` was the release current on 2025-09-23 (when
`build_proofs.rs` was last updated and CI was green), so this
restores the exact fixture set the hardcoded proofs were written
against. `ALL_RELEASES_URL` and the curl-grep-head pipeline are
removed since they no longer do anything.
Test plan
Verified locally — with this pin, all 7 `ef-tests` pass:
```
$ rm -rf testing/ef-tests/mainnet
$ make ef-tests
running 7 tests
test execution_payload_block_hash_proof ... ok
test beacon_block_body_root_proof ... ok
test block_body_execution_payload_proof ... ok
test historical_batch_block_root_proof ... ok
test test_historical_summaries_with_proof ... ok
test beacon_state_historical_summaries_proof_deneb ... ok
test beacon_state_historical_summaries_proof_electra ... ok
test result: ok. 7 passed; 0 failed
```
Notes
This is the follow-up to #1899 promised in that PR's
comment.
The two are independent — this one unblocks the CI on every open and
future PR; #1899 is a separate `vergen` build-deps pin needed by
downstream consumers of the published `ethportal-api 0.12` crate.
When `consensus-spec-tests` next ships a release with new randomness,
the bump should be deliberate: refresh the expected-proof arrays in
`build_proofs.rs` against the new fixtures, then bump the version
constant in the same commit. Worth considering a CI job that warns
when a new `consensus-spec-tests` release is out, but that's beyond
this PR's scope.