Skip to content

fix(ci): pin consensus-spec-tests fixture to v1.6.0-alpha.6#1900

Closed
pauldelucia wants to merge 1 commit into
ethereum:masterfrom
pauldelucia:fix/pin-consensus-spec-tests
Closed

fix(ci): pin consensus-spec-tests fixture to v1.6.0-alpha.6#1900
pauldelucia wants to merge 1 commit into
ethereum:masterfrom
pauldelucia:fix/pin-consensus-spec-tests

Conversation

@pauldelucia

Copy link
Copy Markdown

Summary

`ci/circleci: test` has been silently failing on master for ~7 months
because the Makefile downloads `mainnet.tar.gz` from "the latest
`consensus-spec-tests` release" (`head -n 1` of the releases API),
unpinned. `testing/ef-tests/tests/build_proofs.rs` hardcodes expected
SSZ proofs over `/ssz_random/` fixtures, which are regenerated with
fresh randomness on every release — so the unpinned download rots
the hardcoded proofs the moment a new release ships.

Timeline

Date Event
2025-09-23 06:48 UTC PR #1896 merged; `build_proofs.rs` last touched. CI green (CI 40491)
2025-09-23 12:33 UTC PR #1897 merged. CI still green (CI 40514)
2025-09-24 14:11 UTC `consensus-spec-tests v1.6.0-beta.0` released — new randomness in `bellatrix/ssz_static/.../ssz_random/case_0/value.yaml`
2026-04-28 onward Every PR's CI fetches the post-Sep-24 fixtures; all four bellatrix proof tests fail with "left != right" Merkle path mismatches (e.g. PR #1899)

The 4 failing tests are all bellatrix random-case fixtures
(`beacon_block_body_root_proof`, `historical_batch_block_root_proof`,
`block_body_execution_payload_proof`,
`execution_payload_block_hash_proof`); the 3 passing tests use
fixtures the new release didn't perturb.

Fix

Introduce `EF_TESTS_VERSION` and construct the download URL directly
instead of scraping the GitHub releases API:

```makefile
EF_TESTS_VERSION = v1.6.0-alpha.6
EF_TESTS_URL = https://github.com/ethereum/consensus-spec-tests/releases/download/\$(EF_TESTS_VERSION)/\$(EF_TESTS_TARGET)
```

`v1.6.0-alpha.6` was the release current on 2025-09-23 (when
`build_proofs.rs` was last updated and CI was green), so this
restores the exact fixture set the hardcoded proofs were written
against. `ALL_RELEASES_URL` and the curl-grep-head pipeline are
removed since they no longer do anything.

Test plan

Verified locally — with this pin, all 7 `ef-tests` pass:

```
$ rm -rf testing/ef-tests/mainnet
$ make ef-tests
running 7 tests
test execution_payload_block_hash_proof ... ok
test beacon_block_body_root_proof ... ok
test block_body_execution_payload_proof ... ok
test historical_batch_block_root_proof ... ok
test test_historical_summaries_with_proof ... ok
test beacon_state_historical_summaries_proof_deneb ... ok
test beacon_state_historical_summaries_proof_electra ... ok

test result: ok. 7 passed; 0 failed
```

Notes

This is the follow-up to #1899 promised in that PR's
comment
.
The two are independent — this one unblocks the CI on every open and
future PR; #1899 is a separate `vergen` build-deps pin needed by
downstream consumers of the published `ethportal-api 0.12` crate.

When `consensus-spec-tests` next ships a release with new randomness,
the bump should be deliberate: refresh the expected-proof arrays in
`build_proofs.rs` against the new fixtures, then bump the version
constant in the same commit. Worth considering a CI job that warns
when a new `consensus-spec-tests` release is out, but that's beyond
this PR's scope.

The Makefile downloaded `mainnet.tar.gz` from "the latest
consensus-spec-tests release" (`head -n 1` of the releases API).
`testing/ef-tests/tests/build_proofs.rs` hardcodes expected SSZ proofs
over `*/ssz_random/*` fixtures, which are regenerated with fresh
randomness on each release — so the unpinned download silently rots
the proofs the moment a new release lands.

Timeline:

* 2025-09-23: PR ethereum#1896 last touched `build_proofs.rs`. CI green —
  current release was `v1.6.0-alpha.6` (2025-08-30).
* 2025-09-24: `consensus-spec-tests v1.6.0-beta.0` shipped, with new
  randomness in the bellatrix `ssz_random/case_0/value.yaml` fixtures.
* Every subsequent CI run (e.g. PR ethereum#1899) downloaded `v1.6.0-beta.0`
  (or whatever's newest) and failed all four bellatrix proof tests
  with "left != right" Merkle path mismatches.

Fix: introduce `EF_TESTS_VERSION` (defaulted to `v1.6.0-alpha.6`, the
release `build_proofs.rs` was last updated against) and construct the
download URL from it directly. Drops `ALL_RELEASES_URL` and the
curl-grep-head pipeline since they no longer have anything to do.

Verified locally: with this pin, all 7 ef-tests pass:

    $ rm -rf testing/ef-tests/mainnet
    $ make ef-tests
    test result: ok. 7 passed; 0 failed; 0 ignored

When `consensus-spec-tests` next ships a release with new randomness,
the bump should be deliberate: refresh the expected-proof arrays in
`build_proofs.rs` against the new fixtures, then bump the version
constant in the same commit.
@pauldelucia

Copy link
Copy Markdown
Author

Closing — trin is unmaintained per the README banner; master CI isn't being run and won't be. The diagnosis (unpinned consensus-spec-tests download in the Makefile) and the proposed pin to v1.6.0-alpha.6 (the release current when build_proofs.rs was last updated) are still correct for any fork. Cross-linked from #1899.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant