From 456b1a791276f201f49cd036c9525fe8e386efc4 Mon Sep 17 00:00:00 2001 From: Stephen Matula Date: Mon, 14 Sep 2026 11:30:44 -0400 Subject: [PATCH 1/3] Pin rustup-init to an immutable versioned URL Every plumbing-builder PR pipeline has failed since 2026-09-01, in two different places. Only the second one is ours to fix. The first, GNU autoconf/automake/libtool at step 5 of 29, was a fault in ftpmirror.gnu.org itself and has resolved on its own: curl -fsSL --retry 10 -o autoconf-2.72.tar.gz \ https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.72.tar.gz curl: (22) The requested URL returned error: 404 The redirector had started appending the request path to each mirror's GNU directory rather than to its parent, producing a doubled segment (.../mirrors/gnu//gnu/autoconf/autoconf-2.72.tar.gz). Our URLs were correct throughout; --retry could not help, since curl's transient-error set excludes 404 and the mirror choice is a pure function of the request path, so every attempt landed on the same broken mirror. We were not alone: chimera-linux/cports#6169, fedora-infra/anitya#2073 (which quotes the identical doubled path on the identical mirror we were assigned), LibreELEC/LibreELEC.tv#11754 and #11759, and easybuilders/easybuild-framework #5283 all report it over the same window. GNU published no announcement, but the last third-party 404 is timestamped 2026-09-14T13:13Z and the path serves correctly again now -- reconfirmed over six probes across all three tools, 6/6 HTTP 206 on /gnu// and 6/6 404 on the bare form. So the Containerfile's GNU URLs are left exactly as they are; the mirrors are the availability risk here, not the paths, and every tarball is check_sha256sum-verified after download regardless. That leaves the real bug, rustup-init (build-rust.sh, step 12 of 29): rustup-init: FAILED sha256sum: WARNING: 1 computed checksum did NOT match RUST_DOWNLOAD_URL pointed at https://static.rust-lang.org/rustup/dist, and the fetched filename is a bare "rustup-init" -- no version anywhere in the path. That endpoint always serves the newest rustup, so RUST_HASH pinned the bytes of a moving target and broke as soon as rustup cut a release. The pinned hash 4acc9acc is rustup 1.29.0 exactly; dist/ now serves 1.29.1 (dda72343). Nothing was tampered with: the checksum correctly refused a file that had legitimately changed. Fetch from the immutable .../rustup/archive/${RUSTUP_VERSION}/ path so RUST_HASH pins something that cannot move, with the rustup version held in its own variable alongside RUST_VERSION rather than buried in the URL, and record in a comment that RUST_HASH covers the installer rather than the toolchain. That distinction is easy to miss: RUST_VERSION is passed to --default-toolchain and selects the compiler, while the rustup version only decides which installer runs. Cargo is not pinned separately; it ships with the toolchain and follows RUST_VERSION. Also bump RUST_VERSION 1.95.0 -> 1.98.1, carried over from #418, which was closed so it could be folded in here. Every other fetch_source in builder/build_scripts/ interpolates a *_ROOT variable that carries the version in the filename, so rustup-init was the only download exposed to this class of failure. Verified on the Konflux PR build: all 29 stages ran clean, zero curl errors across 18k log lines, and "rustc 1.98.1 (48a229cea 2026-09-01)" installed. Assisted-by: Claude Opus 5 --- builder/Containerfile | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/builder/Containerfile b/builder/Containerfile index c1f1baef..1364fdb9 100644 --- a/builder/Containerfile +++ b/builder/Containerfile @@ -144,10 +144,16 @@ RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ FROM build_base AS build_rust COPY build_scripts/build-rust.sh /opt/_internal/build_scripts/ +# RUST_HASH is the sha256 of rustup-init (the installer), not of the Rust +# toolchain: RUSTUP_VERSION and RUST_VERSION move independently. The unversioned +# .../rustup/dist path always serves the newest rustup, so RUSTUP_VERSION is +# pinned in the URL; otherwise the hash breaks on every rustup release. Bumping +# RUSTUP_VERSION therefore requires a matching RUST_HASH. RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ - export RUST_VERSION=1.95.0 && \ - export RUST_HASH=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 && \ - export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/dist && \ + export RUST_VERSION=1.98.1 && \ + export RUSTUP_VERSION=1.29.1 && \ + export RUST_HASH=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 && \ + export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION} && \ /tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/build-rust.sh From 6e15e150efbd71e8c41d93eda5c2fb176b9b9ea8 Mon Sep 17 00:00:00 2001 From: Stephen Matula Date: Tue, 15 Sep 2026 07:16:50 -0400 Subject: [PATCH 2/3] Fetch the GNU autotools from ftp.gnu.org instead of the mirror redirector The builder pipeline cannot get through step 5 of 29. autoconf, automake and libtool are fetched from ftpmirror.gnu.org, which 302s to a volunteer mirror chosen per request, and enough of that pool is unusable that a build needing three consecutive successes rarely gets them. Of twelve failed runs whose logs were examined, five died on a 404, two on a 502, two on a 504 and three on a refused connection; both runs today failed here as well. The paths are correct -- this is not the redirector regression that broke the same step earlier this month, which GNU has since fixed. The failures are the mirrors themselves, and they vary per request: autoconf curl: (7) Failed to connect to mirror2.evolution-host.com:443: Connection timed out automake curl: (22) The requested URL returned error: 404 Not Found Two distinct faults on two consecutive runs, and in the second run autoconf succeeded before automake failed -- the redirector picks independently per path, so each download is its own coin flip. The 404s are mirrors that hold the archive at their document root rather than under /gnu/; probing 20 paths across the pool, 4 answer the /gnu/ layout with 404 no matter how healthy they are. --retry cannot route around either fault: curl's transient set excludes 404, and a retry re-requests the same path and so lands on the same mirror. Point the three URLs at ftp.gnu.org, which serves the archive directly. Probed four times per tarball, 12/12 HTTP 206, and a full fetch of autoconf-2.72 matches the pinned sha256. Losing the mirror pool costs redundancy, but a pool this unreliable was supplying variance, not availability, and every tarball is check_sha256sum-verified after download regardless -- the host is an availability concern, not a trust boundary. The CI workflow comment in .github/workflows/test.yml named ftpmirror.gnu.org as the reason for pulling the published image rather than rebuilding. Drop the host-specific claim; the remaining reasons, that Konflux builds the real image and denylist coverage is the job above, stand on their own. A general fix belongs in fetch_source -- reroll onto a different mirror and fall back to the canonical host -- but that changes the one code path every download in the builder shares, so it wants its own review rather than riding along with an outage fix. Assisted-by: Claude Opus 5 --- .github/workflows/test.yml | 5 ++--- builder/Containerfile | 6 +++--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a385f5c6..7b34b44f 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -27,8 +27,7 @@ jobs: - name: Checkout code uses: actions/checkout@v6 - # Always pull the published builder image. Rebuilding it here downloads - # autoconf/automake from ftpmirror.gnu.org, which is unreliable on GHA. - # Konflux builds the real image; denylist coverage is the job above. + # Always pull the published builder image rather than rebuilding it here: + # Konflux builds the real image, and denylist coverage is the job above. - name: Run test suite (pull published builder image) run: ./tests/test-build-wheels.sh --skip-build diff --git a/builder/Containerfile b/builder/Containerfile index 1364fdb9..57a3536e 100644 --- a/builder/Containerfile +++ b/builder/Containerfile @@ -59,21 +59,21 @@ COPY build_scripts/install-autoconf.sh /opt/_internal/build_scripts/ RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ export AUTOCONF_ROOT=autoconf-2.72 && \ export AUTOCONF_HASH=afb181a76e1ee72832f6581c0eddf8df032b83e2e0239ef79ebedc4467d92d6e && \ - export AUTOCONF_DOWNLOAD_URL=https://ftpmirror.gnu.org/gnu/autoconf && \ + export AUTOCONF_DOWNLOAD_URL=https://ftp.gnu.org/gnu/autoconf && \ /tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/install-autoconf.sh COPY build_scripts/install-automake.sh /opt/_internal/build_scripts/ RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ export AUTOMAKE_ROOT=automake-1.18.1 && \ export AUTOMAKE_HASH=63e585246d0fc8772dffdee0724f2f988146d1a3f1c756a3dc5cfbefa3c01915 && \ - export AUTOMAKE_DOWNLOAD_URL=https://ftpmirror.gnu.org/gnu/automake && \ + export AUTOMAKE_DOWNLOAD_URL=https://ftp.gnu.org/gnu/automake && \ /tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/install-automake.sh COPY build_scripts/install-libtool.sh /opt/_internal/build_scripts/ RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ export LIBTOOL_ROOT=libtool-2.5.4 && \ export LIBTOOL_HASH=da8ebb2ce4dcf46b90098daf962cffa68f4b4f62ea60f798d0ef12929ede6adf && \ - export LIBTOOL_DOWNLOAD_URL=https://ftpmirror.gnu.org/gnu/libtool && \ + export LIBTOOL_DOWNLOAD_URL=https://ftp.gnu.org/gnu/libtool && \ /tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/install-libtool.sh COPY build_scripts/install-libxcrypt.sh /opt/_internal/build_scripts/ From b4b34d6f2994780832bbf44d531a219b32be4a18 Mon Sep 17 00:00:00 2001 From: Stephen Matula Date: Tue, 15 Sep 2026 07:25:16 -0400 Subject: [PATCH 3/3] Rename RUST_HASH and RUST_DOWNLOAD_URL to RUSTUP_* Both describe rustup-init, the installer, but sat beside RUST_VERSION and read as though they belonged to it. That misreading is not hypothetical: #418 bumped RUST_VERSION and replaced the hash together, described as "update the Rust release checksum", when the new value was in fact the rustup 1.29.1 installer's. It went green for the wrong reason. RUSTUP_HASH and RUSTUP_DOWNLOAD_URL pair both with RUSTUP_VERSION, which is what actually determines them -- the URL now points at rustup/archive/${RUSTUP_VERSION} and has nothing to do with the Rust release. RUST_VERSION keeps its name; it still selects the toolchain, via --default-toolchain. check_var already fails the build loudly on an unset variable, so a missed site cannot pass silently. Eight in total: two exports and a comment in builder/Containerfile, two check_var calls plus check_sha256sum and fetch_source in build-rust.sh, and a reference in VERSION_UPDATES.md. That document is stale on the version it quotes (1.94.0, against 1.95.0 on main) but that drift is not this commit's to fix. Neither name collides. RUSTUP_* is rustup's own environment namespace (RUSTUP_HOME, RUSTUP_DIST_SERVER, ...) and these are exported in the shell rustup-init runs in, but it reads neither of these names. The comment above the RUN shrinks from five lines to two. Most of it existed to explain that the hash was not the toolchain's, which the names now say. What remains is the part no name can carry: why the rustup version is pinned in the URL at all. Assisted-by: Claude Opus 5 --- builder/Containerfile | 11 ++++------- builder/VERSION_UPDATES.md | 2 +- builder/build_scripts/build-rust.sh | 8 ++++---- 3 files changed, 9 insertions(+), 12 deletions(-) diff --git a/builder/Containerfile b/builder/Containerfile index 57a3536e..d2a30441 100644 --- a/builder/Containerfile +++ b/builder/Containerfile @@ -144,16 +144,13 @@ RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ FROM build_base AS build_rust COPY build_scripts/build-rust.sh /opt/_internal/build_scripts/ -# RUST_HASH is the sha256 of rustup-init (the installer), not of the Rust -# toolchain: RUSTUP_VERSION and RUST_VERSION move independently. The unversioned -# .../rustup/dist path always serves the newest rustup, so RUSTUP_VERSION is -# pinned in the URL; otherwise the hash breaks on every rustup release. Bumping -# RUSTUP_VERSION therefore requires a matching RUST_HASH. +# The unversioned .../rustup/dist path always serves the newest rustup, so the +# version is pinned in the URL; otherwise RUSTUP_HASH breaks on every release. RUN --mount=type=bind,from=static_clang,target=/tmp/cross-compiler,ro \ export RUST_VERSION=1.98.1 && \ export RUSTUP_VERSION=1.29.1 && \ - export RUST_HASH=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 && \ - export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION} && \ + export RUSTUP_HASH=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 && \ + export RUSTUP_DOWNLOAD_URL=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION} && \ /tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/build-rust.sh diff --git a/builder/VERSION_UPDATES.md b/builder/VERSION_UPDATES.md index b17a0972..43029a79 100644 --- a/builder/VERSION_UPDATES.md +++ b/builder/VERSION_UPDATES.md @@ -40,7 +40,7 @@ ```bash # Rust 1.94.0 (rustup-init for x86_64) RUST_VERSION=1.94.0 -RUST_HASH=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 +RUSTUP_HASH=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 # libjpeg-turbo 3.1.3 LIBJPEG_TURBO_VERSION=3.1.3 diff --git a/builder/build_scripts/build-rust.sh b/builder/build_scripts/build-rust.sh index 2c2f813d..c2e966b6 100755 --- a/builder/build_scripts/build-rust.sh +++ b/builder/build_scripts/build-rust.sh @@ -13,8 +13,8 @@ source "${MY_DIR}/build_utils.sh" # Install a more recent Rust check_var "${RUST_VERSION}" -check_var "${RUST_HASH}" -check_var "${RUST_DOWNLOAD_URL}" +check_var "${RUSTUP_HASH}" +check_var "${RUSTUP_DOWNLOAD_URL}" PREFIX=/opt/_internal/rust-${RUST_VERSION} @@ -33,8 +33,8 @@ else exit 1 fi -fetch_source "${RUSTUP_INIT}" "${RUST_DOWNLOAD_URL}/${RUSTUP_ARCH}" -check_sha256sum "${RUSTUP_INIT}" "${RUST_HASH}" +fetch_source "${RUSTUP_INIT}" "${RUSTUP_DOWNLOAD_URL}/${RUSTUP_ARCH}" +check_sha256sum "${RUSTUP_INIT}" "${RUSTUP_HASH}" chmod +x "${RUSTUP_INIT}" # Install Rust using rustup to PREFIX