Unbreak builder CI: canonical GNU host and an immutable rustup-init URL - #421
Conversation
Reviewer's guide (collapsed on small PRs)Reviewer's GuideThe container build now fetches Autoconf, Automake, and Libtool directly from ftp.gnu.org instead of ftpmirror.gnu.org, eliminating redirector and mirror availability failures without changing the pinned source artifacts or integrity checks. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="builder/Containerfile" line_range="62" />
<code_context>
export AUTOCONF_ROOT=autoconf-2.72 && \
export AUTOCONF_HASH=afb181a76e1ee72832f6581c0eddf8df032b83e2e0239ef79ebedc4467d92d6e && \
- export AUTOCONF_DOWNLOAD_URL=https://ftpmirror.gnu.org/gnu/autoconf && \
+ export AUTOCONF_DOWNLOAD_URL=https://ftp.gnu.org/gnu/autoconf && \
/tmp/cross-compiler/entrypoint /opt/_internal/build_scripts/install-autoconf.sh
</code_context>
<issue_to_address>
**nitpick:** The CI comment in `.github/workflows/test.yml` still says that rebuilding the image downloads autoconf/automake from `ftpmirror.gnu.org`, but these URLs now point to `ftp.gnu.org`, so the documented reason for pulling the published image is false.
**Suggested fix:** Update the workflow comment to refer to the canonical GNU host or remove the obsolete mirror-reliability explanation.
</issue_to_address>Sourcery assessment
Approved.
db540df to
6766b02
Compare
Sourcery withdrew this approval because the latest commits introduced blocking findings.
| export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/dist && \ | ||
| export RUST_VERSION=1.98.1 && \ | ||
| export RUST_HASH=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 && \ | ||
| export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/archive/1.29.1 && \ |
There was a problem hiding this comment.
can you extract the version into a variable (similarly to RUST_VERSION) so that we can change that in the future and not have to change version inside the url?
There was a problem hiding this comment.
Done in 456b1a7 — RUSTUP_VERSION now sits alongside RUST_VERSION and the URL interpolates it:
export RUST_VERSION=1.98.1 && \
export RUSTUP_VERSION=1.29.1 && \
export RUST_HASH=dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 && \
export RUST_DOWNLOAD_URL=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION} && \One caveat worth stating, since it's the exact trap that caused this PR: the two versions aren't independently bumpable. RUST_HASH is the sha256 of rustup-init, the installer, so changing RUSTUP_VERSION also requires a matching RUST_HASH. RUST_VERSION can move on its own — rustup verifies the toolchain against its own signed channel manifest. That's in the comment above the RUN now.
4842c8b to
59b060d
Compare
smatula
changed the title
59b060d to
a30a1fe
Compare
Every plumbing-builder PR pipeline has failed since 2026-09-01, in two
different places. Only the second one is ours to fix.
The first, GNU autoconf/automake/libtool at step 5 of 29, was a fault in
ftpmirror.gnu.org itself and has resolved on its own:
curl -fsSL --retry 10 -o autoconf-2.72.tar.gz \
https://ftpmirror.gnu.org/gnu/autoconf/autoconf-2.72.tar.gz
curl: (22) The requested URL returned error: 404
The redirector had started appending the request path to each mirror's GNU
directory rather than to its parent, producing a doubled segment
(.../mirrors/gnu//gnu/autoconf/autoconf-2.72.tar.gz). Our URLs were correct
throughout; --retry could not help, since curl's transient-error set excludes
404 and the mirror choice is a pure function of the request path, so every
attempt landed on the same broken mirror.
We were not alone: chimera-linux/cports#6169, fedora-infra/anitya#2073 (which
quotes the identical doubled path on the identical mirror we were assigned),
LibreELEC/LibreELEC.tv#11754 and #11759, and easybuilders/easybuild-framework
#5283 all report it over the same window. GNU published no announcement, but
the last third-party 404 is timestamped 2026-09-14T13:13Z and the path serves
correctly again now -- reconfirmed over six probes across all three tools,
6/6 HTTP 206 on /gnu/<tool>/ and 6/6 404 on the bare form. So the
Containerfile's GNU URLs are left exactly as they are; the mirrors are the
availability risk here, not the paths, and every tarball is
check_sha256sum-verified after download regardless.
That leaves the real bug, rustup-init (build-rust.sh, step 12 of 29):
rustup-init: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
RUST_DOWNLOAD_URL pointed at https://static.rust-lang.org/rustup/dist, and the
fetched filename is a bare "rustup-init" -- no version anywhere in the path.
That endpoint always serves the newest rustup, so RUST_HASH pinned the bytes of
a moving target and broke as soon as rustup cut a release. The pinned hash
4acc9acc is rustup 1.29.0 exactly; dist/ now serves 1.29.1 (dda72343). Nothing
was tampered with: the checksum correctly refused a file that had legitimately
changed.
Fetch from the immutable .../rustup/archive/${RUSTUP_VERSION}/ path so
RUST_HASH pins something that cannot move, with the rustup version held in its
own variable alongside RUST_VERSION rather than buried in the URL, and record
in a comment that RUST_HASH covers the installer rather than the toolchain.
That distinction is easy to miss: RUST_VERSION is passed to --default-toolchain
and selects the compiler, while the rustup version only decides which installer
runs. Cargo is not pinned separately; it ships with the toolchain and follows
RUST_VERSION.
Also bump RUST_VERSION 1.95.0 -> 1.98.1, carried over from calungaproject#418, which was
closed so it could be folded in here.
Every other fetch_source in builder/build_scripts/ interpolates a *_ROOT
variable that carries the version in the filename, so rustup-init was the only
download exposed to this class of failure.
Verified on the Konflux PR build: all 29 stages ran clean, zero curl errors
across 18k log lines, and "rustc 1.98.1 (48a229cea 2026-09-01)" installed.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
a30a1fe to
456b1a7
Compare
smatula
changed the title
|
Heads-up that this PR has been rescoped since the last review — it's now one commit ( What changed: the GNU autoconf/automake/libtool URLs are reverted to What's left is the one genuine bug: On the red checks — all three are the ongoing quay.io outage, not this change. The Konflux run couldn't even resolve its own Tekton task bundles ( |
| # toolchain: RUSTUP_VERSION and RUST_VERSION move independently. The unversioned | ||
| # .../rustup/dist path always serves the newest rustup, so RUSTUP_VERSION is | ||
| # pinned in the URL; otherwise the hash breaks on every rustup release. Bumping | ||
| # RUSTUP_VERSION therefore requires a matching RUST_HASH. |
There was a problem hiding this comment.
thinking about it more, we wouldn't need this long comment if we had better names for the variables. RUST_HASH makes it seem like it's related to the RUST_VERSION when in reality it's connected to the RUSTUP_VERSION so it would make sense to rename it to RUSTUP_HASH instead, WDYT?
There was a problem hiding this comment.
Agreed — done in b4b34d6. RUST_HASH → RUSTUP_HASH, and I took RUST_DOWNLOAD_URL → RUSTUP_DOWNLOAD_URL with it: it points at static.rust-lang.org/rustup/archive/${RUSTUP_VERSION} and fetches rustup-init, so it read wrong for exactly the same reason. Eight sites across the Containerfile, build-rust.sh and VERSION_UPDATES.md; check_var fails the build loudly if either was missed.
You were right that the comment was carrying the names' weight. It's down from five lines to two:
# The unversioned .../rustup/dist path always serves the newest rustup, so the
# version is pinned in the URL; otherwise RUSTUP_HASH breaks on every release.Most of it existed to say the hash wasn't the toolchain's, which the name now says. What's left is the part no name can carry — why the version is pinned in the URL at all.
One thing I checked while renaming: RUSTUP_* is rustup's own environment namespace (RUSTUP_HOME, RUSTUP_DIST_SERVER, RUSTUP_UPDATE_ROOT), and these are exported in the same shell rustup-init runs in. Neither of the two new names is one rustup reads, so it's fine as is — just worth a glance before adding more RUSTUP_* variables here.
|
/retest |
1 similar comment
|
/retest |
…ctor
The builder pipeline cannot get through step 5 of 29. autoconf, automake and
libtool are fetched from ftpmirror.gnu.org, which 302s to a volunteer mirror
chosen per request, and enough of that pool is unusable that a build needing
three consecutive successes rarely gets them. Of twelve failed runs whose logs
were examined, five died on a 404, two on a 502, two on a 504 and three on a
refused connection; both runs today failed here as well.
The paths are correct -- this is not the redirector regression that broke the
same step earlier this month, which GNU has since fixed. The failures are the
mirrors themselves, and they vary per request:
autoconf curl: (7) Failed to connect to mirror2.evolution-host.com:443:
Connection timed out
automake curl: (22) The requested URL returned error: 404 Not Found
Two distinct faults on two consecutive runs, and in the second run autoconf
succeeded before automake failed -- the redirector picks independently per
path, so each download is its own coin flip. The 404s are mirrors that hold
the archive at their document root rather than under /gnu/; probing 20 paths
across the pool, 4 answer the /gnu/ layout with 404 no matter how healthy they
are. --retry cannot route around either fault: curl's transient set excludes
404, and a retry re-requests the same path and so lands on the same mirror.
Point the three URLs at ftp.gnu.org, which serves the archive directly. Probed
four times per tarball, 12/12 HTTP 206, and a full fetch of autoconf-2.72
matches the pinned sha256. Losing the mirror pool costs redundancy, but a pool
this unreliable was supplying variance, not availability, and every tarball is
check_sha256sum-verified after download regardless -- the host is an
availability concern, not a trust boundary.
The CI workflow comment in .github/workflows/test.yml named ftpmirror.gnu.org
as the reason for pulling the published image rather than rebuilding. Drop the
host-specific claim; the remaining reasons, that Konflux builds the real image
and denylist coverage is the job above, stand on their own.
A general fix belongs in fetch_source -- reroll onto a different mirror and
fall back to the canonical host -- but that changes the one code path every
download in the builder shares, so it wants its own review rather than riding
along with an outage fix.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
smatula
changed the title
Both describe rustup-init, the installer, but sat beside RUST_VERSION and read as though they belonged to it. That misreading is not hypothetical: calungaproject#418 bumped RUST_VERSION and replaced the hash together, described as "update the Rust release checksum", when the new value was in fact the rustup 1.29.1 installer's. It went green for the wrong reason. RUSTUP_HASH and RUSTUP_DOWNLOAD_URL pair both with RUSTUP_VERSION, which is what actually determines them -- the URL now points at rustup/archive/${RUSTUP_VERSION} and has nothing to do with the Rust release. RUST_VERSION keeps its name; it still selects the toolchain, via --default-toolchain. check_var already fails the build loudly on an unset variable, so a missed site cannot pass silently. Eight in total: two exports and a comment in builder/Containerfile, two check_var calls plus check_sha256sum and fetch_source in build-rust.sh, and a reference in VERSION_UPDATES.md. That document is stale on the version it quotes (1.94.0, against 1.95.0 on main) but that drift is not this commit's to fix. Neither name collides. RUSTUP_* is rustup's own environment namespace (RUSTUP_HOME, RUSTUP_DIST_SERVER, ...) and these are exported in the shell rustup-init runs in, but it reads neither of these names. The comment above the RUN shrinks from five lines to two. Most of it existed to explain that the hash was not the toolchain's, which the names now say. What remains is the part no name can carry: why the rustup version is pinned in the URL at all. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
3563b57 to
b4b34d6
Compare
|
Correcting my note yesterday: I said the GNU URLs were going back to Two runs this morning, both dead at step 5, differently each time: The second run got autoconf through before automake failed — the redirector picks a mirror per request path, so each of the three downloads is its own coin flip and we need all three. The 404s aren't our paths: they come from mirrors that keep the archive at their document root rather than under So It worked — the container build got through all 29 stages clean, zero curl errors. First one since 01 Sep. (That run was cancelled partway by the push carrying the rename; the full pipeline, Clair and EC included, is running now on This still deserves a proper fix in |
2 participants
The plumbing-builder pipeline has not produced an image since 2026-09-01. Two
independent download faults were stacked at steps 5 and 12 of 29; this fixes both,
plus a naming cleanup that falls out of the second.
1. GNU autotools — the mirror pool, not our paths (
6e15e15)autoconf, automake and libtool were fetched from
ftpmirror.gnu.org, which 302s to avolunteer mirror chosen per request. Enough of that pool is unusable that a build
needing three consecutive successes rarely gets them. Of twelve failed runs whose logs
were examined: five 404s, two 502s, two 504s, three refused connections. Both runs
today failed here too, differently each time:
In the second run autoconf succeeded before automake failed — the redirector picks
independently per path, so every download is its own coin flip. The 404s come from
mirrors that hold the archive at their document root rather than under
/gnu/; across20 probed paths, 4 answer the
/gnu/layout with 404 however healthy they are.--retryescapes neither fault: curl's transient set excludes 404, and a retryre-requests the same path and lands on the same mirror.
Note this is not the ftpmirror redirector regression that broke this same step
earlier in the month — our paths were correct throughout that, other projects hit it
too (cports#6169,
anitya#2073, LibreELEC #11754 /
#11759, easybuild#5283),
and GNU has since fixed it.
Fix: fetch from
ftp.gnu.orgdirectly. Probed four times per tarball, 12/12 HTTP 206,and a full fetch of autoconf-2.72 matches the pinned sha256. Losing the pool costs
redundancy, but a pool this unreliable supplied variance rather than availability, and
every tarball is
check_sha256sum-verified after download regardless — the host is anavailability concern, not a trust boundary.
2. rustup-init — a hash pinning a moving target (
456b1a7)RUST_DOWNLOAD_URLpointed athttps://static.rust-lang.org/rustup/distand thefetched filename is a bare
rustup-init— no version anywhere in the path. Thatendpoint always serves the newest rustup, so the pinned hash described a file that
changes without notice, and broke the moment rustup cut a release. The pinned
4acc9accis rustup 1.29.0 exactly;dist/now serves 1.29.1 (dda72343). Nothingwas tampered with — the checksum correctly refused a file that had legitimately
changed.
Fetch from the immutable
.../rustup/archive/${RUSTUP_VERSION}/path instead, with therustup version in its own variable rather than buried in the URL. Also bumps
RUST_VERSION1.95.0 → 1.98.1, carried over from #418, which was closed so it could befolded in here.
3.
RUST_HASH→RUSTUP_HASH(3563b57)The hash covers rustup-init, the installer, but sat beside
RUST_VERSIONand read asthough it belonged to it. Not a hypothetical misreading: #418 changed the two together
as "update the Rust release checksum" when the new value was the rustup installer's. The
comment above the
RUNshrinks from five lines to two — most of it existed to say whatthe name now says. Five call sites;
check_varfails loudly if any were missed.Every other
fetch_sourceinbuilder/build_scripts/interpolates a*_ROOTvariablecarrying the version in the filename, so rustup-init was the only download exposed to
fault 2.
A general fix for fault 1 belongs in
fetch_source— reroll onto a different mirror,fall back to the canonical host — but that touches the one code path every download in
the builder shares, so it wants its own review rather than riding along with an outage
fix.