fix(build): retry network downloads in container builds - #535
Merged
Conversation
One transient TLS failure on a mirror broke the whole arm64 initramfs build and with it CI on main. The LVM2 wget had no retry. Every network fetch now retries with backoff. apt-get uses Acquire::Retries, curl uses --retry, wget uses --tries. apk add, git clone, git fetch and go mod download use an until loop that exits non-zero once the attempts run out. Checksum and signature checks stay in place, so a retry cannot hide a corrupt artifact. A new guard test fails the build when a fetch carries no retry, or when a retry loop cannot fail.
The dev and micro-dev stages build the shipped init binary. Both still used Go 1.26.5, which carries six stdlib findings. The vrnetlab test builder used the same pin. Go 1.26.6 fixes GO-2026-5026, GO-2026-5972, GO-2026-6089, GO-2026-6090, GO-2026-6091 and GO-2026-6218.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run 31880810092 on main failed in
Build Flavors (arm64 default, path-filtered):One unretried
wgetbroke the whole build.Change, applied to every network fetch in the container builds:
wget:--tries=5 --waitretry=10 --retry-connrefused --timeout=30plus retry on 429 and 5xxcurl:--retry 5 --retry-delay 5 --retry-connrefused --connect-timeout 30apt-getupdate, install and download:-o Acquire::Retries=5apk add,git clone,git fetch,go mod download: anuntilloop with 5 attempts and 10 second backoffIntegrity is unchanged. The LVM2 sha256 check, the FRR key fingerprint check, the pinned util-linux commit and apt signature checks all stay, so a retry cannot hide a corrupt artifact.
The retry loops fail closed. Each loop exits non-zero once the attempts run out, so an exhausted retry breaks the build instead of passing silently. Compare kubernetes-sigs/image-builder#2138, where retries existed but never took effect.
TestDockerfileDownloadsRetryenforces this. It flags a fetch without retry and a loop without a fail-closed exit. It found three real gaps while I wrote it (go mod downloadin the three test images), and it fails when I remove the wget flags again.Verified locally with docker:
LVM2.2.03.27.tgz: OKdocker buildx build --checkis clean on all five files, apart from twoWorkdirRelativePathwarnings that already exist on maingo test .passesNot changed, reported instead:
test/e2e/clab/vrnetlab/Dockerfileandinitrd.Dockerfilestill pingolang:1.26.5-alpine, one minor release behind the 1.26.6 toolchain that #532 put in the workflows.#536 stacks on this branch and does that bump, including the
micro-devstage. Merge this PR first, then #536.