ci: retry/back off toolchain downloads#577
Conversation
…angci-lint) Signed-off-by: Giles Cope <gilescope@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request improves download reliability across multiple Earthfiles by implementing retries, timeouts, and error handling for curl and wget commands. Feedback on these changes includes removing a redundant echo command in ast/parser/Earthfile, using short options for wget in examples/zig/Earthfile to ensure compatibility with minimal environments, and avoiding the use of wget's continue flag (-c) for downloading the kind binary to prevent potential silent corruption.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
➖ Are we earthbuild yet?No change in "earthly" occurrences 📈 Overall Progress
Keep up the great work migrating from Earthly to Earthbuild! 🚀 💡 Tips for finding more occurrencesRun locally to see detailed breakdown: ./.github/scripts/count-earthly.shNote that the goal is not to reach 0. |
kmannislands
left a comment
There was a problem hiding this comment.
It occurs to me that a DOWNLOAD command built in to EarthBuild might be more elegant for this sort of thing long-term
Extracted from #442 (the buildkit upgrade). The dominant CI flake class in #442's investigation was network-fetch: toolchain downloads from throttling/dropping CDNs (dl.google.com, ziglang.org, kind.sigs.k8s.io, antlr.org, golangci-lint) die mid-transfer and the dependent build reports a misleading
context canceled.Adds resume + retry/backoff to those downloads:
Earthfile— golangci-lint install:curl --retry 7 --retry-all-errorsto a temp file, then run it.ast/parser/Earthfile— antlr jar:curl -fSLO --retry 7 --retry-all-errors.examples/zig/Earthfile— zig tarball: 5-attempt loop withwget --continue+ linear backoff (ziglang throttles, so rapid retries hit the same wall).tests/cloud-push-pull/Earthfile— gcloud SDK:curl -fO --retry 7 --retry-all-errors.tests/with-docker-kind/Earthfile— kind binary: retry loop withwget -c/curl --retry.Only the download-resilience hunks — no
BUILDKIT_PROJECT/image-base/parallelism/group-split changes from the same files (those stay in #442).