Skip to content

ci: retry/back off toolchain downloads#577

Merged
kmannislands merged 5 commits into
mainfrom
giles-earthfile-net-resilience
Jun 19, 2026
Merged

ci: retry/back off toolchain downloads#577
kmannislands merged 5 commits into
mainfrom
giles-earthfile-net-resilience

Conversation

@gilescope

Copy link
Copy Markdown

Extracted from #442 (the buildkit upgrade). The dominant CI flake class in #442's investigation was network-fetch: toolchain downloads from throttling/dropping CDNs (dl.google.com, ziglang.org, kind.sigs.k8s.io, antlr.org, golangci-lint) die mid-transfer and the dependent build reports a misleading context canceled.

Adds resume + retry/backoff to those downloads:

  • root Earthfile — golangci-lint install: curl --retry 7 --retry-all-errors to a temp file, then run it.
  • ast/parser/Earthfile — antlr jar: curl -fSLO --retry 7 --retry-all-errors.
  • examples/zig/Earthfile — zig tarball: 5-attempt loop with wget --continue + linear backoff (ziglang throttles, so rapid retries hit the same wall).
  • tests/cloud-push-pull/Earthfile — gcloud SDK: curl -fO --retry 7 --retry-all-errors.
  • tests/with-docker-kind/Earthfile — kind binary: retry loop with wget -c / curl --retry.

Only the download-resilience hunks — no BUILDKIT_PROJECT/image-base/parallelism/group-split changes from the same files (those stay in #442).

…angci-lint)

Signed-off-by: Giles Cope <gilescope@gmail.com>
@gilescope gilescope added the ai-assisted Authored with AI assistance label Jun 15, 2026
@gilescope gilescope mentioned this pull request Jun 15, 2026
10 tasks

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves download reliability across multiple Earthfiles by implementing retries, timeouts, and error handling for curl and wget commands. Feedback on these changes includes removing a redundant echo command in ast/parser/Earthfile, using short options for wget in examples/zig/Earthfile to ensure compatibility with minimal environments, and avoiding the use of wget's continue flag (-c) for downloading the kind binary to prevent potential silent corruption.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread ast/parser/Earthfile Outdated
Comment thread examples/zig/Earthfile Outdated
Comment thread tests/with-docker-kind/Earthfile
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

➖ Are we earthbuild yet?

No change in "earthly" occurrences

📈 Overall Progress

Branch Total Count
main 5328
This PR 5328
Difference +0

Keep up the great work migrating from Earthly to Earthbuild! 🚀

💡 Tips for finding more occurrences

Run locally to see detailed breakdown:

./.github/scripts/count-earthly.sh

Note that the goal is not to reach 0.
There is anticipated to be at least some occurences of earthly in the source code due to backwards compatibility with config files and language constructs.

Comment thread ast/parser/Earthfile Outdated
Comment thread examples/zig/Earthfile Outdated
@gilescope gilescope marked this pull request as ready for review June 15, 2026 05:46
@gilescope gilescope requested a review from a team as a code owner June 15, 2026 05:46
@gilescope gilescope requested review from kmannislands and removed request for a team June 15, 2026 05:46
@kmannislands kmannislands enabled auto-merge (squash) June 18, 2026 16:22

@kmannislands kmannislands left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It occurs to me that a DOWNLOAD command built in to EarthBuild might be more elegant for this sort of thing long-term

@kmannislands kmannislands merged commit 05a5dcf into main Jun 19, 2026
89 checks passed
@kmannislands kmannislands deleted the giles-earthfile-net-resilience branch June 19, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Authored with AI assistance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants