ci: cache every compile job and key caches on their full inputs - #20
Merged
Merged
Conversation
PUBLIC_API_CRATES was computed with $(shell cargo metadata ...) at parse time, so every make invocation paid a cargo metadata call. In CI this ran before the toolchain setup step, making the rustup shim download the whole nightly toolchain inside the docker startup step (~9s). Evaluate the crate list inside the two public-api recipes that use it instead. Also detach the docker-up-background subshell's stdio so the caller isn't left holding open pipes until the containers finish starting (~5s of CI runner drain time).
Add rust-cache to the clippy and msrv jobs (the last uncached compile jobs) and to the bindings test job on Linux, which builds on the host like the other platforms since no manylinux input is passed. Key every cache on the runner image and a hash of the files defining the build (workflow file, Makefile, root Cargo.toml): rust-cache skips saving on exact primary-key hits, so a blob saved by outdated steps or a different runner image could otherwise never heal. The root manifest is virtual and thus outside rust-cache's own manifest hashing. Build the bindings test wheel with a new fast ci profile instead of the size-optimized release profile that publishing uses, and lock the ci.yml build invocations. Exclude pyiceberg_core_rust from the Windows build: it is a fully serialized tail there and the bindings workflow compiles the same crate on windows-latest for every code PR. Drop the cargo-public-api install-action step: the tool has no prebuilt binaries anywhere, so the step source-compiled a version that differs from the Makefile pin, overwriting the cached binary on every run and forcing the make target to recompile its own.
abnobdoss
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.