0.11.x backport: ci: fix Rust cache action casing (#3249) + replace MinIO with RustFS (#3273) - #3271
Open
xanderbailey wants to merge 2 commits into
Open
xanderbailey wants to merge 2 commits into
xanderbailey wants to merge 2 commits into
Conversation
(cherry picked from commit 832a4eb)
This was referenced Sep 24, 2026
Open
xanderbailey
changed the title
* ci: replace MinIO with RustFS for integration tests `quay.io/minio/minio` no longer allows anonymous pulls, so `make docker-up` fails and the integration tests cannot start. DataFusion hit the same failure and switched to RustFS in apache/datafusion#25706. PyIceberg moved to RustFS earlier in apache/iceberg-python#3928. Run `rustfs/rustfs:1.0.0` as the shared S3 service instead. It keeps the `admin`/`password` credentials, the 9000/9001 ports, and virtual-hosted-style access through `RUSTFS_SERVER_DOMAINS` and the bucket aliases. The healthcheck uses `/health/ready`, which waits for storage and IAM. The `mc` bucket setup becomes `curl --aws-sigv4` run from the RustFS image, and the public bucket policy is dropped because no test reads anonymously. Point the REST fixture, HMS, and Spark at `http://rustfs:9000`, and rename the MinIO-specific test helpers and the `ICEBERG_TEST_MINIO_ENDPOINT` override to RustFS. Closes apache#3272 * ci: use vendor-neutral object-store naming for S3 test service Rename the RustFS service, hostname, bucket aliases, test helpers and env var to a generic object-store name (matching PyIceberg), so future backend swaps only need an image change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: fail fast and gate readiness on test bucket creation Treat any non-200/409 response as a failure instead of ignoring it, and add a healthcheck so `docker compose up --wait` blocks until the buckets exist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: remove orphan containers on docker-up Renaming the minio service leaves stale containers holding port 9000 for anyone with a previously started stack; clean them up automatically. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: make create-buckets a one-shot job RustFS returns 200 when re-creating an existing bucket, so plain chained curl -f calls are idempotent. Replace the status-code parsing, marker-file healthcheck and tail with a one-shot container, and gate spark-iceberg on its successful completion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: ovoievodin <o_voievodin@apple.com> Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 119fad9)
xanderbailey
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only backport to
0.11.xof two commits that are mutually blocking on this branch:832a4eba7a732baebbbd7f76bb6fd6225f1db1e9Swatinem/rust-cachecasing for the ASF actions allowlist119fad92c891c95a7819d87e4195407b3f7c0578Why they are combined
Neither backport can pass CI on its own, so they have to land together:
Tests (default)andTests (doc)fail atmake docker-upwithminio Error unauthorized: access to the requested resource is not authorized— the MinIO image is no longer pullable.asf-allowlist-checkfails withswatinem/rust-cache@6323deb... — NOT ON ALLOWLISTacross 6 workflow refs. (See 0.11.x backport: fix(encryption) [18/N] wire in ags1 file length for tamper proofing #3236 #3270, which hits exactly this on unmodified0.11.x.)