Infra: Replace MinIO with RustFS - #3928
Merged
Merged
Conversation
MinIO Community Edition stopped publishing images in October 2025 and the repository was archived in April 2026, so `minio/minio:latest` resolves to a frozen build that will not receive further security fixes (apache#2664). I'm choosing RustFS since that's what Polaris uses.
rambleraptor
requested review from
Fokko,
geruh and
kevinjqliu
and removed request for
Fokko,
geruh and
kevinjqliu
September 10, 2026 19:57
kevinjqliu
approved these changes
Sep 10, 2026
Contributor
|
nice, will be able to resolve #2664 |
Contributor
|
I linked this PR to #2664, thanks! |
This was referenced Sep 11, 2026
Closed
6 tasks
3 tasks
Sruhvx-jpg
pushed a commit
to Sruhvx-jpg/iceberg-rust
that referenced
this pull request
Sep 24, 2026
* ci: replace MinIO with RustFS for integration tests `quay.io/minio/minio` no longer allows anonymous pulls, so `make docker-up` fails and the integration tests cannot start. DataFusion hit the same failure and switched to RustFS in apache/datafusion#25706. PyIceberg moved to RustFS earlier in apache/iceberg-python#3928. Run `rustfs/rustfs:1.0.0` as the shared S3 service instead. It keeps the `admin`/`password` credentials, the 9000/9001 ports, and virtual-hosted-style access through `RUSTFS_SERVER_DOMAINS` and the bucket aliases. The healthcheck uses `/health/ready`, which waits for storage and IAM. The `mc` bucket setup becomes `curl --aws-sigv4` run from the RustFS image, and the public bucket policy is dropped because no test reads anonymously. Point the REST fixture, HMS, and Spark at `http://rustfs:9000`, and rename the MinIO-specific test helpers and the `ICEBERG_TEST_MINIO_ENDPOINT` override to RustFS. Closes apache#3272 * ci: use vendor-neutral object-store naming for S3 test service Rename the RustFS service, hostname, bucket aliases, test helpers and env var to a generic object-store name (matching PyIceberg), so future backend swaps only need an image change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: fail fast and gate readiness on test bucket creation Treat any non-200/409 response as a failure instead of ignoring it, and add a healthcheck so `docker compose up --wait` blocks until the buckets exist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: remove orphan containers on docker-up Renaming the minio service leaves stale containers holding port 9000 for anyone with a previously started stack; clean them up automatically. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * ci: make create-buckets a one-shot job RustFS returns 200 when re-creating an existing bucket, so plain chained curl -f calls are idempotent. Replace the status-code parsing, marker-file healthcheck and tail with a one-shot container, and gate spark-iceberg on its successful completion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: ovoievodin <o_voievodin@apple.com> Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2664
Rationale for this change
MinIO Community Edition stopped publishing images in October 2025 and the repository was archived in April 2026, so
minio/minio:latestresolves to a frozen build that will not receive further security fixes.I'm choosing RustFS as a replacement since that's what Polaris uses.
After this, I'd like to have dependabot turned on for our Docker images, but we should get off MinIO first.
AI Disclosure: Claude helped me get the Dockerfiles working. I made the docs changes since the best way to ensure that docs are human-first are to have humans write them.
Are these changes tested?
Integration tests should still pass.
Are there any user-facing changes?