[Improvement] Replace MinIO with a maintained S3 integration-test backend

[nevzheng]

What would you like to be improved?

Replace MinIO in Gravitino’s S3 integration tests with a maintained backend while preserving storage and credential-vending coverage. Recommend RustFS primarily for alignment with PyIceberg and Polaris. SeaweedFS remains a viable alternative if compatibility, maintenance, or CI reliability changes the tradeoff.

MinIO’s Docker Hub outage broke test startup (#13111). #13112 restored pulls through Quay, but the community repository is archived; changing registries is a temporary fix.

How should we improve?

Migrate the shared fixture and its fileset/Iceberg consumers to a pinned RustFS image in #13155. Add lightweight image maintenance separately in #13156: supported Dependabot updates plus monthly/manual advisory package scans for release and test images, with a post-publication hook. Track actionable findings in #13157. Keep backend setup in the fixture and shared S3 configuration names neutral so a later replacement remains manageable.

Scope is test infrastructure; production storage migration and public API changes are excluded. The completed source comparison below supports the recommendation. Runtime compatibility remains an adoption requirement.

Acceptance criteria

• One authoritative image version/digest is consumed by both the fixture and update tooling. The pinned image starts reliably in Linux amd64 CI; verify arm64 availability and readiness, bucket/identity setup, logs, and cleanup without MinIO tooling.
• AWS SDK v2 signed AssumeRole accepts Gravitino’s generated inline policy. Direct checks confirm allowed reads/writes and authorization denial outside object-path and bucket-list-prefix scope; a generic client exception does not establish enforcement.
• Existing fileset and Iceberg suites pass without weakening owner/modify access, select-only write denial, or active-role narrowing assertions. Preserve the generated policy’s deliberate Hadoop directory-probing behavior.
• Relevant listing, deletion, and multipart operations work. The migration PR records commands, results, the image pin, and remaining compatibility limits.
• Enable weekly Dependabot version updates for supported Docker/Compose manifests. Verify the hosted update check after merge, document prerelease/RC-to-stable detection limits, and manually review unsupported transitions. Updates run affected integration tests before merge.
• Include the RustFS pin alongside maintained release/test targets in monthly and manual package scans; scan changed fixture pins manually before adoption and images published through the hooked workflow after publication. Retain resolved digests and reports; keep findings/failures warnings-only. Document manual list maintenance, metadata limits, urgent-advisory checks, and risk-based triage in [Improvement] Maintain release and test images with risk-based advisory follow-up #13157. No exhaustive discovery or bespoke binary analysis is required.

Decision requested: Do we agree to adopt RustFS for ecosystem alignment, subject to these compatibility and maintenance criteria?

---

Supporting evidence

Appendix A — Apache context

PyIceberg #3928 merged September 11, 2026, replacing MinIO with RustFS rc.5; Polaris #3679 added RustFS test containers. These are the primary adoption references, although PyIceberg’s static-credential fixture does not establish Gravitino STS compatibility.

Iceberg Java also moved MinIO pulls to Quay. Its earlier RustFS and SeaweedFS proposals closed unmerged; there is no established Apache-wide backend choice.

Appendix B — Gravitino’s compatibility contract

Sources below were checked against upstream commit 20574ccdc7d5ea2c2c77c309c687e61d2fed4c7c.

• S3TokenGenerator sends signed AssumeRole with role ARN, session name, duration, inline policy, and optional external ID. Policies grant GetObject/GetObjectVersion on readable paths, PutObject/DeleteObject only on writable paths, GetBucketLocation, and ListBucket conditioned by StringLike/s3:prefix.
• FilesetS3TokenConnectionIT checks Hadoop S3A connection using temporary credentials and path-style access. Its policy includes the bare location prefix for directory probing. This suite does not test authorization denial.
• IcebergRESTCloudTokenAuthorizationBaseIT exercises Spark/Iceberg reads, writes, select-only denial, and active-role narrowing. Additional direct checks must establish resource/prefix restrictions beyond these assertions.

S3 and STS endpoints are configurable. A replacement may provision a real role instead of retaining MinIO’s dummy ARN.

Appendix C — RustFS vs. SeaweedFS

Research checked September 14, 2026. This is evidence from released source, upstream tests, and documentation. Neither candidate was run against Gravitino during this research. Registry manifests/pulls and comparative startup/resource measurements were not verified.

Dimension	RustFS	SeaweedFS
Release	1.0.0-rc.6, September 11; prerelease.	4.47, September 14; not a prerelease.
Distribution/license	Linux amd64/arm64 assets and container workflow; Apache-2.0.	Linux amd64/arm64 assets and container workflow; Apache-2.0.
Fixture setup	Single object-store container with credentials, health check, and S3 bucket setup; PyIceberg provides a migration reference.	One process can run storage and S3; signed STS additionally needs IAM signing configuration, a named role, trust policy, and base permissions. Configuration.
Inline session boundary	Signed handler issues temporary credentials; IAM authorization intersects parent and session permissions, including root-issued sessions.	Signed handler accepts inline Policy; IAM authorization requires both base and session permission, defaulting to deny.
Relevant restriction tests	Released prefix-policy and STS bucket-filtering tests.	Released read-only/session-boundary and matching/unrelated/omitted-prefix tests.
Role/trust limits	Standard handler derives permissions from caller and inline policy; parsed role ARN/external ID do not implement AWS named-role selection.	Named roles/trust are evaluated. External-ID enforcement is not established: the handler does not read it and trust context supplies principal fields only.
S3 limits	Compatibility matrix covers common operations; some multipart edge cases are outside the default gate.	Policy engine expands PutObject to multipart operations, including abort/list operations; this is not exact AWS action equivalence.

RustFS’s closer fixture model is a secondary benefit to ecosystem alignment. SeaweedFS offers an established release series and direct tests for several relevant permission boundaries. Neither establishes complete AWS IAM/STS emulation; current MinIO tests do not require external-ID enforcement.

Historical bugs are not current disqualifiers: RustFS’s single-string Resource parsing issue was fixed in alpha.79; SeaweedFS merged fixes for STS POST dispatch, multipart authorization, and prefix conditions.

Appendix D — Image maintenance

The proposed maintenance PR #13156 keeps version updates and CVE reporting separate. Dependabot checks supported Docker/chart directories and Compose manifests weekly; Java/Gradle literals, computed values, and the scan list still need manual maintenance. The fixture consumes its canonical Compose pin; the explicit scan list must be updated alongside it. Do not assume prerelease transitions are detected automatically.

Use ordinary Syft/Grype package scans of an explicit release/test image list monthly, manually for changed pins or urgent advisories, and after publication through the hooked workflow. Record the resolved image digest and scan limits. No PR scan gate, automatic issue filing, or submission to Dependabot alerts is proposed. Scheduling starts after default-branch placement; verify the first hosted run.

RustFS rc.6's OS-only inventory does not establish embedded Rust dependency coverage. Record that limitation and consult upstream advisories; bespoke binary analysis is outside this work. This follows a risk-based maintenance approach, not a requirement to eliminate every CVE.

Follow-up #13157 holds the refreshed local baseline, prioritized actions, and detailed report. RustFS remains the recommendation for ecosystem alignment, not because unequal scanner counts establish it is safer.

[nevzheng]

@jerryshao @roryqi @yuqi PTAL and please follow #13154.

The Quay change restored our MinIO image pulls, but we still need a maintained test backend. I recommend RustFS mainly for alignment with PyIceberg and Polaris; the issue includes the SeaweedFS comparison and Gravitino’s acceptance criteria.

Two PRs are underway: the RustFS migration, followed by image-update and vulnerability monitoring. Feedback on the direction and acceptance criteria would be helpful. I’ll add the PR links here as they open.

[nevzheng]

Both implementation PRs are now open for #13154:

• RustFS migration: [#13154] test: Replace MinIO credential-vending fixture with RustFS #13155
• Repository-wide container image advisories (draft): [#13154] ci: Add advisory scans for release and test images #13156

Monitoring covers container images generally, not just RustFS. Findings and scan failures are warnings only, with no merge gate; automatic ticket filing remains disabled. All 25 helper tests passed, and a representative image scan verified warning-only behavior. The complete 41-image hosted sweep remains unrun.

PTAL and follow the PRs; feedback on image coverage and the advisory-only policy is welcome.

[nevzheng]

Follow-up: #13157 tracks the initial container-scan report, prioritized remediation actions, and discovery/coverage gaps. Monitoring implementation is in #13156; the RustFS migration remains in #13155.

[roryqi]

@bharos Could u help review the related pull requests?