ci: unbreak docker-up (backport #3273 RustFS swap and #3174 HMS jar ADD)

[krinart]

Fixes "Tests (default)" on #50. That job fails in make docker-up before any test runs:

mc Error pull access denied for minio/mc, repository does not exist or may require 'docker login'
make: *** [Makefile:110: docker-up] Error 1

(job 110117449151). The images are gone from both registries, so this is an environment break that affects every branch, not something the DataFusion 55 changes caused:

• docker manifest inspect minio/mc:RELEASE.2025-05-21T01-59-54Z / minio/minio:RELEASE.2025-05-24T17-08-30Z -> denied: requested access to the resource is denied
• docker manifest inspect quay.io/minio/{minio,mc}:<same tags> (upstream's first workaround, ci: pull MinIO images from Quay apache/iceberg-rust#3202) -> no such manifest
• docker manifest inspect rustfs/rustfs:1.0.0 -> OCI index returned

This backports upstream apache#3273 (119fad9), which swaps MinIO for RustFS and renames the test helpers and ICEBERG_TEST_MINIO_ENDPOINT to the object-store names. The only conflict, in dev/docker-compose.yaml, was resolved to the RustFS side, keeping this line's REST fixture CATALOG_URI. git grep -i minio finds nothing left outside docs and the opendal minio:// scheme support.

Once the RustFS swap was in place, the first CI run on this PR (job 110224388762) got past the MinIO pulls and failed on the next environmental break in make docker-up: the HMS image build runs apt-get install curl on the Debian 11 base of apache/hive:3.1.3, and Debian's mirrors no longer have those package versions:

E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/c/curl/curl_7.74.0-1.3%2bdeb11u16_amd64.deb  404  Not Found
target hive-metastore: failed to solve

The second commit backports upstream apache#3174 (4687d26). It fetches the two jars with checksum-pinned ADD instead of installing curl.

Neither commit was run locally (the build host has no Docker daemon). This PR's own "Tests (default)" run is the check.

[Copilot]

Copilot review overview

🟢 Approved

The backport consistently updates the integration-test stack and matches the merged upstream implementation.

Review effort: Balanced
Findings: None

What changed in this PR

Backports upstream apache#3273 to restore integration tests by replacing unavailable MinIO images with RustFS.

Changes:

• Adds RustFS and reliable bucket provisioning.
• Renames MinIO-specific endpoints and helpers.
• Updates dependent services, tests, and documentation.

File	Description
Makefile	Removes orphaned containers during startup.
dev/​docker-compose.yaml	Replaces MinIO with RustFS and provisions buckets.
dev/​hms/​core-site.xml	Updates the HMS object-store endpoint.
dev/​spark/​spark-defaults.conf	Updates the Spark object-store endpoint.
crates/​test_utils/​src/​lib.rs	Renames object-store endpoint utilities.
crates/​integration_tests/​src/​lib.rs	Uses the renamed endpoint helper.
crates/​catalog/​hms/​tests/​hms_catalog_test.rs	Updates HMS test configuration.
crates/​catalog/​loader/​tests/​common/​mod.rs	Updates shared catalog test configuration.
crates/​storage/​opendal/​tests/​file_io_s3_test.rs	Updates S3 test helpers and naming.
crates/​storage/​opendal/​tests/​resolving_storage_test.rs	Updates resolving-storage test configuration.
website/​src/​reference/​container-runtimes.md	Documents RustFS as the test object store.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The documented direct Compose commands do not remove renamed MinIO orphan containers, potentially leaving port 9000 occupied.

Review effort: Balanced
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Add orphan removal to direct start/stop commands

dev/​docker-compose.yaml:31

The service rename makes the direct start/stop commands at lines 20–21 stale. Unlike the updated Make targets, they do not remove the old minio/mc orphan containers; an existing minio container can retain port 9000 and prevent object-store from starting. Add --remove-orphans to both documented commands so the advertised workflow handles this migration too.