source.coop endpoint is not compatible with the minio client

[cboettig]

Description of Bug:

Note: I think this counts as a bug rather than a feature request because minio client should work out-of-the-box with an endpoint that implements the S3v4 API (or even partially implements it, like CEPH).

Steps to Reproduce:

see official docs

Expected Behavior:

We should be able to initialize the minio client to perform operations against the source.coop API, like any other S3 endpoint.

Actual Behavior:

 mc alias set sc https://data.source.coop
Enter Access Key: XXX
Enter Secret Key: XXX
mc: <ERROR> Unable to initialize new alias from the provided credentials. Query deserialize error: missing field `list-type`.

[jcushman]

There is also an issue with supporting rclone:

> cat $(rclone config path)
[example-auth]
type = s3
provider = Other
access_key_id = …
secret_access_key = …
endpoint = https://data.source.coop
 
[example-anon]
type = s3
provider = Other
endpoint = https://data.source.coop

> rclone -vv --dump headers copy example-auth:org/bucket .
...
2025/02/02 10:40:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/02/02 10:40:16 DEBUG : HTTP REQUEST (req 0xc0003f4500)
2025/02/02 10:40:16 DEBUG : GET /org?delimiter=%2F&max-keys=1000&prefix=bucket%2F HTTP/1.1
Host: data.source.coop
User-Agent: rclone/v1.69.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 286a8726-7c51-4a7e-a3ea-6d5406bb0f46
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250202T154016Z
 
2025/02/02 10:40:16 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/02/02 10:40:16 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/02/02 10:40:16 DEBUG : HTTP RESPONSE (req 0xc0003f4500)
2025/02/02 10:40:16 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 50
Access-Control-Allow-Credentials: true
Content-Type: text/plain; charset=utf-8
Date: Sun, 02 Feb 2025 15:40:16 GMT
Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
X-Version: 0.1.20```

[jcushman]

Update -- I think rclone does work with provider Minio:

[sc]
type = s3
provider = Minio
access_key_id = ...
secret_access_key = ...
endpoint = https://data.source.coop

However, I found the download very slow through the proxy (10MB/s). Going around the proxy seems better:

[sc-direct-anon]
type = s3
provider = AWS
region = us-west-2

$ rclone sync sc-direct-anon:us-west-2.opendata.source.coop/myorg/mycollection/...

[alukach]

@cboettig @jcushman Could either of you try this again? We've done a complete rebuild of the proxy in #116 and am hoping issues like this were resolved.