Listing bucket contents failing with s3cmd and minio-client

[kastl-ars]

Dear @deepjoy,

I found shoebox and found it to be really easy to get going. Nicely done!

I could successfully use this with aws s3 and the s5cmd (which is using aws-sdk under the hood, piggybacking on its credential and configuration files).

s3cmd

I can upload files to shoebox using s3cmd, but I cannot list bucket contents. I just get a 404 when trying to list the bucket contents:

s3cmd_command="s3cmd \
    --access_key=tux-penguin \
    --secret_key=globalSecretKey1234567890123456789012345 \
    --host=192.168.0.13:8080 \
    --host-bucket=192.168.0.13:8080 \
    --no-ssl"

${s3cmd_command} ls s3://test-bucket/

Uploading files with ${s3cmd_command} put my-file s3://test-bucket/ works.

mc

With minio-client and a configuration file ~/.minio-client/config.json like the following I can list shoebox and get a list of buckets, but I cannot list buckets or upload/download files.

{
        "version": "10",
        "aliases": {
                "shoebox": {
                        "url": "http://192.168.121.197:8080",
                        "accessKey": "tux-penguin",
                        "secretKey": "globalSecretKey1234567890123456789012345",
                        "api": "S3v4",
                        "path": "auto"
                }
        }
}

Working command:

$ mc ls shoebox
[1970-01-01 01:00:00 CET]     0B test-bucket/
$

Failing commands:

$ mc ls shoebox/test-bucket
minio-client: <ERROR> Unable to list folder. Bucket `test-bucket` does not exist.
$

$ mc cp my-file shoebox/test-bucket
minio-client: <ERROR> Unable to list folder. Bucket `test-bucket` does not exist.
$

Adding a trailing slash to the bucket name does not change the behaviour.

Any ideas, tipps or tricks?

BTW, maybe it would be nice to add a section on alternative s3 commands and which are known to be working, configuration hints or similar?

Kind Regards,
Johannes

[deepjoy]

Thanks for the detailed report, really appreciate the config examples and error details.

This is a confirmed bug. The root cause is a routing issue with trailing slashes. When a client sends GET /bucket/ (with a trailing /), the server matches it as an object request with an empty key instead of routing it to the bucket listing handler. That's why you get a 404.

s3cmd and minio-client both append trailing slashes to bucket listing requests. s5cmd doesn't, which is why it works.

Workaround: Use s5cmd for listing operations until this is fixed. Uploads via s3cmd will continue to work fine.

s5cmd --endpoint-url http://192.168.0.13:8080 --credentials-file <(echo '[default]
aws_access_key_id = tux-penguin
aws_secret_access_key = globalSecretKey1234567890123456789012345') ls s3://test-bucket/

I'm in the middle of a larger refactor right now, but this fix (trailing slash normalization in middleware) is straightforward. Expect it in the next release, roughly a week out.

Good call on the compatible clients docs. I'll add that to the README.