S3 Compability Tests

A small project with bash scripts for testing S3 API compability:

• test_put_if_match.sh - tests If-Match
• test_put_if_not_match.sh - tests If-None-Match
• test_list_order.sh - tests global LIST order across pagination for WAL keys
• test_list_perf.sh - uploads many objects in parallel and measures 3 plain LIST calls speed
• test_put_perf.sh - uploads many small objects in parallel and measures average PUT time per file
• test_put_perf_large.sh - sequentially uploads 5x10 MiB and 5x100 MiB files and measures average upload speed

Works with both AWS S3 and S3-compatible providers (MinIO, Ceph RGW, etc.).

Requirements

• aws CLI v2
• S3 access (access key, secret key, region)

AWS CLI installation:

• Official install guide (all platforms): https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
• Linux: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-linux-install
• macOS: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-mac-install
• Windows: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-windows-install

Setup

1. Create .env from the example:

cp .env.example .env

2. Fill required fields in .env:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

3. ENDPOINT_URL:

• keep empty for AWS S3
• set URL for S3-compatible service, for example http://127.0.0.1:9000

Run

Using make:

make test-if-match
make test-if-not-match
make test-list-order
make test-list-perf
make test-put-perf
make test-put-perf-large
make s3-delete-bucket

Useful .env Variables

See in .env.example.

Key variables for list tests:

• CATALOG_PREFIX
• OBJECTS_COUNT
• OBJECT_KEY_WIDTH
• PUT_PARALLELISM
• LIST_RUNS
• SKIP_PUT_STAGE
• CREATE_BUCKET
• CLEANUP_BUCKET