s3p (Ruby)

A Ruby port of s3p — list, copy, sync and compare S3 buckets much faster than sequential list_objects_v2 pagination by bisecting the key space and listing many ranges in parallel.

Standard S3 listing is inherently serial: each page's continuation token comes from the previous page, so a 10M-object bucket takes ~10,000 round trips, one at a time. s3p instead treats listing as a divide-and-conquer problem:

1. Compute a key roughly in the middle of the range (pure string math — see S3P::Keys.bisect_key; S3 keys sort lexicographically, so no request is needed to guess a midpoint).
2. List both halves concurrently (start_after = range start; results past the midpoint are trimmed client-side, since S3 has no "stop at" parameter).
3. Any half that returns a full page has more keys: bisect its remaining sub-range and recurse. When a right half comes back empty, the next left bisection cuts at a directory (/) boundary instead, which adapts to deep directory-shaped key spaces.

The fan-out is bounded by a worker pool (default 100 concurrent list requests). Against a mock S3 with 30ms request latency, listing 500k keys runs ~9x faster than sequential pagination (~255k keys/s), and the advantage grows with bucket size.

Install

gem build s3p.gemspec
gem install s3p-*.gem

Requires Ruby >= 3.0. Uses aws-sdk-s3 and the standard AWS credential chain (env vars, ~/.aws/credentials, instance profiles, ...).

CLI

s3p ls --bucket my-bucket                        # all keys, fast, unordered
s3p ls --bucket my-bucket --prefix logs/2024-    # keys under a prefix
s3p ls --bucket my-bucket --pattern '\.parquet$' # regex-filtered
s3p ls --bucket my-bucket --long                 # date, size, key
s3p ls --bucket my-bucket --raw                  # full items as JSON lines
s3p summarize --bucket my-bucket                 # size stats + histogram

s3p cp --bucket src --to-bucket dst              # parallel server-side copy
s3p cp --bucket src --to-bucket dst --prefix in/ --to-prefix out/
s3p sync --bucket src --to-bucket dst            # copy only missing keys
s3p sync --bucket src --to-bucket dst --overwrite  # also replace changed sizes
s3p compare --bucket src --to-bucket dst         # diff without changing anything
s3p list-buckets
s3p help                                         # all options

cp and sync copy server-side (copy_object, or multipart upload_part_copy above 5GB where copy_object hard-fails) — no object data flows through your machine. sync and compare list source and target in one parallel pass: each covered source key range is listed on the target too, so both sides are complete for the same range and can be diffed. sync never deletes; extra target keys are reported as aws s3 rm commands for you to run, same as the original. --dryrun counts copies without copying.

Same-bucket copies (--add-prefix without --to-bucket) list and copy concurrently, so freshly written target keys can be picked up by the still-running listing and copied again (backup/backup/...). Bound the listing away from the target keys with --prefix, --stop-at or --pattern.

Progress is reported once a second on stderr (--quiet to disable). Note that output is unordered — parallel sub-ranges complete in arbitrary order; pipe through sort if you need sorted keys.

Use --endpoint http://localhost:9000 for minio or other S3-compatible stores.

Library

require "s3p"

client = Aws::S3::Client.new

# Stream every item (unordered); callbacks are serialized across threads.
stats = S3P.each(client: client, bucket: "my-bucket", prefix: "logs/") do |item|
  puts "#{item.key} (#{item.size} bytes)"
end
stats.items          # => 1_234_567
stats.requests       # => 2_512
stats.duration       # => 14.2 (seconds)

# Or batches as list requests complete (faster for bulk accumulation):
S3P.each_slice(client: client, bucket: "my-bucket") { |items| ... }

# All keys as an array:
keys = S3P.list(client: client, bucket: "my-bucket")

# Size statistics:
S3P.summarize(client: client, bucket: "my-bucket")
# => { items: ..., total_size: ..., min_size: ..., max_size: ...,
#      average_size: ..., size_histogram: {...}, human: {...} }

# Copy / sync / compare. Key remapping: to_prefix (replaces prefix),
# add_prefix, or to_key (an order-preserving proc).
S3P.copy(client: client, bucket: "src", to_bucket: "dst", prefix: "in/", to_prefix: "out/")
S3P.sync(client: client, bucket: "src", to_bucket: "dst", overwrite: true, delete_log: $stdout)
S3P.compare(client: client, bucket: "src", to_bucket: "dst",
            on_diff: ->(kind, src, dst) { ... }) # kind: :added/:removed/:changed/:same
S3P.list_buckets(client: client)

Other options accepted by all of the above (see S3P::Lister): start_after:, stop_at: (explicit key range), limit: (keys per request), list_concurrency:, max_list_requests:, filter: (proc), progress: (IO).

Development

Tests run entirely against an in-memory mock of list_objects_v2 (no AWS account needed) and include fuzz tests verifying the parallel listing returns exactly the same key set as a sequential walk:

rake test

Differences from the original s3p

• Large-object copies use SDK multipart upload_part_copy instead of shelling out to aws s3 cp, so the aws CLI is not required.
• Local-folder copy (fromFolder/toFolder upload/download) is not ported; all operations are bucket-to-bucket.
• The CLI each/map commands (which eval JavaScript per item) are not ported — use the S3P.each library API with a Ruby block instead.
• Like the original, keys are assumed to use printable-ASCII characters (space through ~), and sync reports deletions rather than performing them.