A high-performance Python utility designed to benchmark and execute file uploads to Amazon S3. Optimized for stress-testing event-driven architectures, it supports both sequential and parallel execution modes with integrated observability via Amazon CloudWatch.
- Dual Mode: Choose between sequential or high-concurrency parallel uploads using
asyncio. - Smart Sampling: Randomly selects a subset of files from a manifest for targeted stress testing.
- AWS Observability: Direct ingestion of custom metrics and logs to CloudWatch for real-time monitoring.
- Concurrency Control: Built-in semaphore logic to prevent socket exhaustion during massive bursts.
- Python 3.11+
- AWS CLI configured
- Active AWS Account and S3 Bucket
Configure these variables in your .env file or system environment:
| Variable | Description | Required |
|---|---|---|
EXCEL_BASE_PATH |
Path to the .xlsx manifest file |
Yes |
RUNS_BASE_PATH |
Path to the folder where runs are saved locally | Yes |
USE_CLOUDWATCH_LOGS |
Set to True to enable custom metrics |
No |
git clone https://github.com/jonmunm/aws-s3_uploader.git
cd aws-s3_uploader
pip install -r requirements.txtThe tool is invoked via utils.py using the following parameters:
bucket: The target S3 Bucket name.samples: Number of random files to pick from the manifest.run-name: A custom label for the execution.
python utils.py parallel-upload --bucket raw-data-bucket --samples 10 --run-name test-aws-01python utils.py sequential-upload --bucket raw-data-bucket --samples 10 --run-name test-aws-01If USE_CLOUDWATCH_LOGS is enabled, the tool sends real-time metrics to Amazon CloudWatch, including:
- Duration: Precise upload time per file (Latency).
- Throughput: File size and row counts (optimized via fast-count).
- Execution Metadata: Object Keys and upload timestamps.
The tool expects an Excel file with the following columns:
- path: Local path to the file.
- skip: Boolean (True/False) to filter out specific files.