Skip to content

Multipart upload from AWSSDK.S3 TransferUtility stores aws-chunked framing in object body #29

@TheHACKATHON

Description

@TheHACKATHON

I encountered unexpected behavior with Less3 while using a supported Multipart Upload APIs

Expected behavior
When uploading a file with the latest AWSSDK.S3 for .NET(4.0.18.8) using TransferUtility and TransferUtilityUploadRequest(Multipart Upload APIs under the hood), Less3 should store only the original file bytes.

For multipart uploads, Less3 should correctly process the request payload and persist the final object without any transport-level chunk metadata.

Actual behavior
The upload completes successfully, but the object stored by Less3 contains chunked streaming markers in the actual object body, for example:

14000;chunk-signature=ab8f...

Because of that:

the file stored by Less3 is corrupted
downloading the same object returns the same markers
the downloaded content does not match the original uploaded file

Relevant AWS documentation:

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/TTransferUtility.html

Steps to reproduce
Use the latest AWSSDK.S3 for .NET.
Create an AmazonS3Client targeting a Less3 endpoint.
Upload a file using TransferUtility.UploadAsync(new TransferUtilityUploadRequest { ... }).
Let the SDK perform its normal upload flow.
Inspect the stored object in Less3 internal storage, or download it back through S3 API.
Result
The object body contains chunk framing data instead of only the original file bytes.

Image

Minimal client context

Less3 - latest docker image using default compose config
.NET 10
AWSSDK.S3 4.0.18.8

Our code path uses:

Amazon.S3.Transfer.TransferUtility
TransferUtilityUploadRequest
UploadAsync(...)
200mb file

The code was verified against a different S3 provider and functioned as expected.

    private static IAmazonS3 CreateClient(S3StorageOptions options)
    {
        var config = new AmazonS3Config
        {
            ServiceURL = BuildServiceUrl(options),
            ForcePathStyle = true,
            UseHttp = !options.UseSSL,
            AuthenticationRegion = options.Region,
            Timeout = options.RequestTimeout,
        };

        return new AmazonS3Client(new BasicAWSCredentials(options.AccessKey, options.SecretKey), config);
    }

    private async Task<string> UploadWithTransferUtilityAsync(string objectKey, StagedUploadPayload stagedPayload, CancellationToken ct)
    {
        using var transferUtility = new TransferUtility(_s3Client);
        var request = new TransferUtilityUploadRequest
        {
            BucketName = _options.BucketName,
            Key = objectKey,
            FilePath = stagedPayload.FilePath,
        };

        string versionId = string.Empty;
        request.UploadCompletedEvent += (_, args) =>
        {
            versionId = args.Response?.VersionId ?? string.Empty;
        };

        await transferUtility.UploadAsync(request, ct);
        return versionId;
    }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions