Skip to content

Migrate S3 backend from deprecated manager APIs to feature/s3/transfermanager #317

@funkyshu

Description

@funkyshu

Is your feature request related to a problem? Please describe.

The S3 backend currently uses deprecated AWS SDK v2 APIs (manager.NewUploader and manager.NewDownloader) for file upload and download operations. These APIs are marked as deprecated in favor of the new feature/s3/transfermanager package. This generates staticcheck warnings (SA1019) across the codebase:

  • backend/s3/file.go:336 - manager.NewUploader is deprecated
  • backend/s3/file.go:340 - uploader.Upload is deprecated
  • backend/s3/file.go:641 - manager.NewDownloader is deprecated
  • backend/s3/file.go:809 - manager.NewUploader is deprecated
  • backend/s3/file.go:817 - uploader.Upload is deprecated
  • backend/s3/file.go:847-848 - manager.Downloader type is deprecated
  • backend/s3/file.go:853-854 - manager.Uploader type is deprecated

While these APIs still function correctly, they may be removed in future AWS SDK releases, and the new transfermanager provides improved performance and better resource management.

Describe the solution you'd like

Migrate the S3 backend to use the new github.com/aws/aws-sdk-go-v2/feature/s3/manager transfermanager APIs:

  1. Replace manager.NewUploader with the new transfermanager upload APIs
  2. Replace manager.NewDownloader with the new transfermanager download APIs
  3. Update the withUploadPartitionSize and withDownloadPartitionSize helper functions to work with the new API
  4. Ensure partition size configuration options continue to work as expected
  5. Verify that all upload/download scenarios (temp file writes, streaming writes, copy operations) work correctly with the new APIs

The migration should maintain backward compatibility with existing Options and configuration while eliminating the deprecation warnings.

Describe alternatives you've considered

  1. Keep using deprecated APIs with nolint directives (current approach) - This works but kicks the can down the road and may break when AWS SDK eventually removes these APIs
  2. Wait for AWS SDK to force the migration - Not recommended as it could break production code unexpectedly
  3. Implement a compatibility layer - Would add unnecessary complexity when direct migration is feasible

Additional context

  • AWS SDK discussion on the deprecation: S3 Transfer Manager v2 Now Generally Available aws/aws-sdk-go-v2#3306
  • The current implementation uses partition size configuration for performance tuning, which must be preserved in the migration
  • All upload/download operations in backend/s3/file.go are affected:
    • writeFileToS3() - temp file uploads
    • copyWithinS3() - S3-to-S3 copies
    • copyToLocalTempReader() - downloads to temp files
    • getS3Writer() - streaming uploads via pipe
  • The migration should be thoroughly tested with integration tests to ensure no regression in functionality or performance
  • Consider this a good opportunity to review and optimize the S3 transfer logic while updating the API usage

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions