feat(get): stream to an S3 destination (0.21.0)

[johnyaku]

dt get -o s3://bucket/prefix/ streams DVC-tracked data from the source repository's remote (R2 or SSH) straight into object storage, never touching local disk.

This is for handing data to a collaborator working in AWS: they run it on their own instance, and the bytes go remote-to-bucket without either side provisioning scratch for a 358 GiB transfer.

dt get my-registry data/fq/AF013-A -o s3://their-bucket/fastqs/ --dest-profile their-aws
dt get my-registry --csv samples.csv -o s3://their-bucket/fastqs/ --dest-profile their-aws

How

DVC hands us the source filesystem directly, credentials and endpoint already resolved from the clone's .dvc/config:

remote = Repo(clone).cloud.get_remote()
src    = remote.odb.oid_to_path(md5)
remote.fs.open(src, 'rb')

Not a new pattern here — _check_dvc_remote_impl (dt/auth/checks.py:849) already does this. It means no dvc get subprocess per file and no contention on the clone's SQLite state db, the constraint that forces the existing network path to run CSV rows serially.

Two properties fall out of streaming

Verification is free. Bytes pass through this process, so they're hashed in flight and compared against the md5 DVC recorded. A corrupt transfer is caught during the copy and the object deleted — rather than left behind carrying our metadata and passing --check for ever after.

Writes are atomic. s3fs uploads via multipart; the object doesn't exist until commit. An interrupted transfer leaves nothing — the truncated-file failure --check defends against locally cannot occur here.

Credentials

Destination credentials are separate from the source and passed explicitly, never via os.environ. An explicit profile outranks ambient AWS_* in botocore, which is what keeps the two from colliding; mutating the environment would also silently redirect dt auth check, which shells out to aws without --profile.

Preflight resolves the identity, checks the bucket, and probes write permission before any bytes move — printing the account and ARN unconditionally:

Destination: s3://their-bucket/fastqs/AF013-A
Writing as:  account 123456789012 as arn:aws:iam::123456789012:user/handoff

Omitting --dest-profile is legal so an EC2 instance role needs no config; saying which account is the only thing between that and writing to the wrong one.

Guards for the two quiet failure modes

• dt auth setup writes repo-named R2 profiles carrying region = auto into the same ~/.aws/credentials as real AWS profiles, so one is easy to pass by accident. A destination whose region resolves to auto without --dest-endpoint-url is rejected.
• -o s3://... previously collapsed to the relative path s3:/bucket/... and silently created a local directory named s3:. URL destinations reaching the local path are now rejected outright.

Review notes

• decide() now accepts a Path or a destination object, so the resume/check matrix is written once and applies unchanged to S3. All 97 pre-existing test_get.py tests pass untouched.
• The local cache path is deliberately not refactored — resolve_link_types encodes a hard-won EXDEV lesson (15dc42f) and has no fsspec expression.
• ssh:// destinations are deferred. Credentials would be free (ssh config is already the BYO mechanism), but there's no multipart equivalent, so --resume would mean something materially weaker under the same flag name. Reasoning in docs/get-s3.md.

Testing

2069 passed, 1 skipped — 42 new in tests/unit/test_get_s3.py against a fake S3 filesystem, covering addressing, streaming, metadata round-trip, the full decide() matrix on S3, corrupt-source detection, and the credential guards.

One assumption not covered: metadata surviving a real multipart upload is verified at the s3fs source level and by a fake, but never against a live bucket. Worth smoke-testing before trusting --check on a real handoff.

Operational note

Aborted multipart uploads leave orphaned parts that keep accruing storage charges. The receiving bucket wants a lifecycle rule expiring incomplete multipart uploads — documented in docs/get.md, not enforced in code.

🤖 Generated with Claude Code