Idea
Large-output commands (find, sql, and — once aligned per #86 — diff) write their full output to a fixed S3 location: s3://{bootstrap-bucket}/output/{JOB_RUN_ID}. There is currently no way for a user to choose where that output goes.
Aligning diff with the find convention removes its --s3 flag (its only S3-related knob), so after that change none of the large-output commands expose any control over the S3 destination.
Proposal (not doing now)
Consider adding an optional flag to set a custom S3 output location for the large-output commands — e.g. --s3-output s3://my-bucket/my-prefix/. This would make outputs easier to consume in a pipeline (a find writing somewhere a subsequent load reads from — see #184) and gives users control over lifecycle/retention of the output.
Deferring for now; filing so it isn't lost. Relates to #180 (structured output convention) and #184 (find→load round-trip).
Idea
Large-output commands (
find,sql, and — once aligned per #86 —diff) write their full output to a fixed S3 location:s3://{bootstrap-bucket}/output/{JOB_RUN_ID}. There is currently no way for a user to choose where that output goes.Aligning
diffwith the find convention removes its--s3flag (its only S3-related knob), so after that change none of the large-output commands expose any control over the S3 destination.Proposal (not doing now)
Consider adding an optional flag to set a custom S3 output location for the large-output commands — e.g.
--s3-output s3://my-bucket/my-prefix/. This would make outputs easier to consume in a pipeline (afindwriting somewhere a subsequentloadreads from — see #184) and gives users control over lifecycle/retention of the output.Deferring for now; filing so it isn't lost. Relates to #180 (structured output convention) and #184 (find→load round-trip).