A headline guarantee of the SDK is that large transfers stay memory-bounded: you can move multi-GB objects without buffering them whole. Nothing asserts it. A regression that accidentally buffers a whole stream (for example collecting a ReadableStream into one Uint8Array) would pass every functional test and silently break the guarantee.
Proposal: add a Node-only test (the *.node.test.ts convention, since it needs process.memoryUsage()) that pushes a large synthetic stream through the paths that are supposed to stream and asserts peak RSS / heapUsed stays within a small multiple of the part size, not proportional to total size.
Paths to cover (the ones already designed to be bounded):
uploadLargeFile / multipart upload from a streaming ContentSource.
createParallelDownloadStream (ranged parallel download reassembly).
B2Object.createWriteStream (pipeTo-style upload).
Use a generated, non-allocating source (yield reused/zeroed buffers so the source itself does not dominate RSS) and force GC between samples where available. Assert against part-size-relative bounds with enough slack to stay non-flaky on slow CI runners (mirror the LARGE_TEST_TIMEOUT reasoning).
Acceptance criteria:
- A memory-bound regression test over the streaming upload and parallel-download paths, bounded relative to part size.
- Tuned to be non-flaky on the macOS/Windows CI runners.
Related and dependent: #31 and #32 document that the sync path and the missing file-path / Node Readable ContentSource currently force whole-file buffering, so the sync path is explicitly out of scope here until those land. This issue covers the large-file upload and parallel-download paths that already stream.
A headline guarantee of the SDK is that large transfers stay memory-bounded: you can move multi-GB objects without buffering them whole. Nothing asserts it. A regression that accidentally buffers a whole stream (for example collecting a
ReadableStreaminto oneUint8Array) would pass every functional test and silently break the guarantee.Proposal: add a Node-only test (the
*.node.test.tsconvention, since it needsprocess.memoryUsage()) that pushes a large synthetic stream through the paths that are supposed to stream and asserts peak RSS /heapUsedstays within a small multiple of the part size, not proportional to total size.Paths to cover (the ones already designed to be bounded):
uploadLargeFile/ multipart upload from a streamingContentSource.createParallelDownloadStream(ranged parallel download reassembly).B2Object.createWriteStream(pipeTo-style upload).Use a generated, non-allocating source (yield reused/zeroed buffers so the source itself does not dominate RSS) and force GC between samples where available. Assert against part-size-relative bounds with enough slack to stay non-flaky on slow CI runners (mirror the
LARGE_TEST_TIMEOUTreasoning).Acceptance criteria:
Related and dependent: #31 and #32 document that the sync path and the missing file-path / Node
ReadableContentSourcecurrently force whole-file buffering, so the sync path is explicitly out of scope here until those land. This issue covers the large-file upload and parallel-download paths that already stream.