Skip to content

test: assert peak memory stays bounded on the streaming upload/download paths #80

Description

@goanpeca

A headline guarantee of the SDK is that large transfers stay memory-bounded: you can move multi-GB objects without buffering them whole. Nothing asserts it. A regression that accidentally buffers a whole stream (for example collecting a ReadableStream into one Uint8Array) would pass every functional test and silently break the guarantee.

Proposal: add a Node-only test (the *.node.test.ts convention, since it needs process.memoryUsage()) that pushes a large synthetic stream through the paths that are supposed to stream and asserts peak RSS / heapUsed stays within a small multiple of the part size, not proportional to total size.

Paths to cover (the ones already designed to be bounded):

  • uploadLargeFile / multipart upload from a streaming ContentSource.
  • createParallelDownloadStream (ranged parallel download reassembly).
  • B2Object.createWriteStream (pipeTo-style upload).

Use a generated, non-allocating source (yield reused/zeroed buffers so the source itself does not dominate RSS) and force GC between samples where available. Assert against part-size-relative bounds with enough slack to stay non-flaky on slow CI runners (mirror the LARGE_TEST_TIMEOUT reasoning).

Acceptance criteria:

  • A memory-bound regression test over the streaming upload and parallel-download paths, bounded relative to part size.
  • Tuned to be non-flaky on the macOS/Windows CI runners.

Related and dependent: #31 and #32 document that the sync path and the missing file-path / Node Readable ContentSource currently force whole-file buffering, so the sync path is explicitly out of scope here until those land. This issue covers the large-file upload and parallel-download paths that already stream.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions