Skip to content

bug: multer.memoryStorage() buffers entire upload in heap — concurrent large uploads cause OOM crash #12

Description

@brightpixel-dev

Severity

Critical — process crash under load / denial of service

Description

POST /api/ipfs/upload uses multer.memoryStorage(), which holds the entire uploaded file in a single Buffer on the Node.js heap until the upload to Pinata completes. With IPFS_MAX_FILE_MB defaulting to 100 MB, a small number of concurrent uploads will exhaust the process heap and trigger an out-of-memory (OOM) kill.

Code Evidence

```typescript
// src/routes/ipfs.ts:9
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: MAX_FILE_BYTES } });
```

```typescript
// src/services/ipfs.ts:16-20
const res = await axios.post(PINATA_ENDPOINT, form, {
headers: { Authorization: Bearer ${PINATA_JWT}, ...form.getHeaders() },
maxContentLength: Infinity, // ← no upload size cap on the Axios side either
});
```

The upload flow is:

  1. Multer buffers 100 MB into req.file.buffer (heap allocation).
  2. uploadToIPFS appends that buffer to a FormData object (second copy in heap).
  3. Axios serialises the form before sending (potential third copy).

Peak heap cost per upload: ~300 MB. The default Node.js heap limit is ~1.5 GB.

```
Concurrent uploads | Peak heap usage
5 | ~1.5 GB → OOM threshold
10 | ~3.0 GB → guaranteed OOM kill
```

Impact

  • Five concurrent 100 MB uploads from different producers crash the process.
  • No graceful degradation — the process exits and all in-flight requests fail.
  • Pinata uploads that started before the crash leave orphaned pins with no corresponding database record.
  • The global rate limiter does not prevent this because it limits request count, not concurrent body size.

Fix

Replace in-memory buffering with disk-based streaming:

```typescript
import { createWriteStream, unlink } from "fs";
import { tmpdir } from "os";
import { join } from "path";

const upload = multer({
storage: multer.diskStorage({
destination: (_req, _file, cb) => cb(null, tmpdir()),
filename: (_req, file, cb) => cb(null, ${Date.now()}-${file.originalname}),
}),
limits: { fileSize: MAX_FILE_BYTES },
});
```

Stream the file from disk to Pinata using a ReadStream rather than loading the buffer into memory. Delete the temp file in a finally block after the upload completes or errors.

Files Affected

  • src/routes/ipfs.ts
  • src/services/ipfs.ts (uploadToIPFS — signature change from Buffer to file path / stream)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions