ENOSPC after setup on runners where /mnt shares the root disk (nothing-but-nix holster)

[mdarocha]

Summary

On runners where /mnt is not a separate block device (it's just a directory on the root filesystem), the default holster disk reclamation leaves almost no free space on the root disk. Any step that runs after this action and writes to the root disk can then fail with ENOSPC: no space left on device, even though the Nix store has plenty of room.

This surfaced in mdarocha/mdarocha.pl (the deploy workflow) and mdarocha/robo-intern-v2 when Dependabot bumped this action from v1.2.0 → v1.3.0 (v1.3.0 introduced the nothing-but-nix step).

Observed failure

In the mdarocha.pl deploy, devenv test passed (it lives in /nix), then pulumi/esc-action tried to install the Pulumi CLI (pulumi-v3.253.0-linux-x64.tar.gz, ~120 MB compressed) into ~/.pulumi/bin and died:

ENOSPC: no space left on device, copyfile 'pulumi-.../pulumi-language-nodejs' -> '/home/runner/.pulumi/bin/pulumi-language-nodejs'

df at that point:

/dev/root   72G  72G  26M 100%  /       ← 26 MB free on root
/dev/loop0  15G 1.4G  13G  10%  /nix    ← 15 GB image, backed by /mnt/disk0.img

There is no separate /mnt mount in df — /mnt is a directory on /dev/root, and the /nix image (/mnt/disk0.img) physically consumes root's free space.

Run history confirms it's tied to the version bump, not the runners or Nix: every run on main and other branches kept passing on v1.2.0 during the same window; only the v1.3.0 branch failed, and on every retry (run_attempt 2 and 3).

Root cause (in nothing-but-nix @ v10)

holster is level 0. At level 0 the only step that runs is "The Volume", which sizes the /nix image from /mnt:

free_space=$(df -m --output=avail /mnt | tail -n 1)
fallocate -l $((free_space - mnt-safe-haven))M "/mnt/diskN.img"

So it reserves only mnt-safe-haven (default 1024 MB) on whatever device backs /mnt, and gives everything else to /nix.

The root-safe-haven reserve (default 2048 MB) that would protect / is only applied inside add_expansion_disk(), which runs at level ≥ 1 (carve) or after purge at level ≥ 2. In holster (level 0) it is never called.

Consequences on a runner where /mnt lives on /:

• df --output=avail /mnt returns the root disk's free space.
• "The Volume" carves the /nix image out of that shared pool, leaving only mnt-safe-haven (~1 GB) free on the real root disk.
• root-safe-haven never applies in holster, so nothing else protects /.
• The build's own writes to root (workspace, caches) plus the Pulumi install exhaust that ~1 GB → ENOSPC.

The underlying assumption in nothing-but-nix "The Volume" is that / and /mnt are independent devices with independent free space. When /mnt is a path on /, that assumption breaks and the reserves don't protect the real device.

Options

1. Fix upstream (root cause): get nothing-but-nix "The Volume" to detect when /mnt shares the root device (e.g. compare stat -f -c %i / vs /mnt, or findmnt) and reserve against the true backing device / account for a combined safe haven. This fixes it for everyone regardless of protocol.

2. Workaround in this action (holster path only): raise the reserve that actually governs holster — mnt-safe-haven — when calling nothing-but-nix, so "The Volume" leaves more real headroom on the shared disk. Note: bumping root-safe-haven does nothing in holster (only affects rampage). Downside: on a normal runner with a separate /mnt, a larger mnt-safe-haven just leaves that space unused on the temp disk and gives /nix correspondingly less — a cost for no benefit there.

3. Per-repo escape hatch (already used): setting free-up-all-storage: true (rampage) works today because rampage purges preinstalled toolchains and runs add_expansion_disk, freeing real root space — but it's a heavy hammer (deletes toolchains) for what is really "leave more room free."

Notes

• Affected runners: whatever provisions these jobs has no separate /mnt disk (standard GitHub-hosted ubuntu-latest normally mounts a separate /dev/sdb1 at /mnt; these did not).
• Closed PR Reserve more root/mnt space in nothing-but-nix to avoid ENOSPC #21 attempted workaround (2) but conflated the two reserves; leaving the fix decision here.
• References: nothing-but-nix @ 687c797a730352432950c707ab493fcc951818d7 (v10), steps "The Volume" and "The Purge" / expand_nix_volume.sh.