cargo-target build caches have no retention policy — 96 dirs reached 1,155 GiB and took the shared volume to 0 bytes free (cause of #3797)

[tomerweller]

cargo-target/ build caches on the shared volume have no retention policy — 96 of them reached 1,155 GiB and took the volume to literally zero free bytes, killing the mainnet validator for 8 d 10 h (#3797) and every automated loop on the host

Filed at the request of #3749's resolution comment: "A cargo-target retention policy is the real fix. Reclaiming 1.15 TB by hand resolves the symptom; nothing currently prevents 96 more build caches from accumulating to the same point. … Worth its own issue if one doesn't already exist." Confirmed none does — searched cargo-target retention, retention policy build cache in:title, and dedupe dispatch worktree across open and closed issues.

The number

At 2026-08-08T22:45Z, immediately before the manual cleanup:

/dev/nvme1n1   1.7T used   0 bytes free   100%

96 co-tenant */cargo-target/ dirs          ~1,155 GiB
mainnet/ (live validator dataset)           172.30 GiB
74535976/ (monitor session)                   4.51 GiB

Every one of the 305 co-tenant trees was idle ≥ 9.3 days; 239 were idle > 30 d, the largest single tree idle 44 d. No process on the host held a cwd inside /home/tomer/data — every /proc/*/cwd was scanned. The entire 1.15 TB was regenerable compiler output that nothing was reading.

After the cleanup, 2 cargo-target dirs remain, and the retention gap is already visible in them:

  85.58 GiB  idle 37.8 d  b0db5fda    <- skipped: live open fd elsewhere in the tree
   3.79 GiB  idle 87.4 d  74535976    <- the monitor's own; holds the deployed binary

b0db5fda alone is 85.58 GiB of 37-day-idle build cache that survived only because an unrelated zsh had a log file open elsewhere under the same session dir. A policy keyed on the session being live will keep re-making that mistake; the unit of retention has to be the cargo-target directory itself.

Why this is worth a policy and not a cron rm

The failure was non-linear, and that is the part the "~11-day runway" framing in #3749's title got wrong:

1. Growth is bursty, not steady — 6 trees arrived in a 9 h window on 07-30, none in the 17.9 h before the volume filled.
2. At zero free bytes the recovery mechanisms die too. project-loop-watchdog.sh failed on every firing from 2026-07-31T13:30Z with could not create work tree dir … No space left on device; it could not clone a scratch checkout to run the headless tick that would have noticed. The backstop was disabled by exactly the condition it existed to survive.
3. So the observable signal is not a gentle slope into a warning — it is 8.8 days of total silence. See URGENT: mainnet validator OFFLINE 8d10h — ENOSPC kill at 2026-07-31T12:34:56Z, zero alarm signal (recurrence of closed #3478) #3797 §6 for the monitor-side equivalent (three independently-correct gates that together emitted nothing).

A threshold alert on free space would not have helped either: #3749 had one, it fired for weeks, and the reclaim it asked for needed an authorization that no automated actor held.

Suggested policy

Sized to what the census above actually shows:

1. Age-based reclaim of */cargo-target/ specifically, not of session dirs. Delete any cargo-target whose mtime is older than N days (N=7 would have been safe for all 96 — minimum idle was 9.3 d). Leave every sibling path alone: source trees, worktrees, logs, and diagnostic artifacts were correctly excluded by the manual cleanup and should be excluded structurally.
2. Skip on open file handles, but scope the skip to the cargo-target dir, not the parent session. The manual pass skipped the whole of b0db5fda because of an fd on maxtps-opt/live-bandiag.log; 85.58 GiB of unrelated build cache rode along.
3. Exempt an explicit allowlist — at minimum mainnet/ and the live monitor session's cargo-target, which holds the deployed validator binary (74535976/cargo-target/release/henyey, currently 0ac84d42). Deleting that one would turn a disk-pressure event into an unrecoverable one.
4. A shared CARGO_TARGET_DIR is the alternative worth costing out. Per-session target dirs are why the same dependency graph is compiled 96 times; one shared cache with cargo's own eviction would remove the class rather than trimming it. The tradeoff is lock contention between concurrent agent builds, which is why this is listed as an alternative and not the recommendation.
5. Report what was reclaimed. The manual pass produced a clean before/after; an automated one should too, or the next investigator sees a volume that shrank for unexplained reasons — the same ambiguity that made me mis-read buckets/ shrinkage as a sustained reclamation rate on Shared data volume /home/tomer/data at 95% — ~11-day exhaustion runway #3749.

Related

• URGENT: mainnet validator OFFLINE 8d10h — ENOSPC kill at 2026-07-31T12:34:56Z, zero alarm signal (recurrence of closed #3478) #3797 — the validator outage this caused (ENOSPC kill, 8 d 10 h, zero alarm signal).
• Shared data volume /home/tomer/data at 95% — ~11-day exhaustion runway #3749 — the disk-pressure thread and the manual reclaim; this issue is its follow-up 2.
• OFFLINE: validator aborted on ENOSPC (disk full) — bucket-flush fatal abort #3478 (closed) — the previous ENOSPC validator abort, 2026-06-18. Second occurrence on this volume.
• In-process retry-with-backoff / pause-and-resume on ENOSPC during bucket ops (operator-gated, beyond-core) #3498 (closed) — in-process pause-and-resume on ENOSPC; would have kept the process alive to be rescued.
• SQLite busy_timeout expiry silently drops 113 of 139 writes in 26d — including the retention trims that would relieve the contention, and catchup LedgerCloseMeta #3772 — retention trims starved by SQLite busy_timeout; same "the cleanup is the thing that fails first" shape, on the DB rather than the volume.

Reproduce

# per-cargo-target census with idle age — the unit a policy should act on
python3 - <<'EOF'
import os, time
base='/home/tomer/data'; now=time.time()
for d in sorted(os.listdir(base)):
    p=os.path.join(base,d,'cargo-target')
    if os.path.isdir(p):
        s=sum(os.lstat(os.path.join(r,f)).st_size
              for r,_,fs in os.walk(p, onerror=lambda e:None) for f in fs)
        print("%8.2f GiB  idle %5.1f d  %s"%(s/2**30,(now-os.stat(p).st_mtime)/86400,d))
EOF

# nothing is holding any of it
for p in /proc/[0-9]*; do readlink $p/cwd; done | grep -c '^/home/tomer/data/'

Node state

Catching Up — relaunched 2026-08-08T23:07:42Z on 0ac84d42 (henyey-v27.0.0-alpha.1) from persisted lcl 63,733,600 after the 8 d 10 h outage; restore-from-disk in progress, being watched for the #3702 near-tip stall. Volume 1,155.9 GiB free / 31 %. Deploys held on #3702.

Not filed urgent — the acute symptom is #3797 and the volume is no longer under pressure; this is the durable fix that stops it recurring.

— /monitor-tick 2871

[tomerweller]

Sizing correction for the retention policy: du -sb */ under-reports any tree that shares inodes with an alphabetically-earlier sibling, and one tree in the census is affected — 20260722-204000 reads 27.53 GiB but deleting it reclaims only 9.50 GiB, because 18.03 GiB is hardlinked into the live mainnet/ bucket set

Found while investigating an 18 GiB discrepancy in my own repeated measurements of a directory that was provably static (validator dead, nothing writing). It is a measurement artifact, and it lands directly on this issue's core ask — a policy sized on per-tree GiB.

The artifact

du -sb */          → mainnet/ = 172.719 GiB
du -sb mainnet/    → mainnet/ = 190.747 GiB
                                 ---------
                     difference    18.028 GiB

du counts a hardlinked inode once per invocation. In a single du -sb */ the trees are walked in alphabetical order, so whichever tree sorts first claims the shared bytes and every later tree is charged zero for them. 20260722-204000 sorts before mainnet, so it absorbs the shared bytes and mainnet/ is under-reported by exactly that amount.

Confirmed by inode analysis across all 300+ trees:

distinct inodes shared across >1 tree:  8      (18.03 GiB of unique bytes)
the 8 files:  mainnet/buckets/*.bucket.xdr, nlink=2
              incl. the two largest — 7.41 GiB and 6.94 GiB
the only two trees involved:  mainnet/  and  20260722-204000/

190.747 − 18.028 = 172.719 — the arithmetic closes exactly.

A false lead I checked and discarded: find <tree> -type f -links +1 reports 39,291 hardlinked files in b0db5fda and 1,259 in 9eb89c28, which looks like widespread cross-tree sharing. It isn't — those are links within a single tree (cargo build artifacts). Per-inode analysis shows sharing across trees is confined to the 8 files above. nlink>1 alone is not evidence of cross-tree sharing; you have to group by inode.

What this changes for the policy

Corrected census — apparent size vs. what deletion actually frees:

tree	apparent	shared	reclaimable
mainnet (live)	190.75 GiB	18.03	— do not touch
b0db5fda	86.76 GiB	0.00	86.76 GiB
20260722-204000	27.53 GiB	18.03	9.50 GiB
9eb89c28	23.05 GiB	0.00	23.05 GiB
do3357-a6f14d79	22.46 GiB	0.00	22.46 GiB
b9eecd09	22.46 GiB	0.00	22.46 GiB
68b329da	21.75 GiB	0.00	21.75 GiB

Two consequences:

1. Report reclaimable, not apparent. A policy that logs "freed 27.53 GiB" after deleting 20260722-204000 would overstate by 2.9×. Only one tree is affected today, but the reporting rule should be structural: compute per-inode ownership, or at minimum call du once per tree rather than du -sb */, so ordering cannot shift bytes between them.

2. Safety, and this is the sharper point: the shared inodes are LIVE validator bucket files. unlink() is safe — dropping one of two links leaves mainnet/buckets/ intact. But any cleanup that truncates or rewrites in place instead of unlinking would corrupt the live bucket set through the hardlink. Worth an explicit invariant in the policy: reclaim by unlink/rm only; never truncate, never open-for-write. That is not hypothetical caution — the two largest shared files (7.41 GiB and 6.94 GiB) are levels of the current bucket list.

Suggested addition to the scope from my earlier comment: item 6 — size and report by reclaimable bytes (per-inode ownership), and mandate unlink-only reclamation.

Incidental: first arrival since the reclaim

a6f14d79/ appeared at 2026-08-09T01:51:22Z — 13.21 GiB, containing do-3760, no cargo-target yet. First co-tenant arrival since 07-30, and a data point for this issue's arrival term: the process resumed within ~3 h of the volume being freed. Two cargo-target dirs currently exist (b0db5fda 85.58 GiB idle 37.9 d; 74535976 3.79 GiB, the monitor's own, holding the deployed binary).

Reproduce

cd ~/data
du -sb */ | grep mainnet      # under-reported
du -sb mainnet/               # true
find mainnet -type f -links +1 -printf '%n %s %p\n'   # the 8 shared inodes
# per-inode ownership across all trees (the correct sizing)
python3 - <<'PY'
import os, collections
base='/home/tomer/data'; own=collections.defaultdict(lambda:[0,set()])
for t in sorted(os.listdir(base)):
    p=os.path.join(base,t)
    if not os.path.isdir(p) or os.path.islink(p): continue
    for root,_,files in os.walk(p,onerror=lambda e:None):
        for f in files:
            try: st=os.lstat(os.path.join(root,f))
            except OSError: continue
            if st.st_nlink>1:
                e=own[st.st_ino]; e[0]=st.st_size; e[1].add(t)
sh=[(sz,ts) for sz,ts in own.values() if len(ts)>1]
print("shared inodes:",len(sh),"bytes: %.2f GiB"%(sum(s for s,_ in sh)/2**30))
PY

Node state

OFFLINE since 2026-08-09T00:29:20Z (1 h 41 m) on the knit-to-LCL FATAL — see #3797. Volume 1,142.6 GiB free / 31 %, so this is a sizing-accuracy correction, not disk pressure. Tree clean at df2abf83 == origin/main.

Not urgent; unchanged label.

— /monitor-tick 2878

[tomerweller]

Triage Report

Verdict: ACCEPT

Summary: Accurate and well-evidenced. The premise is confirmed by my own manual reclaim (the one this issue was filed from): 96 */cargo-target/ dirs totalling ~1,155 GiB took the shared volume to zero free bytes, which killed the mainnet validator (#3797) and froze every automated system on the host for ~8.8 days — including the project-loop-watchdog.sh crontab backstop, which failed every firing with No space left on device because it could not clone a scratch checkout. The backstop was disabled by the very condition it exists to survive. Nothing prevents recurrence.

Kind: enhancement (infrastructure / operational safety)

Severity: medium-high. No correctness impact, but the demonstrated blast radius is the largest of any open issue: total production-validator outage plus total pipeline outage, and the manual reclaim is not repeatable-by-design.

Recurrence is already measurable. Live census just now:

 3.8 G  idle 38 d  74535976/cargo-target        <- see the trap below
  77 G  idle 37 d  b0db5fda/cargo-target        <- skipped in the reclaim (live fd elsewhere in tree)
  11 G  idle  0 d  a6f14d79/do-3760/cargo-target <- created by a /do dispatch tonight

So ~92 GiB is already back, and one /do adds ~11 GiB. The issue's framing is right: the reclaim bought time, not a fix.

---

⚠️ A load-bearing constraint this issue's methodology would miss

An mtime-based retention policy would have deleted the running validator's own executable.

The validator runs from /home/tomer/data/74535976/cargo-target/release/henyey. That binary's mtime is 2026-07-01 — its build date. A running process does not touch its own executable's mtime, so this directory reads as idle 38 days while it is actively serving the production mainnet node. Any policy of the form "delete cargo-target idle > N days" deletes it.

This is not hypothetical: 74535976/ appears in this issue's own before-cleanup census at 4.51 GiB, and I excluded it during the manual reclaim only because it happened to have a live open file handle (a tail on monitor.log) — an incidental property, not a principled guard. Had that tail not been running, a correct-looking heuristic would have destroyed the binary.

The issue's liveness check is also insufficient for the same reason. It uses:

for p in /proc/[0-9]*; do readlink $p/cwd; done | grep -c '^/home/tomer/data/'

cwd is the wrong link. A process's working directory is typically the repo or /, not its build-output tree. The check that would have caught this is /proc/*/exe (and, for completeness, /proc/*/maps for mapped shared objects, plus open fds via /proc/*/fd/*). Recommend the plan treat "is anything executing from or mapped out of this tree" as a first-class liveness signal, distinct from "has anything modified it recently."

Implication for the policy design: idle-age alone is not a safe deletion criterion. Suggested directions for /plan to weigh (not decided here): require an affirmative liveness check across exe/maps/fd before deleting; and/or make the policy operate only on directories the pipeline itself created and can attribute (the /do-workspace marker convention at ~/data/do-$ISSUE.workspace already gives ownership metadata), rather than sweeping every cargo-target on a shared volume that other tenants and long-lived processes also use. A janitor that only reaps its own dispatches is far safer than a global sweeper, even if it reclaims less.

---

Scope question the planner must settle first: where the policy lives determines whether this is self-modifying pipeline tooling. If it lands in scripts/lib/* or the /do//review-pr skills' reaping logic, that is pipeline-internal and (per my standing policy) should be held for explicit operator go-ahead before implementation. If it lands as a standalone host-level janitor (cron script, systemd timer), it is ordinary infra. I am moving this to ready-for-planning rather than ready-for-doing, and flagging that the plan should state its target location explicitly so the self-modifying question is answered before any /do dispatch.

Not urgent (correctly not labelled so) — the volume is at 31 % and the acute symptom is #3797. This is the durable fix. No concurrent operator PR found.

[tomerweller]

Correcting my own census: it found 32 % of the build cache. The real total is 277.87 GiB across 68 dirs, not 89.37 GiB across 2 — and the path pattern I proposed (*/cargo-target/) matches only 6 of the 68, because 62 are named target, not cargo-target, and nearly all live 2–4 levels deep

My earlier comment sized the retention policy off a depth-1 scan (<session>/cargo-target). That scan is wrong in two independent ways, and a policy written to it would reclaim less than a third of what the manual cleanup already reclaimed by hand. Found while investigating a routine +21 GiB disk move.

What a full-depth scan finds

Walking to depth 4 and accepting any cargo-target/target directory that actually contains debug/ or release/:

build-cache dirs, any depth                              68 dirs   277.87 GiB
  depth-1   <session>/cargo-target                        2 dirs    89.37 GiB   ← my census
  deeper    <session>/<task>/cargo-target, .../worktree/target
                                                         66 dirs   188.50 GiB   ← MISSED

by basename:   target = 62 dirs      cargo-target = 6 dirs

The two defects:

1. Depth. Caches live at <session>/<task>/cargo-target and <session>/<task>/worktree/target, e.g.
   do3357-a6f14d79/do-3357/pr2/cargo-target (18.42 GiB) and
   20260619-002023/do-3439/worktree/target (1.97 GiB). A depth-1 glob sees neither.
2. Name. 62 of 68 are plain target/, the cargo default, not cargo-target/. My proposed pattern matches 6.

Together: the policy as I wrote it reclaims 89 GiB of 278 GiB (32 %).

This would have been a regression against the manual pass. The resolution comment on #3749 removed "96 cargo-target build-cache directories, ~1155 GB" — 96 dirs, so it clearly walked deeper than depth 1. Automating my version of the rule would have been strictly worse than what was already done by hand. Recording that explicitly so the policy is not written to my earlier numbers.

The idle distribution is unambiguous

idle > 11 d :  62 of 68 dirs
idle > 44 d :  ~55 dirs, most in the 51–54 d band
idle < 1 d  :   4 dirs — all in a6f14d79 (the tree that arrived 01:51Z today)

And a striking replication signal: 30+ worktree/target dirs are all exactly 1.97 GiB, ~52 days idle, spanning do-3319 … do-3447. That is one build graph compiled ~30 times and left behind — direct evidence for the shared-CARGO_TARGET_DIR alternative (item 4 of my earlier comment), which now looks like the higher-value fix rather than the footnote I made it.

Corrected scope items

Replacing the sizing language in my earlier comment:

1. Match by structure, not by name or depth. A reclaim candidate is any directory named cargo-target or target that contains a debug/ or release/ subdirectory, at any depth under a session tree. The debug//release/ test is what makes this safe — it will not match a source directory that happens to be called target.
2. Do not descend into a matched dir. Once a cache root is identified, stop — otherwise nested target/ inside debug/ double-counts.
3. Exempt list unchanged and still load-bearing: mainnet/ and the live monitor session's 74535976/cargo-target (holds the deployed 0ac84d42 binary).
4. Reclaimable-bytes reporting and unlink-only reclamation — unchanged from my prior comment, and the hardlink caveat there still applies (8 inodes / 18.03 GiB shared between mainnet/buckets/ and 20260722-204000/).
5. Shared CARGO_TARGET_DIR is promoted from "alternative worth costing out" to the primary structural fix, on the strength of the 30× 1.97 GiB replication above. Trimming 278 GiB on a schedule treats a symptom that a single shared cache would not produce.

Reproduce

cd ~/data
python3 - <<'PY'
import os
base='/home/tomer/data'; hits=[]
for root,dirs,_ in os.walk(base,onerror=lambda e:None):
    if root[len(base):].count(os.sep) > 4:
        dirs[:] = []; continue
    for name in list(dirs):
        if name in ('cargo-target','target'):
            p=os.path.join(root,name)
            if os.path.isdir(os.path.join(p,'debug')) or os.path.isdir(os.path.join(p,'release')):
                hits.append(p); dirs.remove(name)
def sz(p): return sum(os.lstat(os.path.join(r,f)).st_size
                      for r,_,fs in os.walk(p,onerror=lambda e:None) for f in fs)
d1=[h for h in hits if h[len(base)+1:].count(os.sep)==1]
print("total %d dirs %.2f GiB | depth-1 %d dirs %.2f GiB"%(
      len(hits), sum(map(sz,hits))/2**30, len(d1), sum(map(sz,d1))/2**30))
PY

Node state

OFFLINE since 2026-08-09T00:29:20Z (4 h 20 m) on the knit-to-LCL FATAL — #3797. Persisted lcl 63,733,600, mainnet/ intact and not wiped, three rotated logs preserved. Volume 1,121.2 GiB free / 33 % — this is a sizing correction, not disk pressure. Deployed 0ac84d42; tree clean at df2abf83 == origin/main; deploys held on #3702.

Incidental arrival datum: a6f14d79 grew 13.21 → 33.81 GiB in the ~2 h since it appeared (two task subtrees, do-3760 and do-3802), which is the arrival term this issue is ultimately about.

Not urgent; label unchanged.

— /monitor-tick 2883

[tomerweller]

📝 Plan Draft (Round 1)

Summary: Add a standalone, ENOSPC-survivable host-level janitor (scripts/cargo-target-reaper.sh, cron-installed) that reclaims idle Rust build-cache dirs under ~/data using an affirmative exe/maps/fd liveness guard, structural (not name/depth) matching, unlink-only reclamation, and per-tree reclaimable-bytes reporting.

Scope decision (settles the triage's load-bearing question first): The policy lands as an ordinary standalone host janitor — a repo-canonical script + crontab entry, exactly the shape of scripts/project-loop-watchdog.sh — not in scripts/lib/* and not in the /do///review-pr reaping logic. It therefore is not self-modifying pipeline tooling and needs no operator hold. It only reads the pipeline's ~/data/do-$ISSUE.workspace ownership markers as an optional attribution signal; it does not change any skill's behavior.

Files to modify:

• scripts/cargo-target-reaper.sh (new) — the janitor. Canonical copy in repo; a live copy at /home/tomer/data/cargo-target-reaper.sh is what cron runs (same canonical/live split the watchdog documents).
• scripts/cargo-target-reaper.test.sh (new) — TAP-style bash test harness (same bash-runner convention as scripts/lib/tests/test-do-bootstrap.bats, invoked directly with bash, ships its own runner).
• .github/workflows/ci.yml — add a cargo-target-reaper job (mirrors the agent-worktree-contract job) running bash scripts/cargo-target-reaper.test.sh.
• scripts/README.md or a header comment block — install instructions (0 4 * * * /home/tomer/data/cargo-target-reaper.sh) and the canonical/live-copy note. (Header comment only; no separate doc file created.)

Approach:

Discovery (structural, any-depth). Walk ~/data to a bounded depth (4) and treat a directory as a build-cache root iff its basename is cargo-target or target and it directly contains a debug/ or release/ subdir. On match, record it and do not descend further (prevents nested-target double counting). This is the correction from the issue's third comment — 62 of 68 real caches are named target, and most live 2–4 levels deep; a */cargo-target/ depth-1 glob catches only 32%. The debug/|release/ test is what keeps a source dir coincidentally named target from matching.

Affirmative liveness guard (the load-bearing safety fix). Before deleting any candidate, prove nothing is using it. Build the set of "in-use" absolute paths from /proc/[0-9]*/exe (readlink), every mapped file in /proc/[0-9]*/maps, and every /proc/[0-9]*/fd/* target. A candidate is skipped iff any in-use path is equal to or lives under that candidate dir. Crucially the skip is scoped to the candidate cache dir, not its parent session (fixes the b0db5fda "85 GiB rode along on an unrelated log fd" problem). This replaces the triage-flagged unsafe cwd-only / mtime-only heuristic that would have deleted the running validator's own 74535976/cargo-target/release/henyey (its mtime is its build date → reads as 38 d idle while actively serving).

Age gate. Among live-clear candidates, delete those whose dir mtime is older than REAP_AGE_DAYS (default 7; min observed idle across the 96-dir census was 9.3 d, so 7 is conservative-safe). Configurable via env.

Exempt allowlist (defense in depth). Never touch ~/data/mainnet/ or any path the liveness guard flags; additionally hard-exempt the deployed-binary tree by allowlist even if the guard somehow misses it. Allowlist is an env-overridable newline list defaulting to mainnet.

Unlink-only reclamation. Reclaim exclusively via rm -rf (unlink); the script never truncates or opens-for-write any candidate. Documented as an invariant because 8 inodes / 18.03 GiB are hardlinked between mainnet/buckets/ and a session tree — unlink is safe, in-place rewrite would corrupt the live bucket set. (Bucket files are not inside any target/, so structural match won't reach them; the invariant is belt-and-suspenders.)

Reporting (reclaimable, not apparent). Size each candidate with a separate du -sb per tree (never du -sb */, which mis-attributes shared inodes and would overstate 20260722-204000 by 2.9×). Emit a before/after + per-dir reclaimed + total to a bounded log (/home/tomer/data/cargo-target-reaper.log, ~5 MB cap, same rotation as the watchdog).

ENOSPC survivability. Unlike the watchdog, the reaper does no git clone and needs only coreutils + python3 already on the host, so it runs to completion at zero free bytes — it is the recovery mechanism that must survive the exact condition it fixes (the #3797/#3749 lesson). A --dry-run flag (and DRY_RUN=1) prints the plan without deleting, for safe first runs and for the test harness.

Test plan (kind: enhancement → New coverage required):

• New coverage — scripts/cargo-target-reaper.test.sh, TAP-style, each staging a fake ~/data-like root under a mktemp -d (never the repo tree, never real ~/data):
  • test_matches_target_and_cargo_target_with_debug_release — asserts a sess/task/target/debug and a sess/cargo-target/release are both discovered; a plain sess/foo/target/ with no debug/release is not matched (source-dir false-positive guard).
  • test_no_descend_into_matched_root — a target/debug/target/ nested dir is counted once, not twice.
  • test_age_gate — a candidate with mtime older than REAP_AGE_DAYS is reaped; one newer is kept.
  • test_liveness_exe_skip — spawn a sleeper process whose executable is copied inside a candidate (or hold an open fd via exec 3<file) and assert that candidate is skipped while a sibling with no fd is reaped; assert skip is scoped to the cache dir, not the parent (sibling cache under same parent still reaped).
  • test_exempt_allowlist — a mainnet/-named candidate is never reaped even when old and idle.
  • test_unlink_only_and_reporting — reaped bytes are computed per-tree (du -sb per dir) and the report total equals the sum of removed trees; assert the script issues no truncate/> against candidates (grep the script source for the invariant, plus assert hardlinked sibling outside the candidate survives).
  • test_dry_run_deletes_nothing — --dry-run prints candidates but removes nothing.
• Existing tests preserved: scripts/test-agent-worktree-contract.sh, scripts/lib/tests/test-do-bootstrap.bats, scripts/test-shell-lib-cross-shell.sh must keep passing (the reaper adds no code to scripts/lib/*, so these should be unaffected; listed because CI runs them in the same shell-test tier).

Parity considerations: n/a — does not change the observable surface. This is host operational tooling (build-cache reclamation on a shared dev volume); it touches no ledger/bucket hash, result/meta XDR, SCP/overlay wire, archive format, HTTP/RPC/CLI, or crypto output, and no Rust crate.

Out of scope (deferred, not dropped): A shared CARGO_TARGET_DIR (the issue's promoted "primary structural fix") would edit every skill's bootstrap under scripts/lib/* — that is self-modifying pipeline tooling (triage's operator-hold class) and a much larger, separately-reviewable change. It removes the class; this janitor safely trims the symptom and stops recurrence now. Recommend filing it as a follow-up rather than bundling (keeps this PR atomic + no operator hold). Will file on convergence if Critic C agrees.

Risks:

• /proc/*/maps and /proc/*/fd scans race with process churn; mitigated because the guard fails safe (a transient false "in-use" only defers a reap by one cycle; a missed in-use would be the danger, so the scan reads all three sources and treats any hit as authoritative).
• Symlinks under ~/data could point outside it; discovery must not follow symlinked dirs out of the root (use os.walk(..., followlinks=False) and canonicalize).
• Depth cap of 4 could miss an unusually deep cache; census shows real caches at depth ≤ 3, so 4 has margin, but it is a tunable.

[tomerweller]

🔍 Critic A (correctness) — Round 1

Verdict: REVISE-MINOR

Key concerns:

• The /proc liveness scan must treat unreadable entries (EPERM) conservatively — on a shared volume a cache owned by tomer could be in use by another tenant's process whose /proc/<pid>/{exe,maps,fd} the reaper can't read. Not deletion-unsafe in the current single-user setup, but the plan should state the failure mode (log + skip-that-pid) so a future multi-user host doesn't silently lose the guard.
• The "in-use path is under candidate" test must compare by canonicalized path components, not string prefix — otherwise ~/data/foo (candidate) would spuriously match an in-use ~/data/foo-target/..., and a symlinked in-use path could evade the check. Same boundary-check lesson as require_home_data_path in agent-worktree-contract.sh. Plan mentions canonicalize; make the component-boundary compare explicit.
• test_liveness_exe_skip is feasible in CI (cp /bin/sleep cand/release/x; cand/release/x 30 & readlink /proc/$!/exe) — good — but also add an open-fd variant (exec 3<cand/release/lib) since fd and maps are separate code paths from exe and each needs its own assertion.

Full review

The plan correctly solves the stated problem and — importantly — directly fixes the specific trap triage flagged: it keys deletion on an affirmative exe/maps/fd liveness proof rather than mtime/cwd, so the running validator's 74535976/cargo-target/release/henyey (build-date mtime, reads as 38 d idle) is protected because /proc/<validator-pid>/exe resolves under that candidate → skipped. The skip is correctly scoped to the candidate cache dir rather than the parent session, fixing the b0db5fda "85 GiB rode along on an unrelated fd" defect.

Structural matching (basename ∈ {target, cargo-target} ∧ contains debug/|release/, any depth ≤4, no-descend-on-match) is the right correction to the depth-1/name defects documented in the third issue comment, and the debug/release predicate correctly excludes source dirs named target.

The age gate on cache-root mtime is a sound heuristic (not a safety mechanism — the liveness guard is), and the min-idle 9.3 d census justifies the 7 d default with margin.

Test coverage matches every new behavioral branch (match, no-descend, age, liveness, exempt, unlink-only, dry-run). No pub/visibility-narrowing or dead-code concern applies (new standalone shell script; the repo-wide -Dwarnings Rust gate is irrelevant here).

The three minor items above are hardening, not blockers — none invalidates the approach. Recommend the doer fold them in; none needs a round 2.

[tomerweller]

🔍 Critic B (parity) — Round 1

Verdict: APPROVE

Key concerns:

• None. The change does not touch the observable/interop surface defined in docs/PARITY.md.

Full review

Evaluated strictly against docs/PARITY.md's observable/interop surface: ledger/bucket hashes, transaction result & meta XDR, SCP/overlay wire bytes, history archive format, HTTP/RPC/CLI contracts, crypto outputs.

This plan adds a host-level bash janitor that reclaims regenerable Rust build-cache directories under ~/data. It compiles nothing, changes no crate, and emits no bytes on any wire or archive. There is no stellar-core counterpart to mirror — build-cache reclamation on a developer volume is not a protocol behavior.

One adjacent check I made explicitly: the reaper must never touch the live bucket set. The plan's unlink-only invariant plus the fact that mainnet/buckets/*.bucket.xdr files live outside any target//cargo-target/ tree (so structural matching cannot reach them) means the ledger/bucket-hash surface is untouched even for the 8 hardlinked inodes shared between mainnet/ and a session tree. The explicit mainnet/ allowlist is redundant belt-and-suspenders here, which is fine.

Internal/operational tooling, metrics, logging, and host maintenance are explicitly out of the parity surface — this is squarely in the "MAY deviate freely" tier. Approve.

[tomerweller]

🔍 Critic C (scope) — Round 1

Verdict: REVISE-MINOR

Key concerns:

• Scope is just right for a single reviewable PR: one script + one test harness + one CI job, comfortably under ~500 net lines, one atomic concern (age-gated, liveness-guarded reclamation). It addresses the root cause named in the issue title ("no retention policy"), not a symptom.
• The deferral of shared CARGO_TARGET_DIR is the correct cut and is what keeps this from becoming too broad — but the plan says "will file on convergence if Critic C agrees." I agree: file it now as an explicit follow-up so the issue's promoted "primary structural fix" is preserved as backlog rather than living only in a paragraph of this plan.
• Confirm the janitor is genuinely standalone infra (repo script + crontab) and touches nothing under scripts/lib/* or the /do///review-pr reaping paths — that is what keeps it out of the triage's self-modifying/operator-hold class. The plan states this; hold the line in /do (a CI job in ci.yml is fine and not self-modifying).

Full review

Not too narrow: an age + liveness janitor is the durable recurrence-stopping fix the issue asks for, not a band-aid — the band-aid was the one-time manual rm. Not too broad: the one legitimately larger idea (shared CARGO_TARGET_DIR, which removes the duplication class rather than trimming it) is correctly excluded, both because it is a separate architectural concern and because it edits every skill bootstrap under scripts/lib/* and thus lands in the operator-hold class the triage flagged. Splitting it out is right.

My only actionable ask upgrades the plan's conditional ("if Critic C agrees") to a definite: file the shared-CARGO_TARGET_DIR follow-up issue on convergence, label it follow-up, and reference #3798. That preserves the disagreement-free-but-deferred idea as a tracked item. This is a REVISE-MINOR, not a blocker.

[tomerweller]

✅ Converged Plan

Summary: Add a standalone, ENOSPC-survivable host janitor (scripts/cargo-target-reaper.sh, cron-installed) that reclaims idle Rust build-cache dirs under ~/data via an affirmative exe/maps/fd liveness guard, structural (not name/depth) matching, unlink-only reclamation, and per-tree reclaimable-bytes reporting.

Scope decision (settles triage's self-modifying question): Lands as ordinary standalone host infra — repo-canonical script + crontab entry, same shape as scripts/project-loop-watchdog.sh. It does not touch scripts/lib/* or the /do///review-pr reaping logic, so it is not self-modifying pipeline tooling and needs no operator hold. Adding a CI job in ci.yml is ordinary and does not change that classification.

Files to modify:

• scripts/cargo-target-reaper.sh (new) — the janitor. Canonical copy in repo; live copy at /home/tomer/data/cargo-target-reaper.sh is what cron runs (same canonical/live split the watchdog documents). Header comment carries install line 0 4 * * * /home/tomer/data/cargo-target-reaper.sh and the canonical/live note (no separate doc file).
• scripts/cargo-target-reaper.test.sh (new) — TAP-style bash harness, invoked with bash and shipping its own runner (same convention as scripts/lib/tests/test-do-bootstrap.bats).
• .github/workflows/ci.yml — add a cargo-target-reaper job mirroring the agent-worktree-contract job: run: bash scripts/cargo-target-reaper.test.sh.

Approach:

Discovery (structural, any-depth ≤4, no-descend-on-match). A directory is a build-cache root iff basename ∈ {cargo-target, target} and it directly contains debug/ or release/. On match, record and stop descending (avoids nested-target double count). Walk with followlinks=False. This is the correction from the issue's third comment: 62/68 real caches are named target and live 2–4 levels deep; a depth-1 */cargo-target/ glob catches only 32%. The debug/release predicate excludes source dirs coincidentally named target.

Affirmative liveness guard (load-bearing safety). Build the in-use path set from /proc/[0-9]*/exe (readlink), all mapped files in /proc/[0-9]*/maps, and all /proc/[0-9]*/fd/* targets. Skip a candidate iff any in-use path equals or is under it, compared by canonicalized path components (not string prefix). Skip is scoped to the candidate cache dir, not its parent session. Unreadable /proc/<pid> entries (EPERM) are logged and that pid treated conservatively. This is what protects the running validator's 74535976/cargo-target/release/henyey (build-date mtime → reads 38 d idle while serving) and fixes the b0db5fda parent-scoped-skip defect.

Age gate. Among live-clear candidates, delete those with dir mtime older than REAP_AGE_DAYS (default 7; census min idle 9.3 d). Env-configurable.

Exempt allowlist. Never touch ~/data/mainnet/; env-overridable newline list defaulting to mainnet. Belt-and-suspenders atop the liveness guard.

Unlink-only reclamation. Reclaim exclusively via rm -rf; never truncate/open-for-write any candidate (invariant, because 8 inodes/18.03 GiB hardlink mainnet/buckets/ to a session tree — unlink is safe, in-place rewrite corrupts the live bucket set).

Reporting. Size each candidate with a separate du -sb per tree (never du -sb */, which mis-attributes shared inodes). Emit before/after + per-dir reclaimed + total to a bounded /home/tomer/data/cargo-target-reaper.log (~5 MB cap, watchdog-style rotation).

ENOSPC survivability + dry-run. No git clone; needs only coreutils + python3 already on host, so it completes at zero free bytes — the recovery mechanism must survive the condition it fixes (#3797/#3749). --dry-run / DRY_RUN=1 prints the plan without deleting (safe first runs + test harness).

Kind: enhancement (infrastructure / operational safety) (from triage)

Test plan:

• Regression tests: n/a (enhancement, not a bug-fix — no pre-existing code path regressed).
• New coverage (scripts/cargo-target-reaper.test.sh, each staging a fake root under mktemp -d, never repo tree / real ~/data):
  • test_matches_target_and_cargo_target_with_debug_release — target/debug and cargo-target/release discovered; plain target/ w/o debug|release not matched.
  • test_no_descend_into_matched_root — nested target/debug/target counted once.
  • test_age_gate — old candidate reaped, newer kept.
  • test_liveness_exe_skip — sleeper exec'd from inside a candidate (cp /bin/sleep cand/release/x; cand/release/x 30 &) is skipped; a sibling cache under the same parent with no user is still reaped (skip is dir-scoped).
  • test_liveness_fd_skip — separate assertion via an open fd (exec 3<cand/release/lib), exercising the fd/maps path distinct from exe.
  • test_exempt_allowlist — mainnet/-named candidate never reaped even when old/idle.
  • test_unlink_only_and_reporting — per-tree du -sb total equals removed bytes; a hardlinked file outside a reaped candidate survives; grep asserts the script contains no truncate/>-redirect against candidates.
  • test_dry_run_deletes_nothing — --dry-run prints but removes nothing.
• Existing tests preserved: scripts/test-agent-worktree-contract.sh, scripts/lib/tests/test-do-bootstrap.bats, scripts/test-shell-lib-cross-shell.sh (same shell-test CI tier; reaper adds nothing to scripts/lib/*, so unaffected).

Parity considerations: n/a — does not change the observable surface. No crate, no wire/archive/XDR/hash/crypto/HTTP-RPC-CLI output touched. Explicitly in the "MAY deviate freely" tier of docs/PARITY.md.

Minor items to consider during implementation (from critics, fold in during /do):

• Treat unreadable /proc/<pid> entries conservatively (log + skip-that-pid); note the multi-tenant caveat. (Critic A)
• Compare in-use-under-candidate by canonicalized path components, not string prefix, to avoid foo vs foo-target false matches and symlink evasion — same lesson as require_home_data_path. (Critic A)
• Keep fd/maps and exe as separately-asserted test branches. (Critic A)
• Hold the standalone-infra line in /do: nothing under scripts/lib/* or skill reaping paths. (Critic C)

Refuted concerns: none (no REVISE-MAJOR was raised).

Sub-issues filed: #3854 — evaluate shared CARGO_TARGET_DIR (the issue's promoted "primary structural fix"), deferred because it edits every skill bootstrap under scripts/lib/* (operator-hold / self-modifying class) and is a separately-reviewable architectural change.

Convergence: Round 1, verdicts: A=REVISE-MINOR, B=APPROVE, C=REVISE-MINOR (no REVISE-MAJOR → converged without a refute pass).