diff --git a/rfcs/0001-bounded-cargo-build-storage.md b/rfcs/0001-bounded-cargo-build-storage.md new file mode 100644 index 0000000000..96d6419842 --- /dev/null +++ b/rfcs/0001-bounded-cargo-build-storage.md @@ -0,0 +1,335 @@ +# RFC 0001: Bounded Cargo build storage + +- Status: Proposed +- Created: 2026-08-06 +- Owners: Clanker build and developer tooling +- Supersedes: none +- Related: PR #31, `scripts/cargo_target_size.py` + +## Summary + +Clanker should treat local Cargo build output as a bounded, observable resource. +The recommended architecture uses Cargo's stable `build.build-dir` setting to +keep final artifacts in each worktree's normal `target/` while moving +intermediate output under +`{cargo-cache-home}/clanker/build/{workspace-path-hash}`. This creates one +Clanker-owned cache root without forcing concurrent worktrees to share a Cargo +lock domain. + +Managed build commands should report both the current worktree and aggregate +linked-worktree usage, warn before the cache becomes disruptive, and refuse to +start another high-growth build after a hard limit is crossed. Cleanup should be +explicit, safe, and available for either the current worktree or selected linked +worktrees. + +The proposed default budgets are: + +| Scope | Warning | Hard limit | +|---|---:|---:| +| One worktree build directory | 40 GiB | 80 GiB | +| Clanker intermediate-cache root | 100 GiB | 160 GiB | + +Crossing a warning limit prints a concise diagnosis and cleanup command. +Crossing a hard limit prevents managed commands such as `just test`, `just fix`, +and local release builds from starting until storage is reclaimed or the user +sets an explicit, one-command override. The guard does not delete files +automatically. + +## Motivation + +On 2026-08-06, `du` attributed 333 GiB of filesystem blocks to the primary +Clanker checkout's `codex-rs/target`. `cargo clean` removed 310,061 files and +reported 355.7 GiB of logical file content. A linked-worktree scan attributed +another 256 GiB of blocks to 11 target directories, including individual +directories of 86 GiB and 92 GiB. + +Cleaning all linked target directories removed 659,784 generated file entries. +Before deletion, `du` attributed approximately 590 GiB of blocks to those paths; +Cargo reported 627.9 GiB of logical file content. The Btrfs filesystem's free +space did not materially increase after deletion and a filesystem sync, and no +large deleted build artifacts remained open. The extents were therefore still +shared or retained by snapshots. This is important: neither `du` block counts +nor logical file sizes equal exclusive physical bytes or bytes that deletion +will immediately reclaim. + +The generated namespace was still unacceptably large, but the incident also +shows that the current measurement semantics are not clear enough. Reports must +distinguish logical content, blocks attributed to paths, exclusive physical +storage when the filesystem can provide it, and observed free-space changes. + +PR #31 already reduced routine artifact size by disabling incremental +compilation and debug information in the default development and test profiles. +It also added `just target-size`. Those were useful mitigations, but they do not +bound growth: + +- old artifacts survive profile, toolchain, feature, and dependency changes; +- each worktree gets an independent target directory; +- the existing command only inspects one target directory; +- it reports logical file sizes but not block attribution, exclusive storage, + or observed reclaim; +- no command warns, stops, or offers a safe pruning workflow. + +The result is silent, effectively unbounded growth until the filesystem is under +pressure. + +## Research findings + +- Cargo 1.91 stabilized separate `target-dir` and `build-dir` locations. Final + artifacts stay in `target-dir`; compiler, dependency, fingerprint, and build + script intermediates live in `build-dir`. Clanker pins Cargo 1.97. See the + [Cargo build-cache documentation](https://doc.rust-lang.org/cargo/reference/build-cache.html) + and [Rust 1.91 release notes](https://doc.rust-lang.org/stable/releases.html#version-1910-2025-10-30). +- `build.build-dir` supports `{cargo-cache-home}` and + `{workspace-path-hash}`. Cargo upstream is considering the same general + layout as a future default, while warning that projects must test tools that + depend on Cargo's internal build layout. See the + [configuration reference](https://doc.rust-lang.org/cargo/reference/config.html#buildbuild-dir) + and [cargo#16147](https://github.com/rust-lang/cargo/issues/16147). +- Cargo's stable automatic garbage collection covers global downloaded and + extracted caches, not a size budget for workspace build directories. See the + [Cargo 1.88 changelog](https://doc.rust-lang.org/cargo/CHANGELOG.html#cargo-188-2025-06-26). +- Cargo 1.97 refuses `cargo clean --target-dir` when the path does not look like + a Cargo target directory. Clanker's cleanup tooling should preserve that + defense. See the + [Cargo 1.97 changelog](https://doc.rust-lang.org/cargo/CHANGELOG.html#cargo-197-2026-07-09). +- `sccache` can share compiler results between worktrees and caps its local + cache with `SCCACHE_CACHE_SIZE`, but it does not cache every Rust output. It + is a rebuild accelerator, not the storage policy. See the + [sccache Rust documentation](https://github.com/mozilla/sccache/blob/main/docs/Rust.md) + and [local-cache configuration](https://github.com/mozilla/sccache/blob/main/docs/Configuration.md#disk-local). + +## Goals + +- Keep routine local Clanker build output within an explicit disk budget. +- Make total usage across linked worktrees visible from any worktree. +- Make reclaiming generated output safe and obvious. +- Preserve an escape hatch for deliberate profiling and full-matrix work. +- Keep the implementation portable across Linux, macOS, and Windows. +- Avoid surprising deletion during an ordinary build or test command. + +## Non-goals + +- Managing Cargo's global registry and Git caches. +- Managing Bazel, npm, model, application, or operating-system caches. +- Guaranteeing that direct `cargo` invocations obey repository policy. +- Making every worktree share one target directory. +- Preserving target artifacts as durable build outputs. + +## Proposal + +### 1. Separate final artifacts from intermediate output + +Add this checked-in Cargo configuration: + +```toml +[build] +build-dir = "{cargo-cache-home}/clanker/build/{workspace-path-hash}" +``` + +Keep `target-dir` at its default. Final binaries, documentation, and package +output remain discoverable under each worktree's `target/`. Large dependency, +fingerprint, build-script, and compiler intermediates move into one managed +Clanker cache root. + +The workspace-path hash preserves one Cargo lock domain per worktree. A fixed, +shared build directory is not proposed because concurrent agents would contend +on the same Cargo lock and mix branch/feature invalidation state. + +Before rollout, audit the repository for assumptions about `target/debug/deps`, +build-script `OUT_DIR`, and Cargo's internal layout. Validate the new layout on +Linux, macOS, and Windows. + +### 2. Replace the single-directory report with a storage inventory + +Extend the target-size tool to discover linked worktrees using +`git worktree list --porcelain` and query `cargo metadata` for both the target +and build directories. For each resolved directory, report: + +- resolved path; +- unique logical bytes, counting hard-linked files once per target; +- filesystem blocks attributed to the path when the platform exposes them; +- exclusive physical bytes when the filesystem exposes that information + without elevated privileges; +- last artifact modification time; +- top-level profile breakdown such as `debug`, `release`, and `profiling`. + +The default output should end with per-worktree and aggregate totals. A JSON mode +should expose the same data for tests and future automation. The report must +label every measurement. It must not call logical or path-attributed bytes +"physically reclaimable" without evidence. Cleanup commands should also report +filesystem free space before and after deletion, while noting that snapshots, +compression, reflinks, and delayed allocation can affect the result. + +Budget enforcement uses unique logical bytes. That metric is conservative and +portable, and it bounds the generated artifact set independently of filesystem +compression, reflinks, and snapshot policy. Physical and path-attributed +measurements remain diagnostics rather than enforcement inputs. + +The tool should honor `CARGO_TARGET_DIR` and `CARGO_BUILD_BUILD_DIR` for the +current worktree. Worktrees with explicit directories outside their checkout +may be added through repeatable command-line options. Paths are resolved before +inspection, and duplicate directories are counted only once. + +### 3. Add explicit cleanup commands + +Provide these repository commands: + +- `just build-cache-clean-current`: run Cargo's supported cleanup for the + current worktree. +- `just build-cache-clean-worktrees`: print linked build directories, their + sizes, and the exact directories proposed for cleanup; require explicit + confirmation before cleaning non-current worktrees. +- `just build-cache-clean-worktrees --dry-run`: report what would be removed + without mutating anything. + +For a live worktree, cleanup must invoke `cargo clean` with that worktree's +manifest and configuration so Cargo resolves both its target and build +directories. It must reject symlinked directories and paths outside the resolved +Clanker cache root. If Cargo rejects a missing or invalid `CACHEDIR.TAG`, the +tool must stop and explain the issue; it must not silently bypass Cargo's safety +check. An orphaned hash directory may be removed only after the tool confirms +that its owning worktree no longer exists, resolves it as a direct child of the +cache root, and validates its Cargo cache marker. + +No age-based or background deletion is proposed. Modification time is useful +for choosing what to clean, but it is not reliable evidence that another agent +or process has finished with a worktree. + +The tool may automatically clean cache entries whose owning worktree no longer +exists. Cleaning a linked worktree always requires explicit selection. + +### 4. Enforce budgets in managed high-growth commands + +Add a fast preflight to repository-managed commands that can materially grow +the build directory, initially: + +- `just test`; +- `just fix` and `just clippy`; +- local debug and release build recipes; +- packaging recipes that write into the normal Cargo build directory. + +At the warning limit, the command proceeds after printing the largest build +directories and the relevant cleanup command. At the hard limit, it exits before +starting Cargo. Read-only reports and cleanup commands always remain available. + +Defaults are 40/80 GiB for the current build directory and 100/160 GiB +aggregate. These values leave room for a scoped test build and several active +worktrees while +preventing a repeat of the incident. They should be reviewed after 30 days of +telemetry from normal scoped builds. + +An exceptional build may set `CLANKER_BUILD_BUDGET_OVERRIDE_GIB` for that one +invocation. The preflight prints that an override is active and still reports +usage. The override is intentionally not a persistent config file: repeatedly +needing it means the defaults or build behavior should be revisited. + +Direct `cargo` commands cannot be intercepted reliably and remain outside this +guard. Contributor and agent instructions should continue to prefer the `just` +recipes. + +A literal host-level bound requires placing the central cache root on a +filesystem or subvolume with a quota. That is a useful optional workstation +policy, but it cannot be imposed portably by this repository. + +### 5. Keep profiles compact and make exceptional artifacts explicit + +Retain the compact `dev` and `test` profile settings introduced in PR #31. +Symbol-bearing builds should continue to use the explicit `profiling` profile. +Routine instructions should not recommend `--all-features`; scoped crate tests +remain the default. + +The profiling profile counts toward reported usage but may use the one-command +budget override. This makes the cost visible without weakening the normal +policy. + +### 6. Test the policy without producing a large cache + +The inventory and policy logic should operate on injected repository, target, +and build paths so tests can use temporary fixture trees. Tests should cover: + +- one worktree below, at, and above each threshold; +- aggregate usage across multiple linked worktrees; +- duplicate and symlinked target paths; +- `CARGO_TARGET_DIR` and `CARGO_BUILD_BUILD_DIR` handling; +- dry-run cleanup output; +- malformed or missing Cargo cache markers; +- logical-size behavior when filesystem block or exclusive-byte measurements + are unavailable. + +CI should test the tooling and threshold behavior with fixtures. It should not +grow an actual target directory merely to exercise the limit. + +## Why not share one target directory? + +A shared target can deduplicate dependency artifacts across worktrees, but it +also makes concurrent agents contend on Cargo locks and mixes feature, profile, +compiler, and branch churn into one invalidation domain. It changes build +latency and failure isolation in ways that need measurement. It is therefore not +the default proposed here. + +An opt-in experiment may compare a shared target against isolated, bounded +targets. Any adoption would require evidence that saved storage outweighs lock +contention for concurrent agent workloads. + +## Why not add sccache first? + +`sccache` can make rebuilding after cleanup cheaper, but it creates a second +cache that also needs a size limit. It does not remove linked binaries, test +harnesses, build-script outputs, or stale fingerprints from Cargo target and +build directories. It is complementary, not a substitute for a storage budget. + +After the basic guard lands, an opt-in `sccache` experiment should use an +explicit local cap and measure clean-build time, warm-build time, and total disk +usage before becoming a default. + +## Alternatives considered + +### Continue relying on compact Cargo profiles + +Rejected. The incident happened after the repository had a size-report command +and compact profile defaults. These settings reduce the slope but do not impose +a bound or handle stale and duplicated worktree output. + +### Automatically delete least-recently-used worktrees during builds + +Rejected. File timestamps do not prove that a worktree is inactive, and an +ordinary build should not invalidate another agent's artifacts. Explicit cleanup +is slower but predictable and auditable. + +### Run periodic unconditional `cargo clean` + +Rejected as the primary mechanism. It hides the growth pattern, causes +surprising rebuilds, and is difficult to schedule portably. A user may still +schedule the explicit cleanup command locally. + +### Use `cargo clean -p` for selective pruning + +Insufficient. Much of the growth comes from many versions and configurations of +workspace-wide dependencies, test harnesses, and build scripts. Package cleanup +does not provide a reliable global budget. + +## Rollout + +1. Extend `scripts/cargo_target_size.py` with explicit measurement semantics, + worktree aggregation, JSON output, thresholds, and fixture-based tests. +2. Add current-worktree and confirmed multi-worktree cleanup recipes. +3. Add warning-only preflights to high-growth `just` recipes. +4. Enable the separate `build-dir` as a compatibility canary on Linux, macOS, + and Windows. Run scoped tests, then request approval for the complete + workspace suite because this changes all Cargo builds. +5. Enable hard-limit failures after one week of warning-only observations. +6. Review the 40/80 GiB and 100/160 GiB defaults after 30 days. +7. Separately benchmark a 10 GiB `sccache`; do not gate the basic storage policy + on that experiment. + +## Success criteria + +- A managed Clanker build cannot silently grow an already over-budget cache. +- `just target-size` shows aggregate linked-worktree target and build usage and + measurement type. +- A contributor can safely reclaim the current worktree's build output with one + command. +- Cleaning other worktrees requires an explicit, reviewable choice. +- Normal scoped test and lint workflows fit below the hard per-worktree limit. +- No linked build directory again reaches 300 GiB without repeated explicit + overrides that are visible in command output.