Summary
Block objects borrowed by a chain compaction are permanently leaked once the compacted generation is later superseded by a self-contained replace. GC never enqueues them and fsck cannot see them.
Found during adversarial acceptance of #392 (independently reproduced twice via a read-only probe over the public publish API); the mechanism is pre-existing (gc.rs / publish.rs are untouched by #392 — FUSE writeback could already trigger it), but workbench_append + workbench_edit make the triggering workflow (deep append chain, then an edit/overwrite) a first-class agent path.
Mechanism
- A staged-session delta chain reaching
COMPACTION_CHAIN_DEPTH = 8 (publish.rs:644-646) re-materializes into a self-contained generation that borrows the old generations' block object keys (metadata-only, no byte copy; service.rs:1108-1112).
- The collapse that follows (
collapse_chain_gc_mutations, gc.rs:284) deletes the entire superseded chain's chunk-manifest records. Borrowed blocks are in the retained set, so they are (correctly) not enqueued — but their ownership records are now gone.
- A later self-contained replace (e.g.
workbench_edit, or any put_file replace=true) collapses again: the borrowed blocks fail owns_block_object_key (gc.rs:241, prefix blocks/{mount}/{inode}/{generation}/) and are skipped as clone-borrowed, while the owner generations' manifests no longer exist — nothing ever enqueues them.
- No backstop: object GC is queue-driven only, fsck scans the opposite direction (fsck.rs:13-15), the ObjectStore trait has no list, and
remove_file carries the same ownership check.
Reproduction (probe)
- Control: 3 appends (below compaction) + replace + GC with no pins → all superseded blocks reclaimed, LEAKED = 0.
- Leak: 9 appends (compaction fires at depth 8) + replace + GC → only the compacted generation's own block is reclaimed; 8 pre-compaction blocks remain unreachable forever; a second GC pass scans 0;
fsck_dangling_blocks reports 0.
Failure mode in production: a long-running workbench where an agent appends a log ≥8 times and later edits/overwrites it leaks that cycle's pre-compaction blocks every compaction cycle — unbounded disk growth invisible to GC and fsck.
Suggested fix directions
- Re-register borrowed blocks under the compacted generation's ownership at compaction time, or
- At collapse time, enqueue keys that are borrowed and whose owner manifest no longer exists, or
- Add an orphan sweeper (requires object listing).
Snapshot safety is unaffected (verified separately: pinned reads across compaction + collapse return the original content).
Summary
Block objects borrowed by a chain compaction are permanently leaked once the compacted generation is later superseded by a self-contained replace. GC never enqueues them and fsck cannot see them.
Found during adversarial acceptance of #392 (independently reproduced twice via a read-only probe over the public publish API); the mechanism is pre-existing (
gc.rs/publish.rsare untouched by #392 — FUSE writeback could already trigger it), but workbench_append + workbench_edit make the triggering workflow (deep append chain, then an edit/overwrite) a first-class agent path.Mechanism
COMPACTION_CHAIN_DEPTH = 8(publish.rs:644-646) re-materializes into a self-contained generation that borrows the old generations' block object keys (metadata-only, no byte copy; service.rs:1108-1112).collapse_chain_gc_mutations, gc.rs:284) deletes the entire superseded chain's chunk-manifest records. Borrowed blocks are in the retained set, so they are (correctly) not enqueued — but their ownership records are now gone.workbench_edit, or anyput_file replace=true) collapses again: the borrowed blocks failowns_block_object_key(gc.rs:241, prefixblocks/{mount}/{inode}/{generation}/) and are skipped as clone-borrowed, while the owner generations' manifests no longer exist — nothing ever enqueues them.remove_filecarries the same ownership check.Reproduction (probe)
fsck_dangling_blocksreports 0.Failure mode in production: a long-running workbench where an agent appends a log ≥8 times and later edits/overwrites it leaks that cycle's pre-compaction blocks every compaction cycle — unbounded disk growth invisible to GC and fsck.
Suggested fix directions
Snapshot safety is unaffected (verified separately: pinned reads across compaction + collapse return the original content).