Skip to content

[BUG] GC permanently leaks pre-compaction blocks when a compacted append chain is later replaced #393

Description

@wchwawa

Summary

Block objects borrowed by a chain compaction are permanently leaked once the compacted generation is later superseded by a self-contained replace. GC never enqueues them and fsck cannot see them.

Found during adversarial acceptance of #392 (independently reproduced twice via a read-only probe over the public publish API); the mechanism is pre-existing (gc.rs / publish.rs are untouched by #392 — FUSE writeback could already trigger it), but workbench_append + workbench_edit make the triggering workflow (deep append chain, then an edit/overwrite) a first-class agent path.

Mechanism

  1. A staged-session delta chain reaching COMPACTION_CHAIN_DEPTH = 8 (publish.rs:644-646) re-materializes into a self-contained generation that borrows the old generations' block object keys (metadata-only, no byte copy; service.rs:1108-1112).
  2. The collapse that follows (collapse_chain_gc_mutations, gc.rs:284) deletes the entire superseded chain's chunk-manifest records. Borrowed blocks are in the retained set, so they are (correctly) not enqueued — but their ownership records are now gone.
  3. A later self-contained replace (e.g. workbench_edit, or any put_file replace=true) collapses again: the borrowed blocks fail owns_block_object_key (gc.rs:241, prefix blocks/{mount}/{inode}/{generation}/) and are skipped as clone-borrowed, while the owner generations' manifests no longer exist — nothing ever enqueues them.
  4. No backstop: object GC is queue-driven only, fsck scans the opposite direction (fsck.rs:13-15), the ObjectStore trait has no list, and remove_file carries the same ownership check.

Reproduction (probe)

  • Control: 3 appends (below compaction) + replace + GC with no pins → all superseded blocks reclaimed, LEAKED = 0.
  • Leak: 9 appends (compaction fires at depth 8) + replace + GC → only the compacted generation's own block is reclaimed; 8 pre-compaction blocks remain unreachable forever; a second GC pass scans 0; fsck_dangling_blocks reports 0.

Failure mode in production: a long-running workbench where an agent appends a log ≥8 times and later edits/overwrites it leaks that cycle's pre-compaction blocks every compaction cycle — unbounded disk growth invisible to GC and fsck.

Suggested fix directions

  • Re-register borrowed blocks under the compacted generation's ownership at compaction time, or
  • At collapse time, enqueue keys that are borrowed and whose owner manifest no longer exists, or
  • Add an orphan sweeper (requires object listing).

Snapshot safety is unaffected (verified separately: pinned reads across compaction + collapse return the original content).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions