CAS GC: delete a condemn marker only for the condemnation that owns it

docs/en/antalya/cas/architecture/blob-protocol.md

@@ -143,15 +143,15 @@ sequenceDiagram
     end
 
     Note over GC: round n+1 -- graduation, only if still zero
-    GC->>S3: re-verify in-degree, requires confirmed durable Condemned evidence for hash+token t1
-    Note over GC: publishes delete_pending
+    GC->>S3: re-verify in-degree, then GET .meta
+    Note over GC: Condemned at round n or newer -- publishes delete_pending
 
     Note over GC: round n+2 -- the single content-delete site
     GC->>S3: deleteExact(blob, t1)
     alt writer republished
         S3-->>GC: TokenMismatch -- nothing deleted, blob is live at t2
     else genuinely dead
-        S3-->>GC: Deleted -- then drop the .meta
+        S3-->>GC: Deleted -- then drop the .meta only while it is still Condemned at round n or older
     end
 ```
 
@@ -170,9 +170,13 @@ Why this closes the race in both directions:
   `TokenMismatch` and reclaims nothing — the delete names an exact incarnation, never "the object
   at this key".
 - The delete lags condemnation by at least two full rounds, and publishing the one edge that
-  authorizes an irreversible delete requires confirmed durable `Condemned` evidence for that
-  exact `(hash, token)` pair. Without it `GC` never throws — it carries the entry and retries the
-  marker write on the next round.
+  authorizes an irreversible delete requires reading the blob's `.meta` and finding it `Condemned`
+  at the entry's round or newer. A `Clean`, absent, older-round or unreadable marker never
+  graduates the entry: `GC` does not throw, it carries the entry and rewrites the marker at the
+  current round.
+- The delete job removes the `.meta` only while it is still `Condemned` at the entry's round or
+  older, and conditionally on the version it read. A marker rewritten `Clean` by a writer, or
+  condemned again at a later round, stays.
 
 Both directions degrade to a spurious re-upload or a no-op delete. Neither can lose data or leave
 a dangling manifest entry.

docs/en/antalya/cas/architecture/correctness.md

@@ -35,7 +35,7 @@ The full model index (source `.tla` files and proof-run records) lives at
 | `CaGcAckFloorZombie.tla` | `INV_NO_DANGLE` under two fully-interleaved GC leaders | `sab_eagerdelete`: a leader deleting its own fresh (not-yet-pending) graduations — the pre-amendment single-phase behavior — dangles when a deposed leader's pass overlaps a live one |
 | `CaGcRoundDeferCore.tla` | `NoOverDelete` — a deferred round may skip a rebuild only when nothing destructive is pending | `sab_graduate_on_stale`: dropping the "an unfolded delta covers this blob" guard lets a deferred round delete a blob its own unread history still protects |
 | `CaEdgeBeforeObserve.tla` | The writer/`GC` publish order is safe to simplify | `sab_late_edge`: allowing adoption before the precommit closure is durable (the pre-fix order) dangles |
-| `CaGcCondemnMarkerGate.tla` | Graduation requires confirmed durable `Condemned` evidence | Swallowing a failed asynchronous condemn-marker write let a writer adopt a token a later graduation was about to delete |
+| `CaGcCondemnMarkerGate.tla` | Graduation reads the marker and requires `Condemned` at the entry's round or later | Swallowing a failed asynchronous condemn-marker write let a writer adopt a token a later graduation was about to delete |
 | `CaRefTableSnapshotLogCore.tla` | Dense, per-life ref ids with an in-band `_ckpt` recovery frontier | `sab_scanistruth`: trusting a listing as the source of truth for "acked" reproduces the real production incident where a `LIST` omitted two already-durable, already-acknowledged ref entries |
 
 ## Soak and chaos: the empirical oracle {#soak-and-chaos}

docs/en/antalya/cas/architecture/garbage-collection.md

@@ -340,7 +340,8 @@ Recomputes the per-shard in-degree snapshot and computes the round's single dest
 
 - **Runs on:** fold path only
 - **Reads:** streaming `GET` of each referenced parent run segment; one `HEAD` per zero-in-degree
-  candidate; one `.meta` `GET` per graduation candidate lacking in-process marker confirmation. When
+  candidate; one `.meta` `GET` per graduation candidate the round's graduation budget admits, read ahead
+  when `cas_gc_io_concurrency` is above 1. When
   orphan-sweep planning runs: a `LIST` page of `cas/manifests/`, a `GET` per candidate, plus
   `gc/state`, the adopted seal, the catalog, and per-namespace `_ckpt` / tail `_log`.
 - **Writes / deletes:** one `PUT` per rewritten run segment; schedules the async `.meta` condemn
@@ -673,24 +674,31 @@ flowchart LR
     B --> C["retired with condemn_round = n"]
     C --> D{"round n+1: re-verify"}
     D -->|"in-degree recovered"| S["SPARED -- recovery wins, even past the floor"]
-    D -->|"still zero, confirmed durable Condemned evidence for hash and t"| G["GRADUATED -- delete_pending"]
-    D -->|"still zero, evidence unconfirmed"| C2["carried unchanged, retry the marker, never throw"]
+    D -->|"still zero, reads .meta: Condemned at round n or newer"| G["GRADUATED -- delete_pending"]
+    D -->|"still zero, .meta absent, Clean, older or unreadable"| C2["carried, .meta rewritten at this round, never throw"]
     D -->|"current token not equal to t"| SUP["SUPERSEDED -- a writer resurrected, re-condemn the CURRENT token"]
     G --> E["round n+2, pre-CAS: exact-token DELETE of blob at t"]
-    E -->|"Deleted or Absent"| F["then drop the .meta"]
+    E -->|"Deleted or Absent"| F["then drop the .meta if it is still Condemned at round n or older"]
     E -->|TokenMismatch| H["nothing deleted -- live at a newer token, leave the .meta alone"]
 ```
 
-The `.meta` sidecar carries **no token** — it is a per-hash hint. The exact incarnation token lives
-in the condemned sentinel row inside the run, together with the condemn round and two flags,
-`delete_pending` and `marker_confirmed`. `GC`'s marker is add-only: `Clean → Condemned` yes, the
-reverse never, not even when sparing — only a writer that has already displaced the body may clear
-it. A blob whose in-degree reaches zero in round `n` is retired with `condemn_round = n`; it can
-graduate to `delete_pending` in round `n+1` at the earliest and be deleted in round `n+2`, so a
-minimum of two full rounds separate condemnation from deletion. `delete_pending` is never cleared in
-place, but it authorizes a delete only while in-degree stays zero: a fresh edge folded in a later
-round spares the entry and removes it from the retired pipeline (recovery wins, even past the
-floor).
+The `.meta` sidecar carries **no token** — it is a per-hash hint. The exact incarnation token lives in the
+condemned sentinel row inside the run, together with the condemn round and two flags, `delete_pending` and
+`marker_confirmed`; `GC` writes the second with the first and never reads it. Every `GC` marker write stamps the
+round that issues it and replaces an absent marker, `Clean`, or an older-round `Condemned`; `GC` never writes
+`Clean`, not even when sparing — only a writer that has already displaced the body clears it.
+
+Graduation reads the marker in the round that graduates and accepts only `Condemned` at the entry's round or
+newer: an older round may belong to another incarnation of the same content. The marker delete after a redelete
+removes only `Condemned` at the entry's round or older, conditional on the version it read, so a delayed delete
+never removes the marker of a later incarnation. It can leave an orphan `Condemned` marker over an absent body;
+the next writer of that hash republishes and clears it, and `cas-fsck` counts it in `meta_without_body`.
+
+A blob whose in-degree reaches zero in round `n` is retired with `condemn_round = n`; it can graduate to
+`delete_pending` in round `n+1` at the earliest and be deleted in round `n+2`, so a minimum of two full rounds
+separate condemnation from deletion. `delete_pending` is never cleared in place, but it authorizes a delete only
+while in-degree stays zero: a fresh edge folded in a later round spares the entry and removes it from the retired
+pipeline (recovery wins, even past the floor).
 
 ## Sharding {#sharding}
 
@@ -742,6 +750,8 @@ logs:
 | `GET` manifests | 1 per folded owner (manifest) edge — a manifest emits many blob edges but is read once per edge event; no manifest-body cache within a round |
 | `PUT` run segments | 1 per non-pure-carry shard, plus 1 fold seal |
 | `HEAD` blobs | 1 per newly condemned |
+| `GET` blob `.meta` at graduation | 1 per entry the graduation budget admits, plus 1 `GET` and 1 `PUT` per refusal (the rewrite job re-reads the marker before its conditional `PUT`) |
+| `GET` + conditional `DELETE` blob `.meta` after a redelete | 1 `GET` per `Deleted` or `Absent` outcome; the `DELETE` only while the marker is `Condemned` at the entry's round or older |
 | Blob `HEAD` + conditional `DELETE` | 1 `HEAD` per `redelete` entry — an entry that graduated in an *earlier* round, not the current one — up to `cas_gc_round_redelete_budget`; a `DELETE` only when the body is present at the condemned token |
 | Successful lease `CAS gc/state` | 1 |
 | Commit `CAS gc/state` | 1 |
@@ -781,7 +791,7 @@ the chunk size reject `0`:
 | `cas_gc_round_sweep_recovery_op_budget` | 5000 | committed-tail ref-log reads the sweep's recovery walk may spend (phase 9) |
 | `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for write-once families (phases 15, 17); `1` to `1000` |
 | `cas_gc_meta_pool_size` | 16 | bounded pool for condemn-marker writes (phase 12) |
-| `cas_gc_io_concurrency` | 16 | bounded pool for the fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate `HEAD`s (phases 8, 9), the orphan-sweep planning reads (phase 9), the rebuild read-ahead and the `pending_deletes` `HEAD` + conditional `DELETE` fan-out (phase 11); other GC requests run on the round thread; `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead |
+| `cas_gc_io_concurrency` | 16 | bounded pool for the fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate `HEAD`s (phases 8, 9), the graduation `.meta` reads (phase 9), the orphan-sweep planning reads (phase 9), the rebuild read-ahead and the `pending_deletes` `HEAD` + conditional `DELETE` fan-out (phase 11); other GC requests run on the round thread; `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead |
 
 The fold-batching controls `gc_fold_threshold` (default 1), `gc_fold_max_defer_rounds` (default 8)
 and `gc_frontier_probe_budget` (default unbounded) are internal `PoolConfig` fields with no disk
@@ -877,7 +887,7 @@ No writes.
 |---|---|---:|
 | referenced parent run segments | streaming `GET` | one per referenced run |
 | `<pool_prefix>/blobs/...` | `HEAD` | one per zero-in-degree candidate, plus one peek per carried entry that reached zero again |
-| blob `.meta` | `GET` | one per graduation candidate with no in-process marker confirmation |
+| blob `.meta` | `GET` | one per graduation candidate the graduation budget admits |
 | new run segments | `PUT` | one per written run |
 | `<pool_prefix>/cas/manifests/` | `LIST` | one bounded page, only when orphan planning runs |
 | manifest candidate body | `GET` | one per nominated candidate (≤ `cas_manifest_sweep_delete_budget_keys`), through the read-ahead; keys decided from their name alone are never read; only when orphan planning runs |

docs/en/antalya/cas/operations/debugging.md

@@ -242,7 +242,7 @@ mount.
 | `cas-fsck [--detail] [--timeout N] [--namespace PREFIX] [--partial]` | The same reachability scan as `SYSTEM CAS FSCK`, offline. `--detail` adds a per-object `<class>\t<key>\t<size>` listing (`reachable`, `dangling`, `unreachable`, `pending-gc`, `awaiting-gc`, `unaccounted`, `stale-edge`, `corrupted-run`, `chain-broken`, `unchecked`, `lifeless-key`, `janitor-pending`) — the only way to get per-object, not just per-pool, findings. `--timeout`/`--partial` bound a scan on a large pool |
 | `cas-gc-dryrun` | Previews the next round's deletes, read-only, no lease. Over-reports away from quiescence (does not fold new owner events) — a diagnostic only, never a delete source |
 | `cas-inspect '<raw-object-key>'` | Decodes one raw object-storage key (as printed by `cas-fsck`/`cas-gc-dryrun`) straight to JSON |
-| `cas-gc-rebuild [--force]` | Disaster recovery: rebuilds a `gc/state` baseline from raw owner state after the GC guard has refused every regular round. `--force` bypasses only the healthy-state refusal, never a competing leader or a failed `CAS`. See [`SYSTEM CAS GC REBUILD`](/sql-reference/statements/system#system-cas-gc-rebuild) for the destructive-tool caveats |
+| `cas-gc-rebuild [--force]` | Disaster recovery: rebuilds a `gc/state` baseline from raw owner state after the GC guard has refused every regular round. `--force` bypasses only the healthy-state refusal, never a competing leader or a failed `CAS`. See [`SYSTEM CAS GC REBUILD`](/sql-reference/statements/system#system-cas-gc-rebuild) for the destructive-tool caveats. Before rebuilding a lost `gc/state`, stop or restart every server of the pool; a request sent before a server stopped can still reach the object store afterwards, and this rule does not cover it. |
 
 ```bash
 clickhouse-disks -C config.xml --disk cas cas-fsck --detail
@@ -255,6 +255,19 @@ clickhouse-disks -C config.xml --disk cas cas-gc-rebuild --force
 [migration page](/antalya/cas/operations/migration#decommission) alongside the SQL form, since
 decommissioning a pool member is a migration/scale-down operation, not an incident-time tool.
 
+## Upgrading a pool with GC enabled {#upgrade-gc}
+
+`cas_gc_enabled` is read once, when the disk is constructed: only a restart applies a change.
+
+1. On every server still on the old version, set `<cas_gc_enabled>false</cas_gc_enabled>` in each CAS disk's
+   configuration and restart the server. It then runs no GC and refuses `SYSTEM CAS GC RUN`, `START` and `REBUILD`.
+2. Only after step 1 has finished on every server, upgrade the servers, removing the setting on each upgraded one.
+3. Never run `SYSTEM CAS GC` commands on, or re-enable GC on, a server still on the old version.
+
+A delete request sent before a restart can still reach the object store later and remove a marker GC relies on,
+so a blob can be lost; nothing bounds when such requests stop. Entries the old version already scheduled for
+deletion are deleted by the upgraded GC without a fresh marker read. Downgrades follow the same procedure.
+
 ## The CLICKHOUSE_USER_FILES gotcha when reproducing a test manually {#user-files-gotcha}
 
 Running a `CAS` stateless test directly with `tests/clickhouse-test` against a manually started

docs/en/antalya/cas/operations/monitoring.md

@@ -43,7 +43,7 @@ subset for a first health pass; the full list groups by object class (`CASBlob*`
 | `CASRefAppendSealRejected` | Occasional (a deposed writer losing a race is the protocol working); sustained growth is not | A writer keeps retrying after losing its mount and does not yet know it |
 | `CASGCHeartbeatFenceOuts` | Zero on a healthy pool | GC fenced an expired mount; check `system.cas_mounts` for a member that should have cleanly unmounted |
 | `CASGCUnmatchedRemoveDeltas` | Occasional (benign per-key no-op by design) | A persistent nonzero rate means removal deltas are reaching the reducer without their matching activation — a correctness signal worth a look, not an automatic false deletion |
-| `CASGCCondemnMarkerUnconfirmedCarry` | Zero | A durable condemn marker could not be confirmed; deletion is safely postponed but investigate marker write/read failures |
+| `CASGCCondemnMarkerUnconfirmedCarry` | Zero | The graduation read found no `Condemned` marker at the entry's round or later (absent, `Clean`, older round, or unreadable); deletion is postponed and the marker is rewritten, but investigate repeated carries |
 | `CASGCMetaWriteAnomaly` | Zero | The bounded GC metadata pool failed an operation; backend or pool pressure may delay metadata convergence |
 | `CASRefRollbackBestEffortDropFailed` | Zero | A rollback cleanup drop hit a backend failure; refs may remain live and GC may be delayed on that namespace |
 

docs/en/sql-reference/statements/system.md

@@ -517,6 +517,9 @@ baseline — on a pool that has ever completed a GC round this means the object
 `adopted_seal_generation` names which generation's fold seal the rebuild carried holds from; `0`
 when it carried none.
 
+Before rebuilding a lost `gc/state`, stop or restart every server of the pool. A request a server sent before it
+stopped can still reach the object store afterwards; this rule does not cover it.
+
 ### SYSTEM CAS GC STOP / SYSTEM CAS GC START {#system-cas-gc-stop-start}
 
 Pause or resume the background GC scheduler on one content-addressed disk, without affecting reads

src/Common/ProfileEvents.cpp

@@ -912,7 +912,7 @@ The server successfully detected this situation and will download merged part fr
     M(CASGCRetiredRedeleted,   "Number of CAS GC pending deletes executed with an exact object token. Growth indicates physical cleanup activity.", ValueType::Number) \
     M(CASGCRetiredRedeleteFailed, "Number of CAS GC pending blob deletes whose HEAD or exact-token DELETE failed. A non-zero value indicates object storage errors; the affected blobs are retried in the next round.", ValueType::Number) \
     M(CASGCUnmatchedRemoveDeltas, "Number of CAS GC in-degree removal deltas that matched no existing source edge. The in-degree model is a set, not a counter, so this is a per-key no-op by design and never causes a false deletion — but a persistent nonzero rate means removal deltas are reaching the reducer without their matching activation, which is a correctness signal.", ValueType::Number) \
-    M(CASGCCondemnMarkerUnconfirmedCarry, "Number of CAS GC retirements delayed because a durable condemn marker could not be confirmed. A non-zero value indicates marker write or read failures and safely postpone deletion.", ValueType::Number) \
+    M(CASGCCondemnMarkerUnconfirmedCarry, "Number of CAS GC retirements delayed because the condemn marker did not confirm: it could not be read or written, it was a readable `Clean`, or it was `Condemned` at an older round than the entry. Deletion is postponed safely.", ValueType::Number) \
     M(CASGCHeartbeatFenceOuts, "Number of CAS GC operations fenced out for expired mounts. A non-zero value indicates stale mounts or heartbeat delays.", ValueType::Number) \
     M(CASGCMetaWriteAnomaly, "Number of CAS GC metadata operations that failed on the bounded metadata pool. A non-zero value indicates backend or pool pressure and may delay metadata convergence.", ValueType::Number) \
     M(CASMetaPut,          "Number of conditional CAS blob-metadata create attempts, including lost races. Growth indicates metadata reconciliation or write contention.", ValueType::Number) \

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasRecordStreamFormat.h

@@ -90,7 +90,7 @@ struct SourceEdgeRecord
     PersistedEtag token{};
     uint64_t size = 0;
     uint64_t condemn_round = 0;
-    bool marker_confirmed = false;   /// durable Condemned meta confirmed for this entry (graduation gate)
+    bool marker_confirmed = false;   /// written with `delete_pending`; the graduation gate reads the marker, not this bit
 };
 
 /// The header-line `kind` word for the only live `cas_run` kind.