CAS GC: delete a condemn marker only for the condemnation that owns it

[filimonov]

The CAS GC delete job removed whatever .meta condemn marker was current for a blob. A blob re-published and condemned again between the job's scheduling and its run lost the marker of the later condemnation, so its next graduation ran without durable evidence. Two related holes: the in-process confirmation registry let graduation skip the marker read after a leader restart, and a late publisher PUT Clean If-Match could recreate marker bytes under a stale token.

The bug. X is a blob hash; X.meta is its condemn marker — one object shared by every life of X, and the only thing that tells a writer "do not reuse these bytes, GC is about to delete them". Single GC, no failures:

sequenceDiagram
    participant W as Writer
    participant S as Store<br/>X bytes / X.meta
    participant G as GC
    W->>S: PUT X bytes
    G->>S: round c: X has no referrer, HEAD X present
    G->>S: PUT X.meta = Condemned(c)
    Note right of G: remembers "X confirmed"
    W->>S: reconcile after publish: PUT X.meta = Clean
    Note over S: X.meta = Clean
    Note left of W: transaction aborts, X unreferenced again
    Note over G: round c+1: BUG — graduates X from memory,<br/>never reads X.meta, misses Clean
    W->>S: writer 2: HEAD X present, GET X.meta = Clean
    Note left of W: Clean = safe to reuse: adopts X in a new part
    G->>S: DELETE X bytes
    Note over S: writer 2's part now points at deleted bytes

Loading

The race is GC's HEAD/PUT landing between the writer's body PUT and its reconcile — rare, but it needs no failure (spec §2.3, S2; the TLA+ cfg rv_bug reaches it). Three more schedules end the same way: a marker-delete job queued for an EARLIER life of X that carries no round and runs late, after a leader change, removing the current condemnation's marker (S1, S3); and a writer's late PUT Clean If-Match matching a marker GC re-created with the same bytes and so the same S3 etag (S4). One root cause: nothing on the delete or graduation path names the condemnation it acts for, and graduation trusts process memory instead of the object.

After this PR

• Graduation always reads X.meta and accepts only Condemned at round >= c; Clean, absent or an older round refuses and re-stamps the marker with the current round. The in-memory note is gone. Writer 2 then sees Condemned and re-uploads instead of adopting.
• The delete job carries its condemn round and deletes the marker only while it is Condemned at that round or older (S1, S3).
• The re-stamp uses the current round, so the bytes and the etag differ from what a late PUT was conditioned on (S4).

Changes

• The delete job reads the marker and deletes it with If-Match of the token it read, only while the marker is Condemned at the entry's condemn round or older (deleteConfirmedMeta).
• A gate refusal re-stamps the marker with the current round; the gate accepts a marker Condemned at the entry's round or later (writeCondemnedMeta, settleEntry).
• Graduation always reads the marker; the in-process registry is removed. Reads are issued ahead through the fold read-ahead pool (GraduationReadHints, PriorEdgeCursor::scanAhead, kGraduationLookaheadRows), bounded by cas_gc_io_concurrency; a hint never changes a decision — the reservation prefix stops at a touched row, so a round graduates the same set at every concurrency.
• Docs: garbage-collection.md, blob-protocol.md, debugging.md, monitoring.md, correctness.md, system.md (incl. the upgrade procedure).
• Three probabilistic backoff assertions removed from tests: Retry::backoff draws uniform(0, ceiling), so longestPause > 0 could fail by draw.

Risks / notes

• Every graduation candidate now reads its marker. On the gc-ab stand (S11, 2 runs per cell) fold_reduce per working round went 1063 → 821 ms at cas_gc_io_concurrency 16 (default) and 1623 → 2234 ms at 1.
• Upgrade: all pool nodes should run this build before the first GC round on it; see the procedure in garbage-collection.md. Accepted residuals (publisher-side Clean write, lost gc/state) are listed in the spec docs/superpowers/specs/2026-09-30-cas-gc-meta-delete-names-its-condemnation-design.md §10; the writer rule that stops the publisher from turning Condemned into Clean is a separate PR.
• No on-S3 format change.

Testing

• Unit: 11 new/changed gtests (CASGcMetaWriter*, CASGCReadAhead*, CASBlobInDegree*, CASGcAckFloor*), release CAS* 2591, ASan 2594, TSan green; 12 mutation checks, each caught by a named test.
• TLA+: CaGcCondemnMarkerGate.tla with the round rules — 9 bug cfgs violate the expected invariant, 7 fixed cfgs green, 25.8M-state pm_g1_big_noregistry green.
• gc-ab stand S11 measurements above.

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix the CAS GC deleting the .meta condemn marker of a later condemnation of a re-published blob; graduation now always reads the marker (with read-ahead under cas_gc_io_concurrency) and the delete job removes a marker only for the round that owns it.

Documentation entry for user-facing changes

• Documentation written in this PR (docs/en/antalya/cas/architecture/garbage-collection.md, blob-protocol.md, docs/en/sql-reference/statements/system.md)

CI/CD Options

Regression jobs to run:

• CAS (content-addressed storage; Antalya only)

[github-actions]

Workflow [PR], commit [c4ed6fc]

[k-morozov]

Seems it would be expected behavior to have orphans?

https://github.com/Altinity/ClickHouse/pull/2463/changes#diff-6360f673e1c9dd7e85049209f500e3c56f76cb5cdcf88a29cdeaed488b10215dL1368

[k-morozov]

nit: use cv/barriers instead time for avoiding tests flaps.

[DimensionWieldr]

CI Failures Analysis

Run 36864698169 at c4ed6fc. GitHub shows 50 failed conclusions because each job is reported twice. There are 25 unique failed jobs, plus 5 cancelled at the time limit.

Compared with MasterCI on antalya-26.6: 41824f95 (Oct 2), 61d24118 (Sep 29), 66a2ea5c (Sep 17). This PR is not in those commits. The checks database was not queried.

Related to this PR

None. The diff is CAS GC marker deletion (CasGc.cpp, CasGcMetaWriter.cpp, docs, gtests). It does not touch aggregate functions, vector search, RabbitMQ, or MergeTree settings.

Pre-existing (does not block this PR)

• uniqApacheHLL — 12 CAS aggregate-function jobs, both architectures, plain and s3-cache. UNKNOWN_AGGREGATE_FUNCTION. AggregateFunctionUniq.cpp has no ApacheHLL at the Sep 29 green SHA, at this PR, or at Oct 2 MasterCI. The Sep 29 passing report does not mention the scenario. Oct 2 MasterCI fails the same jobs. The suite is calling a function antalya-26.6 does not implement.
• /selects/final/force/concurrent/* — both cas_selects jobs. NETWORK_ERROR: content-addressed disk cas_disk — mount lease not held. Release cas_selects failed on Sep 17 and Oct 2 MasterCI and passed on Sep 29.
• /alter/attach partition/part 3/* — aarch64 cas_alter_attach_3. Same mount-lease NETWORK_ERROR. That job failed on Sep 29 MasterCI (21 scenarios) and the release twin failed on Oct 2.
• Lightweight delete — release cas_lightweight_delete_4 and cas_s3_cache_lightweight_delete_3. Concurrent delete left 251 rows against an expected 250. Immediate removal took 20.42s against a 20s limit, and the row count matched. The same suite failed on Sep 17 and Sep 29 MasterCI and the release job passed on Oct 2.
• 02354_vector_search_rescoring_distance_in_select_list — arm asan/ubsan azure. Failed 3 of 3 reruns in this job, and failed on all 3 MasterCI runs, labeled reproducible.
• 04493_create_or_replace_materialized_view_populate_keeps_subscription_concurrent — memory limit exceeded. The job's own reruns passed 2 of 2.
• 04023_rollup_totals_memory_limit — reproducible only with randomized MergeTree settings; 74 of 74 passed without them. The same job passed on Oct 2 MasterCI (0 failed / 11116).
• 01169_old_alter_partition_isolation_stress — sanitizer timeout (927s). Known on Oct 2 MasterCI as "Random timeout with sanitizer". The harness then killed the server with signal 15, exit 143. No <Fatal> and no OOM in that job. Server died and Scraping system tables are the cascade.

Infrastructure (does not block this PR)

• test_storage_rabbitmq/...::test_rabbitmq_sharding_between_queues_publish — no RabbitMQ consumption within 60s. The same integration shard passed 0/982 on Oct 2 MasterCI. 1 of 1 on this PR.
• Grype — keeper image, 4 high/critical; alpine server image, 1. Same counts on Oct 2 MasterCI (CVE-2026-85091 zlib, CVE-2026-84782 libcrypto3).
• 5 cancelled jobs ran to the time limit and uploaded no failure: stateless tsan cas-s3 1/2 and 2/2, stateless msan cas-s3 2/3 and 3/3 (~6h), and release cas_s3_cache_alter_attach_3 (~5h).

Unknown (blocks until resolved)

• Integration tests (amd_msan, 4/10). Ended in 13 minutes on github-runner-36864698169-110450381019-dedicated.32g.x86. The Run step has no conclusion, S3 has no artifacts, and the GitHub log blob is missing. The same shard passed 0/1203 on Oct 2 MasterCI. A rerun is what resolves it.

Recommendations

1. Rerun Integration tests (amd_msan, 4/10).
2. Nothing else here needs a change in this PR. The uniqApacheHLL failures are a suite-versus-branch mismatch on antalya-26.6, visible on MasterCI without this commit.

[DimensionWieldr]

Integration rerun

Both shards that failed on the first attempt passed on the rerun. They ran the full suites.

• Integration tests (amd_msan, 4/10) — OK, 0/1203, about 2.5h (18:49–21:19 UTC). This was the unknown job. The first attempt died in 13 minutes with no log. infrastructure, passes on rerun. It no longer blocks.
• Integration tests (amd_asan_ubsan, db disk, old analyzer, 5/8) — OK, 0/982. The RabbitMQ consume timeout did not recur.

Nothing in this rerun is a regression from the CAS GC change.