CAS: parallel delete blobs

[k-morozov]

Bench results:

   cas_gc_io_concurrency    pending_deletes       Throughput    Active GC time    Full drain time
  ━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━
                       1            33.99 s      247 blobs/s           34.18 s             88.8 s
  ───────────────────────  ─────────────────  ───────────────  ────────────────  ─────────────────
                      16             2.90 s    2,900 blobs/s            3.10 s             53.3 s

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added cas_gc_io_concurrency to run blob deletes in the CAS GC pending_deletes phase in parallel.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [a2f53e5]

[k-morozov]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T11:40:35.803992Z	8e0ec8d	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e0ec8de87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Batch re-deletes across shard boundaries

When cas_gc_shards > 1, this invocation completes and waits for one shard before the loop reaches the next shard, so re-deletes from different shards never overlap. If each shard has fewer than cas_gc_redelete_min_batch_size pending entries, the entire phase remains sequential despite cas_gc_redelete_concurrency > 1; even larger per-shard batches cannot use more concurrency than one shard contains. Schedule the phase's eligible entries before a single wait so the configured fan-out applies across shards.

Useful? React with 👍 / 👎.

[k-morozov]

Default value gc_shards = 1 . Yes, in some cases we could obtain seq execution. But in these cases we don;t have a lot blobs. I suggest skip this impovment now.

[chatgpt-codex-connector]

Account for worker I/O in pending-delete metrics

When parallel re-delete is enabled, performRedeleteIo executes the blob HEAD and conditional DELETE on these pool threads, while GcPhaseTimer snapshots only the round thread's ProfileEvents. As a result, the pending_deletes row in system.cas_gc_log omits the underlying request and retry counters precisely when cas_gc_redelete_concurrency > 1, breaking per-phase operation accounting; collect the worker deltas or expose equivalent explicit phase metrics.

Useful? React with 👍 / 👎.

[k-morozov]

This is a regression only when the feature is enabled. It is a known limitation that read-ahead also has, and it will be fixed in a separate PR for both places at once.

[filimonov]

doc combining my + AI review https://gist.github.com/filimonov/a5377e47bf29f92fe12b345d10fe29ad

spec + plan to rework (you can pass it to AI agent) https://gist.github.com/filimonov/eb2ef24fbc8a49e2d3cbb43fe43795d8

[alsugiliazova]

PR #2351 distributed-systems audit

AI audit note: This review was generated by AI (Claude Opus 5.5) and manually rechecked against the merged head a2f53e54.

Verdict: two Medium operational defects. No data-safety defect found.

1. GC stop can block for hours during an object-store outage

Gc::redeleteBlobs deliberately continues after per-entry failures so every failure is logged and counted. However, every queued entry gets a separate 90-second retry window, and CasGcScheduler::stop() joins the round without cancellation or a round-level deadline.

With the default 5,000-entry budget, a persistent outage can keep one round busy for about 7.8 hours at concurrency 16, or 125 hours at concurrency 1. SYSTEM ... GC STOP waits for that work to finish.

Fix direction: add cancellation or one shared round deadline, while preserving per-entry accounting for work that already started.

2. A released setting is removed without an alias

cas_gc_read_concurrency shipped in the non-prerelease v26.6.4.20001.altinityantalya. This PR replaces it with cas_gc_io_concurrency without a compatibility alias:

• cas_gc_read_concurrency now fails disk loading with UNKNOWN_SETTING.
• gc_read_concurrency, previously accepted as the legacy spelling, is silently ignored and falls back to concurrency 16.

This can prevent a node from starting or silently increase GC object-store load during an upgrade.

Fix direction: accept both old spellings as deprecated aliases for one release cycle, or at minimum reject both consistently.

Review method: static distributed-systems audit of the final PR code and tests.

[alsugiliazova]

PR #2351 CI Verification Report

Verification (2026-09-24)

Field	Value
PR	Altinity/ClickHouse#2351 — CAS: parallel delete blobs
Base	antalya-26.6
Head	3a13b956ae19153434eea3e31e375639719204be
Run	35341493723 (4 attempts, 2026-09-18 → 2026-09-21)
CI report	ci_run_report.html
Method	pr-ci-failure-triage + DB rates (gh-data.checks, gh-data.clickhouse_regression_results) + artifact and server-log reads

Verdict

No failure is caused by this PR. CI can be approved. Rebase onto antalya-26.6 to pick up #2406, which fixes the two integration shards.

9 checks fail, plus the aggregate PR status:

Category	Checks	Blocks #2351?
regression	0	—
unknown	0	—
infrastructure	4 — Grype keeper, Grype server-alpine, Integration amd_tsan 5/6, Integration amd_asan_ubsan, db disk, old analyzer, 1/8	No
pre-existing-flaky	5 — Stateless amd_tsan, cas s3 storage, parallel, 2/2, cas_alter_attach_3, cas_s3_cache_alter_attach_3, cas_selects, cas_lightweight_delete_4	No
cascade	Parent features and modules in the regression reports fail only because a scenario under them failed	—

All failing jobs failed the same way on every attempt. Rerunning them won't turn them green.

Still red on this run — categorized

Cluster	Category	Blocks #2351?	Notes
Grype clickhouse-keeper, clickhouse-server-alpine	infrastructure	No	CVE-2026-85091 (High) in the Alpine package zlib 1.3.2-r0 in the base image. The same two scans fail on #2427, #2420, #2419 and #2414. Only #2398 passes, and it was scanned before the CVE was published.
Integration amd_tsan 5/6, amd_asan_ubsan, db disk, old analyzer, 1/8	infrastructure	No — rebase	Failed to pre-pull Docker images, caused by pull access denied for minio/minio, minio/mc: repository does not exist. Failed identically on attempts 1–3. #2406 (merged 2026-09-21) moves the images to clickhouse/minio-* mirrors. This branch forked on 2026-09-18, before that merge.
Stateless amd_tsan, cas s3 storage, parallel, 2/2	pre-existing-flaky	No	See Stateless tsan CAS shard.
cas_alter_attach_3, cas_s3_cache_alter_attach_3	pre-existing-flaky	No	See Alter attach part 3.
cas_selects	pre-existing-flaky	No	See cas_selects.
cas_lightweight_delete_4	pre-existing-flaky	No — watch after rebase	See cas_lightweight_delete_4. Weakest verdict in this report.

Pre-existing flaky — evidence

Rates are runs failed / runs over the last 45 days on 26.6, x86_64. "Branch" means antalya-26.6 MasterCI runs, "other PRs" means every other PR run, and "this PR" covers all four commits of #2351.

Stateless tsan CAS shard

Field	Finding
Failing tests	03156_tuple_map_low_cardinality, 02833_concurrent_sessions
Cause	The test logic passed. Both failed only because the server printed ConnectionGroup: Too many active sessions in group Disk, count 8200, warning limit 8000 into the test's stderr.
Warning rate (tsan CAS stateless runs)	Branch 3/40, other PRs 2/82, this PR 2/18
Link to the PR	The PR adds at most 16 concurrent GC requests, against an 8000-session warning threshold.

Alter attach part 3

Field	Finding
Errors	CAS write could not be committed ... persistent CAS contention, the checkpoint contribution was not published and ... gave up at the policy deadline (NETWORK_ERROR)
Mechanism	Every failing scenario falls inside a two-minute burst of 503 Service Unavailable from RustFS (16:29–16:30 server time, 14:27–14:31 UTC in the test log). None of the three nodes logged a failed GC delete (pending delete of blob ... failed).
History	Both signatures appear on the branch and on 21 other PR runs, and predate this PR (first seen 2026-09-05).
Feature-level rates	Branch about 20–30%, other PRs about 40%, this PR 20–60% (5 runs)

Suite	Branch	Other PRs	This PR
cas_alter_attach_3	4/6	16/27	1/2
cas_s3_cache_alter_attach_3	1/5	12/26	2/2

cas_selects

Field	Finding
Error	content-addressed disk 'cas_disk' -- mount lease not held ... TRANSIENT unavailability (NETWORK_ERROR)
Mechanism	About 76 concurrent JOIN queries used up the server's local ports to MinIO (Cannot assign requested address, errno 99, starting 11:09:36). Mount-lease renewal then failed 65 times within 4 seconds, and the disk stopped accepting operations.
Link to the PR	GC's pending-delete path wasn't active at the time. Only orphan-manifest sweeps ran, and they used the same thread pool before this PR.
Suite rate	Branch 4/5, other PRs 28/32, this PR 3/3
History	Signature seen since 2026-08-20, on 9 branch runs and 42 other-PR runs. The same class is tracked for #2300 as #2332.

cas_lightweight_delete_4

Field	Finding
Failing scenario	/lightweight delete/concurrent delete/MergeTree/random delete half of the table without overlap
Assertion	SELECT count() returned 512 where 500 was expected, so 12 rows were not deleted.
Scenario rate	Branch 3/15 (510, 503 and 501 rows left), other PRs 1/32, this PR 3/4
Family rate (all "without overlap" scenarios)	Branch about 12%, this PR about 19% (small sample)
Link to the PR	Leftover rows mean a delete didn't take effect. GC only removes blobs nothing references, so it can't bring deleted rows back.
Caveat	Weakest verdict here. If it keeps failing more often than on the branch after a rebase, investigate it on its own.

Code-review notes (not CI)

These don't affect the CI verdict.

• The changelog entry names cas_gc_redelete_concurrency, but the code adds cas_gc_io_concurrency.
• cas_gc_read_concurrency shipped in v26.6.4.20001.altinityantalya and is removed with no alias.
  • A config that still sets it fails with UNKNOWN_SETTING when the disk loads.
  • The unprefixed spelling gc_read_concurrency is ignored with no error.
• Pending deletes now run 16 at a time by default, and a failed delete no longer stops the batch.

Recommendations

1. Rebase onto antalya-26.6 to pick up #2406, then rerun the integration shards.
2. Track the Grype CVE on the base image separately. It affects every 26.6 PR.
3. After the rebase, check that cas_lightweight_delete_4 no longer fails more often here than on the branch.