[WIP] CAS draft (adopting to CI/CD, not for review / merge) by filimonov · Pull Request #2073 · Altinity/ClickHouse

filimonov · 2026-07-17T17:36:22Z

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

content addressable storage - draft PR

Documentation entry for user-facing changes

TBD.

Exclude tests:

Regression jobs to run:

github-actions · 2026-07-17T17:37:47Z

Workflow [PR], commit [81642c8]

…ent_addressed stateless lanes) Config Workflow check failed with 'Workflows are outdated' for master.yml, pull_request.yml, pull_request_community.yml, release_builds.yml. Regenerated via 'python3 -m praktika yaml'. The regeneration adds the two CAS stateless jobs to the generated workflows: 'Stateless tests (arm_binary, content_addressed storage, parallel)' and 'Stateless tests (arm_binary, content_addressed s3 storage, parallel)' (the rustfs-backed lane). CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=927ea142c9cb14759623861eb004261d0b4b1c8f&name_0=PR&name_1=Config+Workflow PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…teless lane The lane's start_rustfs expected a pre-extracted binary at ci/tmp/rustfs and failed on CI runners where nothing provisions it (the workflow wipes ci/tmp on every run). Download the static musl build for the runner architecture from the RustFS GitHub release (1.0.0-beta.9) when the binary is absent, mirroring how setup_minio.sh downloads minio/mc. Validated locally: the beta.9 binary passes the conditional-operation semantics the CA pool requires (second 'If-None-Match: *' PUT -> 412, wrong-etag conditional DELETE -> 412, right-etag DELETE succeeds), and download_rustfs provisions an executable binary end-to-end. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Fast test fails at cmake generation: 'Target "dbms" links to ch_contrib::crc32c but the target was not found' — the fast-test job initializes a limited submodule list that does not include contrib/crc32c, so the unconditional add_contrib is skipped while the dbms link line still references the target. The dependency is dead: it was wired in for per-block CRC32C in the early CAS run-file format (5f1272c), which was later replaced by the text record-stream codecs; no source file includes the library today. Restore the pre-CAS state: crc32c is built only for google-cloud-cpp, and dbms does not link it. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=835251f81cb5af73ad9eaa3a835f50f0c8b678db&name_0=PR&name_1=Fast+test PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Fast test builds without SSL and failed on the unconditional 'openssl/evp.h' include in CasBlobHashingWriteBuffer.cpp. Wrap the OpenSSL-backed Sha256 hashing write buffer and the one-shot digest in '#if USE_SSL'; on non-SSL builds selecting blob_hash = 'sha256' now fails closed with SUPPORT_IS_DISABLED. CityHash128 and XXH3-128 blob hashes are unaffected. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

… test regression) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Fast+test PR: #2073 A CAS parser commit grouped `RELOAD_DICTIONARY`/`RELOAD_MODEL`/ `RELOAD_FUNCTION` with `CONTENT_ADDRESSED_GARBAGE_COLLECTION` into a format case that prints only the optional disk, dropping the reload targets: `SYSTEM RELOAD MODEL my_model` formatted as `SYSTEM RELOAD MODEL` (failed 04117_parser_system_query_variants and 04124_parser_system_query_extra in Fast test). Fold all four types back into the generic target-printing case (table / target_model / target_function / disk else-if chain) — for the CA GC command the disk branch produces the identical output. Both stateless tests verified locally via clickhouse-local against their references. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…als (arm_tidy, T13 batch 1) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Removes default arguments from all virtual/override methods flagged by `google-default-arguments` (147 sites: `CasBackend.h` interface, `IObjectStorage.h`/`S3ObjectStorage.h`, all backend implementers, test helpers/fixtures) and adds non-virtual convenience overloads on the base classes that forward the previous default values. Derived classes gain `using` declarations to unhide the base overloads. Qualified parent-implementation calls in test fault backends switched to the explicit 3-arg form — the 2-arg form would now route through the base forwarder and re-enter the derived override virtually (double fault injection; caught by the battery). Bulk edits produced by codex (gpt-5.6-luna) per the T13 brief; overload visibility and qualified-call fixes plus verification by Claude. Battery 919/919 green. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Semantics-preserving conformance for the remaining flagged classes: readability-container-contains, readability-isolate-declaration, google-runtime-int (AWS SDK retry-API overrides keep `long` with targeted NOLINT — the override contract owns the type), readability-duplicate-include, cppcoreguidelines-init-variables, cert-msc, modernize-raw-string-literal, modernize-use-starts-ends-with, bugprone-empty-catch (comments only — no new behavior), googletest naming, bugprone-argument-comment, bugprone-optional-value-conversion, bugprone-misplaced-widening-cast (CasTypes.h site audited: not a real precision bug — the value is range-validated to 0-5; cast made explicit without value change). CasRefCowMap's own `contains` keeps its `find` with NOLINT (self-recursion). Bulk edits by codex (gpt-5.6-luna) per the T13 brief (.superpowers/sdd/task-13-batch2-report.md); one over-removed include (PartFolderAccess.h) restored and verification by Claude. Battery 919/919. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

…n't abort the server (STID 0883) CI PR#2073 (content_addressed storage lanes) crashed with "Too large size (9223372036854775870) passed to allocator" (LOGICAL_ERROR, server abort) running the regression test 04070_no_crash_extreme_compress_block _size. Root cause: an extreme max_compress_block_size (2^63-1) flows into ContentAddressedTransaction::writeFile's buf_size and, unclamped, reaches the CaContentWriteBuffer base-class allocation (Memory::alloc), where checkSize (>= 0x8000000000000000) fires. The ordinary MergeTree writers clamp compress -block sizes to 256 MiB (MergeTreeWriterSettings::MAX_COMPRESS_BLOCK_SIZE) for exactly this reason; the CAS write path received the value unclamped. Fix: clamp buf_size and adaptive_write_buffer_initial_size to 256 MiB at the CAS allocation site (both CaContentWriteBuffer ctors), mirroring the ordinary clamp. New gtest CasContentWriteBuffer.ExtremeBufferSizeIsClampedNotPassedToAllocator reproduces the exact crash number without the clamp (verified RED) and passes with it. CA gtest gate 1057/1057. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=latest&name_0=PR Related: #2073

…cle (UNMOUNT/FSCK) Design for STID 3982-3b48 (CI PR#2073). Five parts: (1) MountLeaseKeeper's vanished-backing-store case throws FILE_DOESNT_EXIST (not LOGICAL_ERROR) so a background renewal thread never aborts the server + new CasMountLeaseLost counter; (2) SYSTEM CONTENT ADDRESSED UNMOUNT <disk> (shutdown() the pool, auto-remount on next access) as a clean alternative to rm -rf under a live mount; (3) online SYSTEM CONTENT ADDRESSED FSCK <disk> via read-only runFsck(Pool&); (4) rewrite the no-leftovers test teardown to GC RUN -> FSCK -> UNMOUNT -> rm -rf; (5) rename the offline clickhouse-disks fsck applet to ca-fsck for consistency. Auto-teardown-on-DROP stays out of scope. Related: #2073

…ence Task 7 added the SYSTEM_CONTENT_ADDRESSED_FSCK AccessType but only updated the AccessType.h macro list, not the 01271_show_privileges reference — the test enumerates every privilege, so the new row (right after MOUNT, matching the AccessType declaration order) was missing and fasttest failed with a one-line diff. 01271 was not in the local gtest/FSCK-access test runs, so the gap surfaced only in CI. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=d780fa7473807203916125e96cf73a9774c84464&name_0=PR&name_1=Fast%20test PR: #2073 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb

Add two DESIRABLE items under §3 (GC correctness/observability) surfaced by triaging the CAS-s3 stateless run (Altinity PR #2073, run 30019911967): - [RECOVERED-INDEGREE-ATTRIBUTION] the "delete_pending recovered in-degree — structurally impossible … investigate" warning is a false alarm (dedup-adopt vs condemn TOCTOU, spared, no data loss). Downgrade the GC log to a ProfileEvent + Debug and move the real adopt-without-resurrect detector to the writer's edge-commit. - [CONDEMN-GRACE-WINDOW] a cool-down before condemning a just-zeroed blob to kill hot-dedup churn (tiny system-log blobs) at the source; flagged higher-risk (condemn-timing/ack-floor, TLA-gated, protocol-veto). Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…GABRT) CI-confirmed on Altinity PR #2073 (run 30019911967, asan_ubsan CAS-s3 stateless): a mount-lease renewal PUT that times out client-side (ambiguous — may have applied server-side) gets soft-retried with a stale token, mismatches against its own bumped body, and falls through MountLeaseKeeper's classifier (same uuid/epoch, unfenced — none of the 3 cases match) into the base class's generic LOGICAL_ERROR, aborting the server under ASan. A third variant of the STID 3982-3b48 family (parts 1a/1b covered vanished/absent-at-release). Also notes: amd_msan/amd_tsan CAS-s3 stateless jobs in the same run hit the 6h job timeout with zero artifacts — unknown if same crash-loop or a plain hang. Docs-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The unsharded CA-s3 sanitizer lanes do not fit the 6h GitHub job timeout, and the kill lands after the test loop but before result upload, so the lane reports NO artifacts at all and praktika's aggregate stays RUNNING forever. In run 30203355812 (sha e2d04bf): tsan finished all 10990/10990 tests at 5h59m and was killed 16s later during teardown; msan was at 5824/10990 (53%, steady progress, no hang) when killed; asan_ubsan passed at 5.4h — too close. This is the full explanation of the "msan/tsan CA-s3 cancelled at exactly 6h with zero artifacts" pattern seen in three consecutive runs. Shard asan_ubsan and tsan 2 ways and msan (the slowest, ~11h projected) 3 ways, following the existing "amd_tsan, s3 storage, parallel, N/M" convention; ci/jobs/functional_tests.py already parses the N/M batch token generically. Workflow YAML regenerated with praktika yaml. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

…irectory The refusal in prepareAdoptFromManifest throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds instead of behaving like a catchable exception, so the EXPECT_THROW form killed unit_tests_dbms on all three sanitizer CI lanes (asan_ubsan/tsan/msan, 0s failures with "Logical error: 'Relink target ... does not address a content-addressed part directory of a live table'"). Split it the same way as the CasWiringOpsDeathTest precedent in this file: EXPECT_THROW stays for plain release builds, and a CasWiringExchangeDeathTest EXPECT_DEATH variant proves the same refusals positively abort under DEBUG_OR_SANITIZER_BUILD. Verified: build_asan runs the death test (1 test, OK), build (release) runs the EXPECT_THROW variant (9 tests from 2 suites, all OK). Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

The pre-scrape cleanup removed only filesystem_caches/*/status (one level), but dynamic cache disks created by tests nest their path — the binary CA-s3 lane died on filesystem_caches/disks/cache_03517/status with "Cannot lock file ... Another server instance in same directory is already running" (StatusFile.cpp flock, EWOULDBLOCK) raised from FileCache::initialize when clickhouse-local initialized the disk map. Make the glob recursive so any depth is covered, same mechanism as before: removing the file lets clickhouse-local create and lock a fresh inode even when the not-fully-stopped server still holds the old one. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

…lanes Tag it no-content-addressed-storage: the coverage is the inline content-addressed disk the test creates itself, so it stays meaningful on every ordinary lane. On lanes whose DEFAULT MergeTree storage is content-addressed, system.remote_data_paths (no disk_name pushdown — the applyFilters TODO in StorageSystemRemoteDataPaths.cpp) also walks the huge shared default pool with the whole run's data, and on the S3 (RustFS) variant that walk exceeds the 600s test timeout (recurred in two consecutive runs, plus a "Some queries hung" ride-along). The BACKLOG pushdown item stays open as the real fix; also note the ConnectionGroup Disk-session pressure signal from the same run. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=e2d04bfe37eec5af58bde837e1cc89d11016ef6f&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

Same class as 5f292e0: PreparedPartWrite's one-shot-terminal rejection throws LOGICAL_ERROR, which aborts the whole process in debug/sanitizer builds (Exception.cpp's handle_error_code) instead of behaving like a catchable exception, so CasPartFolderAccess.PreparedPartWriteRejectsASecondTerminal killed unit_tests_dbms on all three sanitizer CI lanes (0s failures). The file's author knew the class (the NETWORK_ERROR/MEMORY_LIMIT_EXCEEDED choices further down cite it) — these two spots were the leftovers: - PreparedPartWriteRejectsASecondTerminal: split per the CasWiringOpsDeathTest precedent — expectThrowsCode stays for plain release builds, a CasPartFolderAccessDeathTest EXPECT_DEATH twin proves the same rejections positively abort under DEBUG_OR_SANITIZER_BUILD. - PreparedPartWriteMoveTransfersTheTerminalDuty: the single moved-from abort check is guarded in place (EXPECT_DEATH forks, so the rest of the test continues in the parent). A sweep of all CAS gtest files for unguarded LOGICAL_ERROR throw expectations found no other gaps (blob_upload_pool, ref_writer, ref_install_safety, upload_detached, upload_fanout all carry guards). Verified: build_asan 39/39 with both death tests OK; build (release) 30/30 with the EXPECT_THROW variants. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=8417137aeb842f463066fd68a75b60ffeae30d63&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

CasRefInstallSafetyDeathTest.DenyGuardStopsAnAllocation dispatched on DEBUG_OR_SANITIZER_BUILD, but DENY_ALLOCATIONS_IN_SCOPE itself is gated on MEMORY_TRACKER_DEBUG_CHECKS (MemoryTracker.h: defined only under !NDEBUG). Sanitizer builds define NDEBUG, so there the guard compiles to static_assert(true) and the death test "failed to die" on all three sanitizer CI lanes — visible verbatim in the CI error output. (These lanes only now reached this test at all: the earlier LOGICAL_ERROR aborts in gtest_ca_wiring / gtest_cas_part_folder_access killed the process before it, so each fix un-shadows the next tail failure.) Gate the death test on MEMORY_TRACKER_DEBUG_CHECKS instead, and drop the throw-only EXPECT_ANY_THROW branch as dead code: MEMORY_TRACKER_DEBUG_CHECKS implies !NDEBUG implies DEBUG_OR_SANITIZER_BUILD, so whenever the guard exists its LOGICAL_ERROR aborts — death is the only observable outcome (the old comment claimed the opposite implication). Verified: build_debug runs the death test (OK, guard fires); build_asan compiles it out, remaining CasRefInstallSafety* 21 tests x5 repeats green. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

RCA of the fence cascades on the CA-s3 sanitizer lanes (run for 07f8398, msan 2/3 lane): clickhouse-server logged 351 bursts of 'Connection refused' (e.code() = 111) to rustfs at localhost:11121 spread over the whole run — the service stayed up between bursts, so this is fd/accept exhaustion, not a crash. rustfs was launched WITHOUT the open-files-limit raise that start_azurite performs for exactly this failure mode, while the server under parallel sanitizer load holds 10k+ active Disk-group S3 sessions. Most bursts were absorbed by the CAS request retries (max_attempts=16 / 90s deadline); the one at ~02:00-02:01 outlived (a) in-flight writes' 90s budgets (Code 210 UNCERTAIN) and (b) the mount-lease renewal confirm window (TTL 30s), so the mount fence tripped (fail-closed by design) and every writing test failed with Code 668/210 from 02:02:52 until remount completed ~02:05:00, with straggler failures to ~02:10 — ~40 of the run's 66 test failures. Also add rustfs.log to the uploaded artifacts: it was already written to ci/tmp/rustfs.log but never uploaded (azurite/kafka/minio logs are), which was exactly the missing evidence for this class of triage. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=07f8398acddff2c676aded5f804f122984845a4a&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

ilejn · 2026-07-28T21:09:16Z

+
+```sql
+SYSTEM CONTENT ADDRESSED GC RUN [ON CLUSTER cluster_name] [disk_name]
+```


GC DISABLE ?

… razor RefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed on 5 of 6 sanitizer-lane runs across two CI rounds (asan/tsan/msan, 'refLaneWedgedForTest — Actual: false, Expected: true') while passing every quiet local run. Reproduced locally 14/14 by adding full CPU load with the test otherwise unchanged and filtered alone — so not test-ordering contamination, pure timing. Root cause: the test set attempt_timeout_ms == operation_deadline_ms == 100, which turns the request controller's pre-send gate (putIfAbsentControlled: 'now + attempt_timeout > deadline' returns Unresolved WITHOUT sending) into a zero-width race — it only passes when no millisecond tick elapses between the deadline capture and the gate. This test uniquely burns that window encoding the ~1700-op removal chunk, so on a loaded or sanitizer-slow machine the gate fired first, the injected ambiguous fault was never reached, the sweep failed CLEAN — nothing sent, nothing ambiguous, and the product CORRECTLY did not wedge the lane. The test had over-specified the failure mode it would meet. Ambiguity is guaranteed by max_attempts = 1 alone; widen operation_deadline_ms to 5000 so the PUT is always actually sent and the wedge is deterministic on any machine. Verified: 20/20 green (5 repeats x 4 sweep tests) on ASan under half-core CPU load; the sibling wedge tests keep the tight budget deliberately — their capture-to-gate window is empty. Report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=834c9517f56a83927b0c941ac08d8dfd82460579&name_0=PR PR: #2073 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky

… (PR 2073 review) Addresses the three review recommendations from strtgbb on #2073 - All 10 CAS `Job.ParamSet` entries move out of `JobConfigs.functional_tests_jobs` into a new `AltinityJobConfigs.cas_functional_tests_jobs`. Every workflow that picked them up implicitly -- including `release_builds` (`binary` filter), `release_branches` and `backport_branches` (`asan` filters) -- now concatenates the two lists in the original order, so the job sets are unchanged. Proven by an empty `diff -r` of the regenerated `.github/workflows` and by an identical dump of `workflow.jobs` names for every `ci/workflows/*.py` module before and after. - The CAS ParamSet comments are cut to the load-bearing constraints: RustFS rather than MinIO OSS for enforced conditional deletes, and sharded sanitizer lanes because an unsharded one exceeds the 6h GitHub job timeout. - The CAS comments in `ci/jobs/scripts/clickhouse_proc.py` are compressed the same way, keeping why the disks are opened read-only, why the substitution is keyed on the `<metadata_type>cas</metadata_type>` marker, and why `grep -R` and `sed --follow-symlinks` are required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN

CASRefCheckpointJoin.EncodedCkptSizeIsIndependentOfCardinality and CASRefWriterStalePrecommitSweep.BoundedBatchesAndInterruptionResumeAcrossMounts failed identically on all three sanitizer unit-test lanes (asan_ubsan, tsan, msan) at 9ad3e15, thrown from ordinary product code, not a sanitizer report -- both are the same class of bug `8f9e63c7a19` already fixed once for the second test: a CPU-bound encode step (committing `MANY_REFS`/ ~`ref_txn_max_ops` ref ops in one call) measured against a real-wall-clock deadline (`mount_lease_ttl_ms` for the first test, `operation_deadline_ms` for the second), which a slow-enough sanitizer build -- msan above all -- can outrun regardless of how wide the deadline is set. Both the mount fence and the ref-log request controller (`CasRefLedger`'s `controller_boot_ms_fn`) already read time through `PoolConfig::boot_ms_fn`, the same injectable seam a few lines below already uses (the `resumer` pool in the second test). Freeze it for the pools that do the CPU-bound work instead of widening the deadline further, removing the race rather than giving it more room. Not build-verified: the shared dev box was at 63 GB free disk (existing full sanitizer builds run 43-51 GB each) and near swap exhaustion, so no fresh build was attempted. Verified by tracing the exact production wiring (PoolConfig::boot_ms_fn -> CasRefLedger::controller_boot_ms_fn -> CasRequestController::now_ms, and CasMountRuntime::bootMsNow) and matching the pattern already in use for this file's `resumer` pool. CI: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=9ad3e15b68823633cc1c2a1539292f20de3e7b5e&name_0=PR PR: #2073 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1r6FXkwCGRmniKKVfs2ZN

…er test parks a holder inside an ambiguous write Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

Eligibility was memoized per build prefix and every INSERT is its own build, so a page of 1000 manifests read the mount key ~3000 times (two guaranteed 404s per namespace prefix chain). The floor is one body per server root and retirement is permanent, so one observation per namespace decides every build of the page. New phase metrics floor_lookups and floor_reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…ogs and ref snapshots Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…hint is never delivered, the store's answer always is Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…nder a chosen retry profile with absence as success Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…ad counts a dropped hint as wasted at once Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

… the catalog cut The freeze-before-cut read guarded a same-key rebirth that cannot happen: manifest keys are write-once. Retain decisions come from the key, the floor, the protection view and the premise; the body is read for candidates only, so a page of live manifests costs no GET. The budget is a candidate budget and its description says so. Also updates gtest_cas_orphan_manifest_sweep.cpp's CursorPageCannotDeleteManifestReplacedAfterObservation, which exercised the removed freeze discipline: it forced a same-key rewrite between the LIST and the catalog cut and asserted the old token lost the delete. With the freeze gone, the body read happens after that rewrite, so the plan correctly captures the live incarnation and deletes it under its own current token; the test is renamed to CursorPageDeletesTheIncarnationSeenAfterTheCatalogCut and its assertions flipped to match. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…e-once keys through every backend and the engine Keys are WriteOnceKey, mintable only by Layout for manifests, ref logs and ref snapshots. Native mode issues one DeleteObjects under the control-plane profile; the emulated single-process mode deletes under its lock with the same token bookkeeping as the single-key delete. The engine sends one chunk under the ordinary attempt loop and reissues the whole chunk on a fault. New setting gc_bulk_delete_chunk_keys (1 to 1000) sizes the consumers' chunks. Backend is abstract, so every direct implementation across the CAS gtest fixtures needed the new pure virtual too, not only the two the plan named: gtest_cas_pool.cpp (WriteCountingBackend, ProbeWatchingBackend, ForwardingBackend, FenceInAdoptWindowBackend) and gtest_cas_mount.cpp (AlwaysVanishesBackend), all forwarding to inner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…through the read-ahead; hints stop at the epoch and are discarded at a seal Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…ost during creation step 1 is FencedOut Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…of write-once keys One DeleteObjects per chunk instead of one conditional delete per manifest; every chunk that succeeded is recorded before a later one can fail. manifests_deleted now counts keys deleted or found already absent, which a batch response cannot tell apart; the column text says so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…uest plane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…the work-budget and read-ahead doc comments, and give the read-ahead parity test real candidates and a window-spanning tail walk Review findings addressed: the missing-cursor epoch cross now discards the old epoch's outstanding hints before jumping, matching the seal path; a work-budget break, a corrupted-tail throw, and the missing-cursor cross no longer strand hinted-but-untaken keys against the shared reader for the rest of the page (a small RAII guard discards them on any walk exit that isn't an ordinary same-epoch advance); the work-budget doc comment now states it counts taken logs, not hinted ones; the CasGcReadAhead doc comment states the one bounded exception where a hint may overshoot a seal; planManifestCursorPage's read_pool/read_concurrency parameters are documented. The PageIsIdenticalInlineAndWithReadAhead test now uses a fixture with real orphan candidates and a committed-tail walk spanning more than one read-ahead window, entirely within one epoch so the GET multiset is concurrency-invariant, and asserts full nomination equality and CASGCReadAheadHit > 0 so the comparison can no longer pass vacuously. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…r chunk on RustFS Drops a table whose parts leave >= 6 owner-removed manifests, forces the GC pool's cas_gc_bulk_delete_chunk_keys down to 2, and asserts the manifest_deletes phase's request count matches the chunking math exactly (ceil(attempted / chunk_keys)), with the engine's own CASBulkDeleteRequests / DiskS3DeleteObjects ProfileEvents counters agreeing. Sums phase rows across however many GC rounds it took, since a DROP's manifests are not guaranteed to fold in one round. The originally planned "external remove_object beats GC, absent key is accepted" scenario turned out to be architecturally unreachable on a single node: fold_ref_intake's foldManifestEdges must read the manifest body itself to compute the owner-removal edge's per-blob deltas, so deleting a live manifest before GC gets to it clamps the namespace permanently (system.cas_log: "owner-removal: edge-bearing committed body missing at removal-fold") instead of exercising the tolerated- absence path. That coverage already exists at the backend/engine unit level; this test proves the chunking behavior on the wire instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…idated cohorts One catalog and gc/state revalidation per chunk of write-once keys, then one batch delete; no per-key HEAD. The authority window widens from one key to one chunk: what it can delete is this life's keys named by this round's durable plan, which a successor derives too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

… test discriminates the two schedules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

… throw; the cache weight is precomputed `leave` is called from a noexcept guard, but it read the injected clock before taking the lane's mutex; a throw there aborted the process before the guard's sole duty -- removing the item -- ever ran. The locked section now erases the item first and stays non-throwing: the clock read for the stuck-holder snapshot is wrapped in its own try/catch with a no-snapshot fallback, and the queue-wait profile event and log line move outside the lock into the existing catch-all. `RememberedWeight::operator()` called `Etag::render`, which allocates and can throw, from inside the cache's own accounting after it had already subtracted the entry being replaced; a throw there left the size accounting able to underflow on the rescue path. The weight is now computed once in `remember`, before `cache->set`, and stored on `Remembered`; the functor just returns it, noexcept. Also added chassert guards: `leave` on the lane lookup succeeding, and the destructor on no lane outliving its pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…t casAdmitEntry test gets its death-test twin `TheCacheForgetsWhatItCannotVouchFor` used `s3Error`/`Aws::S3::S3Errors` unconditionally in its Refused sub-case, so the file did not compile without AWS S3; the sub-case, and the assertion that depends on its side effect on the cache, are now under `#if USE_AWS_S3`, matching `ResultsAreTheEnginesOwn`'s existing guard. `AStaleHintCasAdmitEntryAdmitsWhenTheStoreHasRoom`'s stale-hint decide inserts a duplicate namespace before the lane ever rereads, so `encodeRefCatalog` throws `LOGICAL_ERROR` on the hint attempt; under debug/sanitizer builds constructing that exception aborts before the test's own rescue runs. Split like the file's other `casAdmitEntry` refusal tests: the original stays under `#ifndef DEBUG_OR_SANITIZER_BUILD`, and a `CASRefCatalogDeathTest` twin under `#if defined(DEBUG_OR_SANITIZER_BUILD)` asserts `EXPECT_DEATH` instead. Reworded a comment in `ADecideRunsWithTheLaneMutexReleased`: a regression here hangs the whole `CAS*` gate, since the gate runs the bare binary with no per-test timeout of its own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…fter round_commit Review findings: the GcFence BeforeFirstDelete race test moved gc/state's lease during the round's hot scan, before round_commit's own gc/state CAS -- the round threw before cleanupRefObjects ever ran, so the "nothing deleted" assertions passed vacuously rather than testing cleanup's own refusal. Fixed by arming the move instead of firing it immediately, and gating the actual bump on the catalog-then-gc/state adjacency authorityHolds's own revalidation produces (no other read in a round is that pair back to back), which lands the move well after round_commit. The rebirth race test asserted nothing about the reborn life (its listing was empty). Fixed by seeding a _log and a _snap under the reborn life from inside moveAuthority's CatalogRebirth branch, right after the catalog rewrite lands (any earlier and an unknown-incarnation sweep elsewhere in the round can claim them); added a second round asserting the old cohort's untouched second key survives once the plan is for the reborn life instead. Also: both tests renamed BeforeFirstDelete -> BeforeFirstChunk, dead per-key comment paragraph in CasGc.cpp removed, and last_read_key (dead in the prior round, load-bearing again here for the adjacency gate) documented accordingly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…sserts it found its item `TheCacheForgetsWhatItCannotVouchFor`'s `#if USE_AWS_S3` block commits ticket 2, so the terminal `expectBytes` was wrong under `USE_AWS_S3=0` (the object is "1,3,4,5,6,7" there, not "1,2,3,4,5,6,7"). The expected bytes are now a constant defined next to the same guard and used by that assertion. `gtest_cas_hot_keys.cpp` reached `USE_AWS_S3` only transitively; added `#include "config.h"` directly, as `gtest_cas_requests.cpp` does, so an undefined macro cannot make the guard silently false. `CasHotKeys::leave` now asserts the item was actually found in its lane's queue before erasing it, the symmetric assertion to the lane lookup's own `chassert`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…e rebirth round-2 claim Review findings against the previous fix round: 1. Data race: `catalog_seam_armed` and `last_read_key` were plain members written on every `read`, which also runs on the GC read-ahead pool's threads (`gc_read_concurrency` defaults to 16) -- UB on a String, and a pool-thread read landing between the revalidation's two reads could silently stop the seam from firing. Fixed with a dedicated mutex guarding both, and narrowed the tracked "previous key" to control keys only (catalog/gc-state) so an unrelated concurrent read from a read-ahead worker -- which never touches either -- cannot perturb the adjacency signal even while running lock-free between the critical sections. 2. The rebirth test's second-round assertion ("the old cohort's second key survives") passed for the wrong reason: the reborn life had no checkpoint, so round two's recovery grounding found it unusable and suppressed the round's destructive work wholesale -- the key survived because nothing ran, not because the plan moved to the reborn life. Seeded a real checkpoint for the reborn life (life_epoch/committed_through/checkpoint_snapshot_id naming the already-seeded log and snapshot) so round two actually runs. With a real round, the key does NOT survive: it is reclaimed by the namespace janitor as dead-life debris (`CasNamespaceJanitor.cpp`, a physical life the catalog no longer resolves), exactly matching spec §D's own text. Rewrote the assertion to attribute the delete precisely -- the `ref_object_cleanup` phase's `suppressed` metric (confirms the round ran), the `namespace_cleanup` phase's `janitor_deleted` metric, and the global `CASRefCleanupObjectsDeleted` counter delta (confirms `cleanupRefObjects` itself deleted nothing) -- instead of asserting "survives" against a mechanism this task never touches. Minor: the `chassert` guarding `moveRefCleanupAuthorityBeforeFirstChunk` against `Authority::GcFence` misuse is compiled out in release; replaced with a real throw. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

… post-merge steps (ASan run of the death twin, the acceptance measurement) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…d rows; BACKLOG: measured table, E evidence refined, two new items (harness NeedsRecovery trip, blob pending_deletes now dominant) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

…tion test, CASRefCleanupObjectsDeleted text, cohort loop advances by the chunk actually sent, small test and header tidy-ups Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

… floor memo, late manifest reads with read-ahead, removeManyWriteOnce for manifests and ref objects, ref-cleanup cohorts) Design: docs/superpowers/specs/2026-09-04-cas-gc-immutable-key-round-cost-design.md Measured on real GCS: steady-state fold_reduce 300-380 s -> 2-5 s; manifest_deletes 617 s -> 2 s on a 1506-key round; ref_object_cleanup 204 s -> under 1 s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

…nder item E, and the sweep's double tail walk as its own item; GCS compose header names fsck_only_gcs.xml in the prefix bump; soak report's host ports corrected Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

One FIFO ticket per pool and key above the CAS request engine, a last-known-object cache under one rule, the flat conflict pause with the growing schedule after a settled fault, the catalog as the first caller, the pool owning the lane. Gate CAS* 2406/2406; reviewed per task, whole-branch (opus) and end to end (codex gpt-5.6-sol high); rulings in docs/superpowers/cas/2026-09-04-hot-key-lane-phase-a-rulings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

filimonov mentioned this pull request Jul 17, 2026

[WIP] CAS draft (adopting to CI/CD, not for review / merge) #2066

Closed

ilejn reviewed Jul 28, 2026

View reviewed changes

alsugiliazova mentioned this pull request Jul 30, 2026

CAS — consolidated static analysis audit findings (tracking) #2031

Open

40 of 82 tasks

svb-alt added antalya antalya-26.6 roadmap Key features and improvements for Antalya project labels Jul 30, 2026

strtgbb requested changes Jul 31, 2026

View reviewed changes

Comment thread ci/defs/job_configs.py Outdated

Comment thread ci/defs/job_configs.py Outdated

Comment thread ci/jobs/scripts/clickhouse_proc.py Outdated

filimonov and others added 30 commits September 4, 2026 16:47

ca-engine: the lane's clock read leaves the mutex; the throttled-hold…

4cf344d

…er test parks a holder inside an ambiguous write Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-engine: the throttled-holder test admits its waiters one at a time

2e5f1a1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-layout: WriteOnceKey, mintable only by Layout for manifests, ref l…

d35bc0e

…ogs and ref snapshots Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

ca-engine: the hot-key lane's cache pinned by tests — a verdict on a …

74eff7b

…hint is never delivered, the store's answer always is Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

object storage: removeObjectsIfExistUnderProfile, one DeleteObjects u…

bcf7871

…nder a chosen retry profile with absence as success Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

ca-gc: key reader with hint, take and discard; GcReadAhead::discardRe…

e788a1e

…ad counts a dropped hint as wasted at once Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

ca-gc: orphan sweep reads candidate bodies and both ref-stream walks …

ee8f72d

…through the read-ahead; hints stop at the epoch and are discarded at a seal Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0124yU8tjzfrEGtNdZhS4TnV

ca-catalog: catalog mutations take the pool's hot-key lane; a fence l…

c76d395

…ost during creation step 1 is FencedOut Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-catalog: the creation-lifecycle tests give their rival its own req…

56080da

…uest plane Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-pool: the pool owns one hot-key lane and its three planes share it

2b46c3d

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-catalog: the step-1 true-refusal test reaches the lane; the pacing…

1216f81

… test discriminates the two schedules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-docs: BACKLOG records phase A's deferred review minors and the two…

16fd9bc

… post-merge steps (ASan run of the death twin, the acceptance measurement) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

ca-docs: rulings and review record for the hot-key lane phase A branch

724d6f0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEdk5vAo6xYr3ro75LX2xz

Navigation Menu

[WIP] CAS draft (adopting to CI/CD, not for review / merge) - #2073

[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073
filimonov wants to merge 5188 commits into
antalya-26.6from
cas-gc-rebuild

filimonov commented Jul 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

ilejn Jul 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

filimonov commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

filimonov commented Jul 17, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading