Conversation
35 tasks done
zvonand
force-pushed
the
feature/antalya-26.8/pr-2159
branch
from
October 1, 2026 13:33
4e107a1 to
c7c09a1
Compare
…next commit) --- Original cherry-pick message follows: Merge pull request #2159 from Altinity/feature/antalya-26.6/CAS [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 # Conflicts: # .github/workflows/fast_builds.yml # .github/workflows/master.yml # .github/workflows/pull_request.yml # .github/workflows/pull_request_community.yml # ci/jobs/functional_tests.py # ci/jobs/scripts/clickhouse_proc.py # ci/jobs/scripts/clickhouse_service.py # ci/workflows/pull_request.py # ci/workflows/pull_request_community.py # ci/workflows/release_branches.py # docs/concepts/features/configuration/server-config/storing-data.mdx # programs/disks/DisksApp.cpp # programs/disks/ICommand.h # programs/server/Server.cpp # src/Common/FailPoint.cpp # src/Common/ThreadStatus.h # src/Core/ServerSettings.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.h # src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.cpp # src/Disks/DiskObjectStorage/MetadataStorages/IMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/MetadataStorageFactory.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Disks/DiskObjectStorage/ObjectStorages/Local/LocalObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/RegisterDiskObjectStorage.cpp # src/Disks/IDiskTransaction.h # src/IO/ReadPipeline.cpp # src/IO/ReadPipeline.h # src/IO/S3/copyS3File.cpp # src/IO/S3/copyS3File.h # src/IO/S3Common.cpp # src/IO/WriteBufferFromS3.cpp # src/IO/tests/gtest_writebuffer_s3.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Interpreters/ServerAsynchronousMetrics.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/tests/gtest_Parser.cpp # src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp # src/Storages/MergeTree/DataPartStorageOnDiskFull.cpp # src/Storages/MergeTree/DataPartsExchange.cpp # src/Storages/MergeTree/IMergeTreeDataPart.cpp # src/Storages/MergeTree/MergeTask.cpp # src/Storages/MergeTree/MergeTreeData.cpp # tests/integration/test_replicated_database/test.py # tests/queries/0_stateless/02253_empty_part_checksums.sh # tests/queries/0_stateless/02254_projection_broken_part.sh # tests/queries/0_stateless/02255_broken_parts_chain_on_start.sh # tests/queries/0_stateless/02369_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02370_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02444_async_broken_outdated_part_loading.sh # tests/queries/0_stateless/04215_replicated_missing_covered_part_on_start.sh # tests/queries/0_stateless/04327_reader_executor_metrics.sql # tests/queries/0_stateless/04328_reader_executor_kpi_async_metric.sql
Kept the antalya-26.8 shape everywhere and layered the CAS changes on top: - `DiskObjectStorageTransaction`: base already routes every metadata effect through `addOperation` (eager vs queued, driven by `appliesOperationsEagerly`), which is the same mechanism as the PR's `dispatch`. Kept `addOperation` at the call sites, made it also honour `transactionIsStagingOverlay`, and dropped the duplicate `dispatch` template. - `copyS3File`: threaded `copy_mode` through the new `copyS3FileImpl`; `copyS3FileRange` passes `ObjectStorageCopyMode::Default`. - `DataPartStorageOnDiskFull`: moved the in-flight read-your-writes checks into the new `*Impl` methods (packed skip-index handling now lives in the base wrappers) and restored `getPackedFileUncompressedSize`, which the auto-merge had mixed up. - `DataPartStorageOnDiskBase::freeze`/`freezeRemote`: clone-transaction cleanup keeps the base `txn_version.txt.tmp` removal and `writeInvalidatedSystemColumnsFile`. - `LocalObjectStorage`: added the directory check to the relocated `tryGetObjectMetadata`. - `ThreadStatus`: base already keeps the parent `ThreadGroup` alive via `ThreadGroup::parent`, so the PR's duplicate `parent_thread_group` member is not added. - `RegisterDiskObjectStorage`: `use_fake_transaction` no longer exists on antalya-26.8 (object storage disks always use real transactions), so the PR's guard is not needed. - `MergeTreeData::removePartsInRangeFromWorkingSet...`: base already commits the empty covering part's disk transaction unconditionally. - `DataPartsExchange`: kept base `..._WITH_INVALIDATED_SYSTEM_COLUMNS = 10` and the PR's `..._WITH_CA_CONFIRM = 11` as the advertised maximum. - CI: added `cas_functional_tests_jobs` to the 26.8 `FUNCTIONAL_TESTS_JOBS` lists, `start_rustfs` into the 26.8 `start` sequence, removed a duplicated `cas_functional_tests_jobs` block from `altinity_jobs.py`, and hand-merged the generated workflow YAMLs (only CAS jobs added). - Removed duplicated `copyObject` counter and merged both `CopyObject` mocks in `gtest_writebuffer_s3.cpp`. - Stateless tests deleted on antalya-26.8 (converted to integration tests) stay deleted. Source-PR: #2159 (#2159)
…next commit) --- Original cherry-pick message follows: Merge pull request #2300 from Altinity/feature/antalya-26.6/CAS-improvements CAS improvements # Conflicts: # src/Common/ErrorCodes.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h # src/IO/ReadBufferFromS3.cpp # src/IO/ReadSettings.h # src/IO/S3/tests/TestPocoHTTPServer.h # src/IO/WriteSettings.h
Kept antalya-26.8's renumbered fork error codes (1016-1018) and added the PR's CAS codes 1037-1038 with END = 1038; dropped the PR's comments that describe the old 1010/1011 numbering, which does not exist on antalya-26.8. Adapted: setClickhouseAttemptNumber -> setClickHouseAttemptNumber (renamed on antalya-26.8 by e8e1e12) Adapted: S3ObjectStorage client_slot merged with antalya-26.8's client_restricts_server_credentials member Adapted: S3IteratorAsync attempt_seed added alongside antalya-26.8's description member Adapted: readObject keeps antalya-26.8's UnknownSize mapping and object.etag, passes the PR's refresh_callback Adapted: TestPocoHTTPServer keeps antalya-26.8's makeMockServerParams with the PR's dedicated thread pool Source-PR: #2300 (#2300)
…next commit) --- Original cherry-pick message follows: Merge PR #2307 (cas/configurable-mount-lease-timing) into feature/antalya-26.6/CAS-improvements-cicd-fixes Brings k-morozov's configurable CAS mount lease TTL / renewal period (#2307) so the heartbeat-model work can land on top of it while keeping the author's commits intact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> # Conflicts: # docs/en/antalya/cas/configuration.md # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp
The changes from #2307 (`mount_lease_ttl_ms` / `mount_renew_period_ms` settings, members, initializers, `PoolConfig` wiring, validation, and the doc rows) are already on the branch. Every conflict was only the "theirs" side removing later work built on top of them (`unsafe_remount_no_delay`, the `attempt_timeout_ms` check, `connect_timeout_cap_ms`, the updated `cas_mount_renew_period_ms` doc text). #2307 does not remove those, so "ours" is kept in every hunk. Source-PR: #2307 (#2307)
…next commit) --- Original cherry-pick message follows: Merge pull request #2349 from Altinity/cas/memworker-split-allocated-source MemoryWorker: use sanitizer info for allocated counter # Conflicts: # src/Common/MemoryWorker.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #2385 from Altinity/fix/antalya-26.6/lldb-budget-and-sigcont CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server # Conflicts: # ci/tests/test_print_stacktraces.py # tests/clickhouse-test
tests/clickhouse-test: applied the PR's lldb budget and JIT-loader flag on top of antalya-26.8's get_stacktraces_from_lldb, which no longer has _ensure_lldb_installed and already attaches via sudo on macOS with keep_output_on_error; kept the base's liveness-check wording in the print_c_stacktraces docstring. ci/tests/test_print_stacktraces.py: kept the deletion from 3a69545 on antalya-26.8 (ci/tests unit tests removed, no automated CI tests); restoring it would revive ~176 pre-existing lines outside the PR's diff. Dropped: ci/tests/test_print_stacktraces.py budget tests — file deleted on antalya-26.8 by 3a69545 Source-PR: #2385 (#2385)
zvonand
force-pushed
the
feature/antalya-26.8/pr-2159
branch
3 times, most recently
from
October 2, 2026 09:33
82f8845 to
eff8803
Compare
zvonand
force-pushed
the
feature/antalya-26.8/pr-2159
branch
from
October 2, 2026 15:01
eff8803 to
e330851
Compare
zvonand
marked this pull request as ready for review
October 2, 2026 15:01
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add experimental content-addressed storage (
cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks,SYSTEM CAScommands, system tables,clickhouse-diskstools,ALTER TABLE ... EXPORT PARTITIONsupport and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage.Add experimental content-addressed storage (
cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks,SYSTEM CAScommands, system tables,clickhouse-diskstools,ALTER TABLE ... EXPORT PARTITIONsupport and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage (#2159 by @filimonov, #2300 by @filimonov, #2307 by @k-morozov, #2312 by @k-morozov, #2326 by @k-morozov, #2349 by @k-morozov, #2360 by @filimonov, #2384 by @filimonov, #2385 by @filimonov, #2393 by @filimonov).CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 10 PR(s) (group
pr-2159). Cherry-picked from #2159, #2300, #2307, #2312, #2326, #2349, #2360, #2384, #2385, #2393.