Skip to content

Antalya 26.8: pr-2159: combined port of 10 PRs - #2461

Open
zvonand wants to merge 15 commits into
antalya-26.8from
feature/antalya-26.8/pr-2159
Open

zvonand wants to merge 15 commits into
antalya-26.8from
feature/antalya-26.8/pr-2159

Conversation

@zvonand

@zvonand zvonand commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Dropped from this backport: the AI dropped these surfaces rather than pulling in a missing prerequisite. Reviewers: confirm each is genuinely optional.

  • ci/tests/test_print_stacktraces.py budget tests — file deleted on antalya-26.8 by 3a69545

Changelog category (leave one):

  • Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add experimental content-addressed storage (cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks, SYSTEM CAS commands, system tables, clickhouse-disks tools, ALTER TABLE ... EXPORT PARTITION support and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage.
Add experimental content-addressed storage (cas) for object-storage disks, which stores MergeTree parts as deduplicated blobs in a shared S3/GCS pool that multiple servers can mount, replicates by relinking existing blobs instead of copying them, and comes with lease-coordinated garbage collection, online integrity checks, SYSTEM CAS commands, system tables, clickhouse-disks tools, ALTER TABLE ... EXPORT PARTITION support and configurable mount lease timing. Also stop object-storage listings from parsing every listed key as a JSON command, which removes one thrown and caught exception per key on every object storage (#2159 by @filimonov, #2300 by @filimonov, #2307 by @k-morozov, #2312 by @k-morozov, #2326 by @k-morozov, #2349 by @k-morozov, #2360 by @filimonov, #2384 by @filimonov, #2385 by @filimonov, #2393 by @filimonov).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Combined port of 10 PR(s) (group pr-2159). Cherry-picked from #2159, #2300, #2307, #2312, #2326, #2349, #2360, #2384, #2385, #2393.

@zvonand zvonand added ai-needs-attention Releasy stopped on a conflict it could not resolve — needs human review releasy Created/managed by RelEasy antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [e330851]

@zvonand zvonand mentioned this pull request Oct 1, 2026
35 tasks done
@zvonand
zvonand force-pushed the feature/antalya-26.8/pr-2159 branch from 4e107a1 to c7c09a1 Compare October 1, 2026 13:33
@svb-alt svb-alt added the antalya label Oct 1, 2026
mkmkme and others added 14 commits October 1, 2026 18:19
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2159 from Altinity/feature/antalya-26.6/CAS

[experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6
# Conflicts:
#	.github/workflows/fast_builds.yml
#	.github/workflows/master.yml
#	.github/workflows/pull_request.yml
#	.github/workflows/pull_request_community.yml
#	ci/jobs/functional_tests.py
#	ci/jobs/scripts/clickhouse_proc.py
#	ci/jobs/scripts/clickhouse_service.py
#	ci/workflows/pull_request.py
#	ci/workflows/pull_request_community.py
#	ci/workflows/release_branches.py
#	docs/concepts/features/configuration/server-config/storing-data.mdx
#	programs/disks/DisksApp.cpp
#	programs/disks/ICommand.h
#	programs/server/Server.cpp
#	src/Common/FailPoint.cpp
#	src/Common/ThreadStatus.h
#	src/Core/ServerSettings.cpp
#	src/Disks/DiskObjectStorage/DiskObjectStorage.cpp
#	src/Disks/DiskObjectStorage/DiskObjectStorage.h
#	src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.cpp
#	src/Disks/DiskObjectStorage/MetadataStorages/IMetadataStorage.h
#	src/Disks/DiskObjectStorage/MetadataStorages/MetadataStorageFactory.cpp
#	src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h
#	src/Disks/DiskObjectStorage/ObjectStorages/Local/LocalObjectStorage.cpp
#	src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp
#	src/Disks/DiskObjectStorage/RegisterDiskObjectStorage.cpp
#	src/Disks/IDiskTransaction.h
#	src/IO/ReadPipeline.cpp
#	src/IO/ReadPipeline.h
#	src/IO/S3/copyS3File.cpp
#	src/IO/S3/copyS3File.h
#	src/IO/S3Common.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/tests/gtest_writebuffer_s3.cpp
#	src/Interpreters/InterpreterSystemQuery.cpp
#	src/Interpreters/ServerAsynchronousMetrics.cpp
#	src/Parsers/ASTSystemQuery.h
#	src/Parsers/tests/gtest_Parser.cpp
#	src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp
#	src/Storages/MergeTree/DataPartStorageOnDiskFull.cpp
#	src/Storages/MergeTree/DataPartsExchange.cpp
#	src/Storages/MergeTree/IMergeTreeDataPart.cpp
#	src/Storages/MergeTree/MergeTask.cpp
#	src/Storages/MergeTree/MergeTreeData.cpp
#	tests/integration/test_replicated_database/test.py
#	tests/queries/0_stateless/02253_empty_part_checksums.sh
#	tests/queries/0_stateless/02254_projection_broken_part.sh
#	tests/queries/0_stateless/02255_broken_parts_chain_on_start.sh
#	tests/queries/0_stateless/02369_lost_part_intersecting_merges.sh
#	tests/queries/0_stateless/02370_lost_part_intersecting_merges.sh
#	tests/queries/0_stateless/02444_async_broken_outdated_part_loading.sh
#	tests/queries/0_stateless/04215_replicated_missing_covered_part_on_start.sh
#	tests/queries/0_stateless/04327_reader_executor_metrics.sql
#	tests/queries/0_stateless/04328_reader_executor_kpi_async_metric.sql
Kept the antalya-26.8 shape everywhere and layered the CAS changes on top:

- `DiskObjectStorageTransaction`: base already routes every metadata effect through
  `addOperation` (eager vs queued, driven by `appliesOperationsEagerly`), which is the same
  mechanism as the PR's `dispatch`. Kept `addOperation` at the call sites, made it also honour
  `transactionIsStagingOverlay`, and dropped the duplicate `dispatch` template.
- `copyS3File`: threaded `copy_mode` through the new `copyS3FileImpl`; `copyS3FileRange`
  passes `ObjectStorageCopyMode::Default`.
- `DataPartStorageOnDiskFull`: moved the in-flight read-your-writes checks into the new
  `*Impl` methods (packed skip-index handling now lives in the base wrappers) and restored
  `getPackedFileUncompressedSize`, which the auto-merge had mixed up.
- `DataPartStorageOnDiskBase::freeze`/`freezeRemote`: clone-transaction cleanup keeps the base
  `txn_version.txt.tmp` removal and `writeInvalidatedSystemColumnsFile`.
- `LocalObjectStorage`: added the directory check to the relocated `tryGetObjectMetadata`.
- `ThreadStatus`: base already keeps the parent `ThreadGroup` alive via `ThreadGroup::parent`,
  so the PR's duplicate `parent_thread_group` member is not added.
- `RegisterDiskObjectStorage`: `use_fake_transaction` no longer exists on antalya-26.8 (object
  storage disks always use real transactions), so the PR's guard is not needed.
- `MergeTreeData::removePartsInRangeFromWorkingSet...`: base already commits the empty covering
  part's disk transaction unconditionally.
- `DataPartsExchange`: kept base `..._WITH_INVALIDATED_SYSTEM_COLUMNS = 10` and the PR's
  `..._WITH_CA_CONFIRM = 11` as the advertised maximum.
- CI: added `cas_functional_tests_jobs` to the 26.8 `FUNCTIONAL_TESTS_JOBS` lists, `start_rustfs`
  into the 26.8 `start` sequence, removed a duplicated `cas_functional_tests_jobs` block from
  `altinity_jobs.py`, and hand-merged the generated workflow YAMLs (only CAS jobs added).
- Removed duplicated `copyObject` counter and merged both `CopyObject` mocks in
  `gtest_writebuffer_s3.cpp`.
- Stateless tests deleted on antalya-26.8 (converted to integration tests) stay deleted.

Source-PR: #2159 (#2159)
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2300 from Altinity/feature/antalya-26.6/CAS-improvements

CAS improvements
# Conflicts:
#	src/Common/ErrorCodes.cpp
#	src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp
#	src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h
#	src/IO/ReadBufferFromS3.cpp
#	src/IO/ReadSettings.h
#	src/IO/S3/tests/TestPocoHTTPServer.h
#	src/IO/WriteSettings.h
Kept antalya-26.8's renumbered fork error codes (1016-1018) and added the
PR's CAS codes 1037-1038 with END = 1038; dropped the PR's comments that
describe the old 1010/1011 numbering, which does not exist on antalya-26.8.

Adapted: setClickhouseAttemptNumber -> setClickHouseAttemptNumber (renamed on antalya-26.8 by e8e1e12)
Adapted: S3ObjectStorage client_slot merged with antalya-26.8's client_restricts_server_credentials member
Adapted: S3IteratorAsync attempt_seed added alongside antalya-26.8's description member
Adapted: readObject keeps antalya-26.8's UnknownSize mapping and object.etag, passes the PR's refresh_callback
Adapted: TestPocoHTTPServer keeps antalya-26.8's makeMockServerParams with the PR's dedicated thread pool
Source-PR: #2300 (#2300)
…next commit)

---
Original cherry-pick message follows:

Merge PR #2307 (cas/configurable-mount-lease-timing) into feature/antalya-26.6/CAS-improvements-cicd-fixes

Brings k-morozov's configurable CAS mount lease TTL / renewal period
(#2307) so the heartbeat-model
work can land on top of it while keeping the author's commits intact.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx

# Conflicts:
#	src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

# Conflicts:
#	docs/en/antalya/cas/configuration.md
#	src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp
#	src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h
#	src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp
The changes from #2307 (`mount_lease_ttl_ms` / `mount_renew_period_ms` settings, members,
initializers, `PoolConfig` wiring, validation, and the doc rows) are already on the branch.
Every conflict was only the "theirs" side removing later work built on top of them
(`unsafe_remount_no_delay`, the `attempt_timeout_ms` check, `connect_timeout_cap_ms`, the
updated `cas_mount_renew_period_ms` doc text). #2307 does not remove those, so "ours" is kept
in every hunk.

Source-PR: #2307 (#2307)
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2349 from Altinity/cas/memworker-split-allocated-source

MemoryWorker: use sanitizer info for allocated counter
# Conflicts:
#	src/Common/MemoryWorker.cpp
Kept the speculative RSS reservation block present on antalya-26.8 and
adapted it to the `MemoryUsage` return type of `getMemoryUsage`
introduced by #2349 (`resident` -> `memory_usage.resident`).

Source-PR: #2349 (#2349)
…ed-variable

Fix the coverage build: unused variable in gtest_cas_throttling_gate.cpp

Source-PR: #2360 (#2360)
…time-knobs

CI: cap the RustFS blocking-thread pool on the CAS-S3 lanes and collect its log
Source-PR: #2384 (#2384)
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2385 from Altinity/fix/antalya-26.6/lldb-budget-and-sigcont

CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server
# Conflicts:
#	ci/tests/test_print_stacktraces.py
#	tests/clickhouse-test
tests/clickhouse-test: applied the PR's lldb budget and JIT-loader flag on top of antalya-26.8's get_stacktraces_from_lldb, which no longer has _ensure_lldb_installed and already attaches via sudo on macOS with keep_output_on_error; kept the base's liveness-check wording in the print_c_stacktraces docstring.

ci/tests/test_print_stacktraces.py: kept the deletion from 3a69545 on antalya-26.8 (ci/tests unit tests removed, no automated CI tests); restoring it would revive ~176 pre-existing lines outside the PR's diff.

Dropped: ci/tests/test_print_stacktraces.py budget tests — file deleted on antalya-26.8 by 3a69545
Source-PR: #2385 (#2385)
@zvonand
zvonand force-pushed the feature/antalya-26.8/pr-2159 branch 3 times, most recently from 82f8845 to eff8803 Compare October 2, 2026 09:33
…-on-listing-keys

Temporary workaround: do not probe object-storage listing keys as JSON

Source-PR: #2393 (#2393)
@zvonand
zvonand force-pushed the feature/antalya-26.8/pr-2159 branch from eff8803 to e330851 Compare October 2, 2026 15:01
@zvonand zvonand added ai-resolved Port conflict auto-resolved by Claude and removed ai-needs-attention Releasy stopped on a conflict it could not resolve — needs human review labels Oct 2, 2026
@zvonand
zvonand marked this pull request as ready for review October 2, 2026 15:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-resolved Port conflict auto-resolved by Claude antalya antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions releasy Created/managed by RelEasy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants