Remove object_storage_cluster_join_mode=local

[ianton-ru]

Changelog category (leave one):

• Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Remove object_storage_cluster_join_mode=local
Solved #1827

Improved default 'allow' behavior to work in swarm mode (when remote nodes do not have knowledge about tables).

Improved 'global' mode to work with old analyzer.

Documentation entry for user-facing changes

'local' is a legacy setting, behavior is the same as 'allow' after changes in upstream.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [0f70f78]

[ianton-ru]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 317db7ca6f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve initiator-local joins for legacy local mode

When local is selected for an object-storage cluster query that joins a table existing only on the initiator—the scenario in tests/integration/test_s3_cluster/test.py::test_joins—returning here leaves the join in query_to_send, which is dispatched to every shard; for example, s0_1_0 cannot resolve the test's join_table. The previous LOCAL path stripped the join and forced FetchColumns so it ran on the initiator, whereas the new ALLOW behavior requests WithMergeableState; therefore existing local queries and the integration test fail rather than retaining compatible behavior.

Useful? React with 👍 / 👎.

[ianton-ru]

Good point!

[chatgpt-codex-connector]

Format the allow setting value as inline code

Replace the prose quotation 'allow' with the inline-code form `allow`. This is a literal ClickHouse setting value, and the repository requires such literals in documentation and comments to be wrapped in inline code.

AGENTS.md reference: AGENTS.md:L7-L7

Useful? React with 👍 / 👎.

[ianton-ru]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f10d74d7e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not globalize expression aliases used as IN sets

When allow_experimental_analyzer=0 and object_storage_cluster_join_mode='global', a valid constant-set alias such as WITH (1, 2) AS vals ... WHERE key IN vals remains an ASTIdentifier; this branch converts it to globalIn, but GlobalSubqueriesMatcher::Data::addExternalStorage accepts only an ASTTableIdentifier, ASTSubquery, or ASTTableExpression, so analysis throws WRONG_GLOBAL_SUBQUERY instead of evaluating the set. Restrict this rewrite to actual table/subquery operands or resolve expression aliases first.

Useful? React with 👍 / 👎.

[ianton-ru]

This reproduction does not fail. rewriteASTInFunctionsToGlobalIn runs before TreeRewriter. For WITH (1, 2) AS vals ... WHERE key IN vals, the RHS is still an ASTIdentifier at that point, so it is rewritten to globalIn. Then QueryNormalizer inlines the WITH alias to the constant tuple, and GlobalSubqueriesVisitor sees a literal/function RHS and converts globalIn back to in. addExternalStorage is not invoked, so there is no WRONG_GLOBAL_SUBQUERY.
The suggested fix — skip ASTIdentifier and only rewrite table/subquery operands — would be incorrect here. At rewrite time, IN join_table is also an ASTIdentifier; MarkTableIdentifiersVisitor has not run yet. Dropping that case would skip the rewrite that object_storage_cluster_join_mode=global needs for initiator-local tables.
No code change for this comment.

[ianton-ru]

@blau-ai

[blau-ai]

CI triage for #2228

Verdict: 0 PR-caused failures. Three reds on this run — the PR aggregate gate plus two leaf checks — and both leaf failures are infra/flaky/pre-existing, unrelated to this change. All functional suites that ran (Fast test, all Stateless variants incl. cas s3 storage, builds amd/arm) are green on the substance.

Head SHA analyzed: 40e4fcd90298595f68b81d5ac94292e94acf8908 (from result_pr.json).

---

1. Stateless tests (amd_binary, cas s3 storage, parallel) — FAIL → flaky/infra, not PR-caused

• Single failing test: 01880_materialized_view_to_table_type_check (Failed: 1, Passed: 11015).
• The report's own auto-diagnosis: reran the test 48 times with the same randomized settings → 48 passed, 0 failed → "All reruns passed. The failure is not reproducible (likely a transient issue)." The result is tagged flaky.
• Decisive log line:

  Code: 499. DB::Exception: ... Poco::Exception. Code: 1000, e.code() = 111,
  Connection refused ..., bucket test, key cas_s3/cas/ref_catalog, object size 311262. (S3_ERROR)
  (query: CREATE TABLE test_input(id Int32) ENGINE=MergeTree() order by id;)
  

  The failure is a transient S3 connection-refused during a plain CREATE TABLE ... ENGINE=MergeTree, i.e. the CAS/S3 test backend, not this PR's code path (object-storage-cluster JOIN rewriting).
• This check is also flagged do_not_block_pipeline_on_failure: true, so it does not gate the pipeline.
• Next step: safe to re-run; no code change needed. (The test exercises MergeTree table creation and is unrelated to object_storage_cluster_join_mode.)

2. GrypeScanServer (-alpine) / Grype Scan — FAIL → pre-existing/infra, not PR-caused

• "Grype Scan Completed with 7 high/critical vulnerabilities" — this is a CVE scan of OS packages inside the built clickhouse-server:...-alpine image.
• This PR changes only src/Core/{Settings.cpp,SettingsEnums.h}, src/Interpreters/InterpreterSelectQuery.cpp, src/Storages/IStorageCluster.{cpp,h} and four tests/integration/** files — no Dockerfiles, no contrib/ submodules, no dependency/package changes. A code change to JOIN handling cannot add or remove image-level CVEs, so this is a branch-wide/base-image condition, not a regression from this PR.
• Next step: not actionable from this PR. If it must go green it needs a base-image / package bump (separate dependency PR), independent of this change.

3. PR — FAIL → aggregate gate only

• This is the roll-up status of the child checks above; it carries no independent error. It clears once the two items above are re-run/waived.

---

Health check

The change is well-scoped and its functional CI is clean: Fast test (9313 passed), all Stateless variants including cas storage, cas s3 storage (the sole fail being the flaky S3 one above), s3 storage, debug parallel/sequential, and both amd & arm release/binary/debug builds are green. Nothing in the red checks points at the JOIN-mode rewrite logic.

One coverage note (not a failure): the integration suites this PR actually edits — test_s3_cluster, test_storage_iceberg_with_spark/test_cluster_joins.py, test_storage_iceberg_with_spark/test_remote_initiator.py, test_database_iceberg — show as skipped in this pipeline (sanitizer/arch tag exclusions), so the behavioral change to object_storage_cluster_join_mode isn't directly validated by this run. If you want that coverage before merge, it's worth kicking off the relevant integration job.

Bottom line: nothing to fix in the PR. Re-run the cas s3 storage stateless job to clear the flake; the Grype finding is a pre-existing image scan unrelated to this diff.

_{🤖 automated CI triage by @blau-ai — evidence from praktika result_pr.json for the head SHA.}

[Selfeer]

High: With the old analyzer, ORDER BY ALL on an allow or local cluster join crashes, and GROUP BY ALL aggregates on the shards before the join.

rewriteQueryForInitiatorLocalJoin calls removeJoin, which clears the ORDER BY and GROUP BY clauses but leaves order_by_all and group_by_all set. The non-join branch of the same function sets both flags to false. The join branch does not. IStorageCluster::read then builds the remote query with InterpreterSelectQuery, and TreeRewriter still honors the flags. expandOrderByAll dereferences orderBy(), which is now null (enable_order_by_all defaults to true), so SELECT ... FROM s3Cluster(...) JOIN local_table ORDER BY ALL with allow_experimental_analyzer = 0 crashes. GROUP BY ALL is expanded onto the stripped select list, so shards collapse duplicate left rows before the initiator joins and the joined row count is smaller than it should be.

[Selfeer]

ORDER BY ALL cluster join crash

Build: 26.6.2.20001.altinityantalya (40e4fcd90298595f68b81d5ac94292e94acf8908), log swarms/test.log.

Scenario /swarms/feature/order by all join/order by all with old analyzer failed. The same join through plain s3(), with allow_experimental_analyzer = 0, returned the expected rows. The following s3Cluster query segfaulted the server. GROUP BY ALL did not run, because the process was already dead.

Received signal Segmentation fault (11)
Address: 0x8. Access: read. Address not mapped to object.

Query:

SELECT t1.k, t1.v, t2.name
FROM s3Cluster('replicated_cluster', ...) AS t1
INNER JOIN order_by_all_dim_... AS t2 ON t1.k = t2.k
ORDER BY ALL
SETTINGS allow_experimental_analyzer = 0, object_storage_cluster_join_mode = 'allow'

query_id e60d7936-7b18-484e-8366-6f817e95d7b7. Settings recorded on the fatal line include allow_experimental_analyzer = false and object_storage_cluster_join_mode = 'allow'.

The fault is a read at address 0x8, which is a null object plus a small member offset. ClickHouse printed the symbolized stack starting at frame 2 (TreeRewriter::analyzeSelect); frame 1 is the signal handler and was not given a symbol. expandOrderByAll is inlined into analyzeSelect, so it has no frame of its own. The client then got Code: 32. ATTEMPT_TO_READ_AFTER_EOF while reading the response, because the server had already died.

Received signal Segmentation fault (11)
Address: 0x8. Access: read. Address not mapped to object.
Stack trace: 0x000000001ea64827 0x000000001e61ad9d 0x000000001e613a4c 0x000000001e60e454 0x000000001e60e219 0x000000001f20fced 0x000000001e625bad 0x000000001e6231f6 0x000000001e6633ab 0x000000001e6644fa 0x000000001eb20152 0x000000001eb1a9dd 0x000000002047f8a0 0x00000000204a66a4 0x0000000025fde687 0x0000000025fdeb26 0x0000000025f9be9c 0x0000000025f9a44f 0x0000766251d67ac3 0x0000766251df9850

2. DB::TreeRewriter::analyzeSelect(boost::intrusive_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000001ea64827
3. DB::InterpreterSelectQuery::InterpreterSelectQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000001e61ad9d
4. DB::InterpreterSelectQuery::InterpreterSelectQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001e613a4c
5. DB::InterpreterSelectQuery::InterpreterSelectQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x000000001e60e454
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000001e60e219
7. DB::IStorageCluster::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000001f20fced
8. DB::InterpreterSelectQuery::executeImpl(DB::QueryPlan&, std::optional<DB::Pipe>) @ 0x000000001e625bad
9. DB::InterpreterSelectQuery::buildQueryPlan(DB::QueryPlan&) @ 0x000000001e6231f6
10. DB::InterpreterSelectWithUnionQuery::buildQueryPlan(DB::QueryPlan&) @ 0x000000001e6633ab
11. DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001e6644fa
12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001eb20152
13. DB::executeQuery(std::basic_string_view<char, std::char_traits<char>>, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001eb1a9dd
14. DB::TCPHandler::runImpl() @ 0x000000002047f8a0
15. DB::TCPHandler::run() @ 0x00000000204a66a4
16. Poco::Net::TCPServerConnection::start() @ 0x0000000025fde687
17. Poco::Net::TCPServerDispatcher::run() @ 0x0000000025fdeb26
18. Poco::PooledThread::run() @ 0x0000000025f9be9c
19. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000025f9a44f
20. ? @ 0x0000000000094ac3
21. ? @ 0x0000000000126850

[ianton-ru]

Crash should be fixed