Antalya 26.6: Added support for alter add column with first/after

[subkanthi]

closes: #2091

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added support for the FIRST and AFTER clauses with ALTER TABLE ADD COLUMN and ALTER TABLE MODIFY COLUMN for Iceberg tables (DataLakeCatalog).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [60ccd82]

[subkanthi]

 set allow_insert_into_iceberg=1;

SET allow_insert_into_iceberg = 1

Query id: a24c0e73-e0cb-4a2b-9f38-a003b769a98e

Ok.

0 rows in set. Elapsed: 0.007 sec. 

Ubuntu-2404-noble-amd64-base :) alter table ice.`flowers.sample` add column value1 Nullable(Int64) first;

ALTER TABLE ice.`flowers.sample`
    (ADD COLUMN `value1` Nullable(Int64) FIRST )

Query id: a83c2a54-6343-4114-a876-655889a30e9e

Ok.

0 rows in set. Elapsed: 0.182 sec. 

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.sample`;

SHOW CREATE TABLE ice.`flowers.sample`

Query id: 0e533740-a3f7-435e-bfa8-b0fddfaae688

   ┌─statement─────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.sample`                                ↴│
   │↳(                                                                ↴│
   │↳    `value1` Nullable(Int64),                                    ↴│
   │↳    `value` Nullable(Int64),                                     ↴│
   │↳    `boolean_col` Nullable(Bool),                                ↴│
   │↳    `dec_col_2` Nullable(Decimal(10, 2)),                        ↴│
   │↳    `value2` Nullable(Int64),                                    ↴│
   │↳    `value3` Nullable(Int64)                                     ↴│
   │↳)                                                                ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/sample/') │
   └───────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.017 sec. 

[subkanthi]

alter table ice.`flowers.sample` modify column value2 Nullable(Int64) first;

ALTER TABLE ice.`flowers.sample`
    (MODIFY COLUMN `value2` Nullable(Int64) FIRST )

Query id: a96ecd40-d7c7-4e4d-b6b9-1a62453b4418

Connecting to database ice at localhost:9008 as user default.
Connected to ClickHouse server version 26.6.2.

Ok.

0 rows in set. Elapsed: 0.258 sec. 

Ubuntu-2404-noble-amd64-base :) show create table ice.`flowers.sample`;

SHOW CREATE TABLE ice.`flowers.sample`

Query id: 901ee53f-8b02-4b95-b4d7-e2de4a8992f1

   ┌─statement─────────────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`flowers.sample`                                ↴│
   │↳(                                                                ↴│
   │↳    `value2` Nullable(Int64),                                    ↴│
   │↳    `value1` Nullable(Int64),                                    ↴│
   │↳    `value` Nullable(Int64),                                     ↴│
   │↳    `boolean_col` Nullable(Bool),                                ↴│
   │↳    `dec_col_2` Nullable(Decimal(10, 2)),                        ↴│
   │↳    `value11` Nullable(Int64),                                   ↴│
   │↳    `value3` Nullable(Int64),                                    ↴│
   │↳    `value6` Nullable(Int64)                                     ↴│
   │↳)                                                                ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/flowers/sample/') │
   └───────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.013 sec. 

[DimensionWieldr]

CI triage

Not caused by this PR. Iceberg FIRST/AFTER tests passed (test_writes_modify_column_position.py, Iceberg regression, unit tests). Run 33461281948 / SHA d9e23e0e.

Category	What
This PR	none
Pre-existing flaky	test_concurrent_create_drop_race_condition (timeout; 26 fails / 9 PRs upstream); 00661_optimize_final…, 01055_compact_parts (0 fails on diagnosis); 02265_column_ttl (CAS S3 Code 210)
Infrastructure	asan session-timeout (3/8: 496 passed, 0 failed); Spark iceberg cluster Connection refused (schema_inference); fake GCS UNCERTAIN (test_cas_gcs); tiered_storage docker teardown hang; 5h cancelled CAS jobs; empty 10‑min GitHub placeholders
Unrelated (test suite)	settings snapshot missing export_merge_tree_part_ignore_extra_source_columns / export_merge_tree_part_schema_match_mode after #2220 — will fail every antalya-26.6 PR until the 26.6_antalya snapshot is updated

Approve from a #2289 perspective. Optional: rerun integration 3/8 and 4/8; fix settings snapshots in clickhouse-regression separately.

[DimensionWieldr]

Tests in Altinity/clickhouse-regression@a7fcec8 for FIRST / AFTER are passing on Glue and Ice rest catalogs.

LGTM

[DimensionWieldr]

Build (amd_debug) and Build (amd_asan_ubsan) both fail compiling gtest_iceberg_metadata_generator.cpp. Calls with only a name and a type are ambiguous:

error: call to member function 'generateAddColumnMetadata' is ambiguous
    gen.generateAddColumnMetadata("z", makeNullable(std::make_shared<DataTypeInt64>()));

Same error at lines 140, 234, 251, and 264. MetadataGenerator.h now declares both of these:

void generateAddColumnMetadata(const String & column_name, DataTypePtr type, bool first = false, const String & after_column = {});
void generateAddColumnMetadata(const String & column_name, DataTypePtr type);

The new overload's default arguments make a two-argument call match the old declaration as well. The .cpp definition was rewritten to the four-argument signature, so the old declaration has no body either.

@subkanthi Maybe drop the two-argument declaration and keep the one with the defaults? The existing tests then resolve to that function. The other builds on this run passed because they do not compile unit_tests_dbms.

[DimensionWieldr]

PR #2289 CI Triage

Run 35752871749, 3b72ad5487. The earlier compile failure is gone: Build (amd_debug) and Build (amd_asan_ubsan) are green after the duplicate generateAddColumnMetadata declaration was removed.

Summary

Category	Count	What
regression	0	—
infrastructure	7	submodule clone, Grype CVE, CAS contention, fuzzer SIGSEGV, two stress jobs
pre-existing-flaky	1	named-collections race timeout
cascade	parents of the rows below	suite wrappers, and Check failed after the asan stress server did not stay up
unknown	4	three lightweight-delete counts, one settings snapshot

Nothing in the remaining list matches this diff (Iceberg ADD/MODIFY FIRST/AFTER).

Infrastructure

• Source upload. git submodule clone failed: could not read Username for 'https://github.com'.
• Grype, keeper and server alpine. Both fail on CVE-2026-85091 (High) in Alpine zlib 1.3.2. The Ubuntu server image passed. Base-image CVE, not this change.
• CAS jobs. Same family of Code 210:
  • cas_alter_attach_3: CAS write could not be committed (ref catalog deadline, then checkpoint contention).
  • cas_selects: content-addressed disk 'cas_disk' -- mount lease not held (TRANSIENT unavailability).
  • 02346_text_index_bug108874: CAS checkpoint contention. The harness reran with the same settings, 4/4 passed.
  • Stress (amd_debug, cas s3 storage): Server died / lost connection. fatal.log is empty, no OOM, and the non-CAS debug stress job on this SHA passed.
  • Stress (amd_asan_ubsan, cas s3 storage): Cannot start clickhouse-server after connection refused. Check failed is the exit of that. The non-CAS asan stress job passed.
• AST fuzzer (amd_debug, targeted, old_compatibility). The debug binary SIGSEGV on clickhouse --query "SELECT 1" while printing Decompressing the binary. No stack. The other debug fuzzer jobs on this SHA finished normally.

Pre-existing flaky

• test_named_collections/test.py::test_concurrent_create_drop_race_condition: pytest timeout at 900s. Upstream over 60 days: 37 fails in 104,289 runs (0.04%). This PR: 1 of 1. The diff does not touch named collections.

Unknown

These block a clean approval until the branch rate is checked. None of them is exercised by this diff.

• Lightweight delete, one run each, concurrent row counts: random delete 75 percent of the table got 251 vs 250; random delete 25 percent of the table without overlap (s3 cache) got 762 vs 750; random delete entire table without overlap got 2 vs 0. Parent suite rows are cascade.
• /settings/default values/iceberg_manifest_min_count_to_compact: SnapshotNotFoundError. The server returned '{"default":"30"}'. That line exists in default values>=26.8.snapshot and is absent from default values>=26.6_antalya.snapshot. This PR does not touch those files. One antalya-26.6 settings run would show whether the branch already fails the same way.

cas_s3_cache_selects was cancelled after 3h 30m in the run step. No report was uploaded.

[DimensionWieldr]

Run 35915202409 on 4d61da54 (merge from antalya-26.6). Builds are green, unit tests passed, and the earlier transient failures came back green. Nothing still red matches the Iceberg FIRST/AFTER change.

The remaining red checks are outside this PR:

• Grype (keeper and alpine server): CVE-2026-85091 in Alpine zlib 1.3.2. The Ubuntu server image passed.
• settings: missing snapshot line for iceberg_manifest_min_count_to_compact in default values>=26.6_antalya.snapshot. The server returns default 30, which is already in the 26.8 snapshot.
• Source upload: submodule clone failed again with could not read Username for 'https://github.com'.
• cas_selects: cas_disk mount lease not held (Code 210).
• cas_lightweight_delete_4: concurrent delete counted 252 rows vs 250. The s3-cache shard passed.
• clickhouse_keeper_failover: could not bring up docker-compose cluster.
• rbac_3: the run step died after about 19 minutes. Shards 1 and 2 passed. No report was uploaded.
• Stateless 02421_truncate_isolation_no_merges and 04053_arrow_view: failed once, then passed on the in-job rerun.

LGTM, just waiting on dev review process

[subkanthi]

AI audit note: This review was generated by AI (gpt-6-sol). The analysis is static; no tests or fault-injection runs were performed locally.

Audit update for PR #2289 (Iceberg ALTER TABLE column positioning):

Confirmed defects

High: A retry of position-only MODIFY COLUMN dereferences a null type; typed retries can falsely succeed.
Impact: A catalog conflict or uncertain commit outcome can terminate the server process for MODIFY COLUMN c FIRST without a type, or report success while leaving a typed column in the wrong position.
Anchor: Mutations.cpp:889,118 → MetadataGenerator.cpp:360-367.
Trigger: A commit attempt is retried. The predicate calls getIcebergType(type, ...) when type may be null; when a type is supplied, it checks only that type, not FIRST or AFTER.
Fix direction: Make the already-applied check understand optional types and verify the requested position before declaring success.
Regression test direction: Inject a conflicting or ambiguously reported catalog commit for both position-only and typed positioning commands.

Medium: Positioning bypasses rejection of an unrecordable type change.
Impact: MODIFY COLUMN c UInt32 FIRST can succeed when Iceberg still records the column as int (and reads it as Int32), despite the requested type change.
Anchor: MetadataGenerator.cpp:807-841.
Trigger: Request a different ClickHouse type with the same Iceberg representation, such as Int32 → UInt32, together with FIRST or AFTER. The existing rejection runs only when !needs_reposition.
Fix direction: Reject unrecordable type changes independently of whether positioning is requested.
Regression test direction: Repeat the existing indistinguishable-type rejection test with each positioning clause.

Medium: AFTER the column itself fails instead of preserving its position.
Impact: An otherwise valid MODIFY COLUMN b ... AFTER b raises an exception.
Anchor: MetadataGenerator.cpp:897-910.
Trigger: The target and after_column are both b; the loop skips b, never marks an insertion, then throws. The shared column-order logic treats this as a no-op.
Fix direction: Handle self-placement as no reposition, while still applying any valid type change.
Regression test direction: Cover self-AFTER with and without a type change.

Coverage summary

• Scope reviewed: All six changed files and their parser → alter validation → storage → schema generator → metadata/catalog commit and retry paths. ADD and MODIFY default, FIRST, AFTER, rejection, and retry transitions were traced.
• Categories failed (static fault injection): placement edges, type-representation parity, commit retry/conflict handling.
• Categories passed (static fault injection): ordinary placement, missing-anchor rejection, local exception rollback, metadata commit boundaries, and unchanged ownership/locking, integer, resource, and logging paths.
• Assumptions/limits: No local execution or runtime interleaving test was performed in read-only Plan mode. The targeted linked CI report records 12 passing positioning tests, but does not exercise the triggers above.

[DimensionWieldr]

I added test coverage for the aforementioned defects. Will test fixes once they are pushed and the build is available.