Antalya 26.6: Added support for iceberg v3 unknown data type

[subkanthi]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support for iceberg v3 unknown datatype which is maps to Nullable(Nothing) in Clickhouse. Read and write path(Parquet).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [0966bc5]

[subkanthi]

Create table in ice

Altinity/ice#219

create-table ns1.t1 --format-version=3   --schema '[{"name":"id","type":"int","required":true},{"name":"payload","type":"unknown"}]'

CH

show create table ice.`ns1.t1`;

SHOW CREATE TABLE ice.`ns1.t1`

Query id: 482b62d6-9778-4392-b504-8d5e437292a8

   ┌─statement─────────────────────────────────────────────────┐
1. │ CREATE TABLE ice.`ns1.t1`                                ↴│
   │↳(                                                        ↴│
   │↳    `id` Int32,                                          ↴│
   │↳    `payload` Nullable(Nothing)                          ↴│
   │↳)                                                        ↴│
   │↳ENGINE = Iceberg('http://localhost:9000/bucket1/ns1/t1/') │
   └───────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.012 sec. 

WRITE PATH

Ubuntu-2404-noble-amd64-base :) insert into ice.`ns1.t1` values(1, null);

INSERT INTO ice.`ns1.t1` FORMAT Values

Query id: 4169dc98-cebc-46be-81ab-41a0a45acfbe

Ok.

1 row in set. Elapsed: 1.169 sec. 

Ubuntu-2404-noble-amd64-base :) select * from ice.`ns1.t1`;

SELECT *
FROM ice.`ns1.t1`

Query id: 9df62c95-2ca1-4749-a98b-8728e6ac135b

   ┌─id─┬─payload─┐
1. │  1 │ ᴺᵁᴸᴸ    │
   └────┴─────────┘

1 row in set. Elapsed: 0.035 sec. 

[xieandrew]

An edge case and potential optimization I found, other than that it looks good.

[xieandrew]

It looks like this doesn't check for Nothing type inside nested types (tuple, array, or map), so those nested Nothing values are not filtered for the writer. It would be good to check if that causes the parquet writer to fail.

[xieandrew]

The std::find on every column/chunk could removed if the constructor computes a list of column indices to keep instead of nothing_column_indices. Then you would only need to iterate the kept column indices and directly add columns[i] to filtered_columns.

[subkanthi]

AI audit note: This review comment was generated by AI.

Audit update for PR #2363 (Iceberg v3 unknown type: read mapping, metadata mapping, Parquet write path)

Confirmed defects

Medium: Complex column with an unknown subfield loses all of its real data on write, silently

• Impact: A struct or map that mixes real fields with an unknown subfield is dropped from the data file as a whole column. For example, nested = (5, NULL) for Tuple(a Nullable(Int64), u Nullable(Nothing)) reads back as (NULL, NULL). There is no error or warning.

• Anchor: src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp, the constructor, which uses containsNothing per top-level column.

• Trigger: A v3 table containing struct<a: long, u: unknown>, followed by an INSERT with a non-NULL value for a.

• Why defect: The only thing that cannot be serialized is the Nothing leaf. The sibling values are valid, the user supplied them, and they are discarded.

  if (!containsNothing(sample_block->getByPosition(i).type))
      kept_column_indices.push_back(i);   // whole Tuple excluded if any leaf is Nothing

[xieandrew]

It looks like this excludes the entire column from being written, even if other nested fields are not the Nothing type. Only the unknown field in the nested should be stripped otherwise there might be data loss.

[subkanthi]

yes its this AI finding #2363 (comment)

[xieandrew]

This insert should write the struct containing the unknown e.g. (42, NULL) and the next SELECT should check that the non-unknown nested field is properly written.

[Selfeer]

I don't think these are mentioned here anywhere. Can you please check?

Medium: Promoting unknown to a real type is rejected, and reads of files written before a promotion do not apply it.

Iceberg v3 allows unknown to be promoted to any type. checkValidSchemaEvolution only allows int to long, float to double, and a wider decimal with the same scale, so ALTER ... MODIFY COLUMN from Nullable(Nothing) to a concrete type throws Iceberg spec doesn't allow schema evolution. On the read path, allowPrimitiveTypeConversion has the same gap: getSchemaTransformationDag leaves the old Nullable(Nothing) node in the transform when the current schema type is different. A file written while the column was unknown is therefore not cast to the promoted type.

Medium: unknown is written into tables whose format-version is still 2.

unknown exists only in Iceberg format version 3. getIcebergType maps Nothing to "unknown" with no version check, and both createEmptyMetadataFile and generateAddColumnMetadata store that type on the table's current format-version. iceberg_format_version defaults to 2, so creating a table with Nullable(Nothing), or adding that column to an existing v2 table, produces metadata other v2 readers are not required to accept.

Medium: An insert into a table that has a top-level unknown column drops min/max bounds for every column, and records a column size for a column that was not written.

MultipleFileWriter::consume updates data-file statistics from the full chunk, including the Nullable(Nothing) column, and only afterwards strips that column from the Parquet file. The column is all nulls, so getExtremes stores POSITIVE_INFINITY. canDumpIcebergStats treats that as null and canWriteStatistics then skips lower_bounds and upper_bounds for the whole file, including columns such as Int64 and String that would otherwise have bounds. column_sizes is still written for the unknown field id, using the in-memory size of a column that the writer removed from the data file.

Medium: A write that strips every column produces a Parquet file the reader rejects.

If every top-level column contains Nothing (a table whose only column is unknown, or whose only column is a struct/list/map that containsNothing drops entirely), kept_column_indices is empty and the Parquet file is written with zero columns. SchemaConverter::checkHasColumns then throws INCORRECT_DATA (Parquet file has no columns / Schema root has no children) before missing columns can be filled with nulls. The insert succeeds and the following select fails.