Support aggregate function states in Parquet and Iceberg

docs/en/engines/table-engines/integrations/iceberg.md

@@ -106,6 +106,72 @@ The following table shows how Iceberg data types are mapped to ClickHouse data t
 | `map` | `Map` |
 | `struct` | `Tuple` |
 
+### Aggregate function states {#aggregate-function-states}
+
+Iceberg has no aggregate-state type, so ClickHouse stores the two aggregate-state types as ordinary
+Iceberg values and records the ClickHouse type name in a `clickhouse.type` key on the schema field:
+
+| ClickHouse type | Iceberg type | Stored as |
+|---|---|---|
+| `AggregateFunction(f, T...)` | `binary` | The serialized state, the same bytes `f` writes with the `-State` combinator |
+| `SimpleAggregateFunction(f, T)` | whatever `T` maps to | An ordinary value of type `T` |
+
+Other query engines ignore the key and see a plain `binary` (or `T`-typed) column.
+
+Creating such a table requires
+[`allow_experimental_aggregate_function_states_in_iceberg`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_iceberg),
+and so does every query that reads a column whose `clickhouse.type` names an `AggregateFunction` - an
+`INSERT` or an `ALTER TABLE ... EXPORT PART` as much as a `SELECT`. A state is an opaque blob handed
+to the deserializer of the aggregate function the table's metadata names, so with the setting enabled
+it is that metadata, not the query, which chooses the deserializer; keep it disabled for tables from
+untrusted sources, where such a field is then rejected rather than read as `String`. A
+`SimpleAggregateFunction` column is not gated on read, holding ordinary values of its storage type.
+
+The setting is read from the query that parses the schema, so a session `SET` or a `SETTINGS` clause
+on that query supplies it. The `iceberg*` table functions build a fresh storage object per query, so
+for them it takes effect per query; the table engine parses the schema once and keeps it, so the
+query that first touches the table after `ATTACH` decides, and that outcome holds - including for
+queries that do not set it - until `DETACH TABLE` or a server restart.
+
+Writing additionally requires
+[`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet),
+since the data files are written by the ordinary Parquet writer. An object storage table engine
+freezes its format settings at `CREATE TABLE` - the server settings plus that query's `SETTINGS`
+clause, session settings ignored - so for `INSERT` the Parquet setting belongs there and has no
+effect if given on the `INSERT` instead:
+
+```sql
+CREATE TABLE agg (k UInt32, u AggregateFunction(uniq, UInt64), s SimpleAggregateFunction(sum, UInt64))
+ENGINE = IcebergLocal('/path/to/table/')
+PARTITION BY k
+SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1,
+         allow_experimental_aggregate_function_states_in_parquet = 1;
+
+-- The states merge exactly as they do in an AggregatingMergeTree table.
+SELECT k, uniqMerge(u), sum(s) FROM agg GROUP BY k
+SETTINGS allow_experimental_aggregate_function_states_in_iceberg = 1;
+```
+
+`ALTER TABLE ... EXPORT PART` and `ALTER TABLE ... EXPORT PARTITION` from an `AggregatingMergeTree`
+table into such an Iceberg table work as well, so a partition of pre-aggregated states can be moved
+into the lake without finalizing it. Both take the settings off the `ALTER` query itself:
+
+```sql
+ALTER TABLE mt EXPORT PART 'all_1_1_0' TO TABLE agg
+SETTINGS allow_experimental_aggregate_function_states_in_parquet = 1;
+
+-- EXPORT PARTITION, which only ReplicatedMergeTree implements, records both values in its manifest,
+-- so every replica executing the task applies them in place of its own profile.
+ALTER TABLE rmt EXPORT PARTITION ID '1' TO TABLE agg
+SETTINGS allow_experimental_aggregate_function_states_in_parquet = 1,
+         allow_experimental_aggregate_function_states_in_iceberg = 1;
+```
+
+The Parquet footer of each data file carries the same information, so `DESCRIBE file('data.parquet')`
+reports the aggregate types too; that path goes through Parquet schema inference and needs the
+Parquet setting rather than the Iceberg one. See
+[aggregate function states in Parquet](/interfaces/formats/Parquet#aggregate-function-states).
+
 ## Schema evolution {#schema-evolution}
 ClickHouse supports reading Iceberg tables whose schema has evolved over time. This includes tables where columns have been added, removed, or reordered, as well as columns changed from required to nullable. Additionally, the following type casts are supported:
 

docs/en/interfaces/formats/Parquet/Parquet.md

@@ -95,6 +95,41 @@ On write, top-level columns of type `Point`, `LineString`, `Polygon`, `MultiLine
 
 Geometry columns must appear at the root of the schema or nested inside `Tuple` (`struct`); nesting them inside `Array` or `Map` is not supported. `Nullable` is not supported for geo columns either.
 
+## Aggregate function states {#aggregate-function-states}
+
+Parquet has no aggregate-state type, so ClickHouse writes an [`AggregateFunction`](/sql-reference/data-types/aggregatefunction.md) column as a plain `BYTE_ARRAY` holding the serialized state - the same bytes the `-State` combinator produces - with no `STRING`/`UTF8` logical type, since a state is arbitrary binary data rather than text. A [`SimpleAggregateFunction(f, T)`](/sql-reference/data-types/simpleaggregatefunction.md) column is written as an ordinary value of type `T`.
+
+Neither type can be recovered from the Parquet schema alone: every `AggregateFunction` state is just a binary column, and a `SimpleAggregateFunction` is indistinguishable from its storage type. ClickHouse therefore records the type names in a `clickhouse.column_types` key in the file-level Parquet metadata, as a JSON object mapping column name to ClickHouse type name. The recorded name includes the state version, which is what pins the serialized layout across server versions.
+
+Both writing an `AggregateFunction` column and reconstructing one from that metadata on read are gated by [`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet), which is disabled by default. A state is an opaque blob passed to the deserializer of whichever aggregate function the file names, so with the setting enabled it is the file, not the query, choosing that deserializer; keep it disabled for files from untrusted sources, where such a file is then rejected rather than read as `String`. With it disabled, writing is refused with `UNKNOWN_TYPE`, exactly as in versions that did not support states in Parquet at all. `SimpleAggregateFunction` is gated in neither direction: it is stored as an ordinary value of its storage type and has always been written that way.
+
+With the setting enabled the states round-trip without any hint:
+
+```sql
+SET allow_experimental_aggregate_function_states_in_parquet = 1;
+
+INSERT INTO FUNCTION file('states.parquet')
+SELECT k, uniqState(v) AS u, sumSimpleState(v) AS s FROM source GROUP BY k;
+
+DESCRIBE file('states.parquet');
+-- k  UInt64
+-- u  AggregateFunction(uniq, UInt64)
+-- s  SimpleAggregateFunction(sum, UInt64)
+
+SELECT k, uniqMerge(u), sum(s) FROM file('states.parquet') GROUP BY k;
+```
+
+An explicit structure overrides the recorded metadata and is unaffected by the setting, so a state column can also be read as `String` to get the raw serialized bytes:
+
+```sql
+SELECT uniqMerge(CAST(u AS AggregateFunction(uniq, UInt64)))
+FROM file('states.parquet', Parquet, 'u String');
+```
+
+Reading fails rather than guessing if the recorded type does not describe the data actually in the file. Min/max statistics are never used for pruning a state column, because bounds over serialized states are meaningless.
+
+Other query engines see a plain binary (or `T`-typed) column and ignore the metadata key. The same mechanism backs [aggregate-state support in Iceberg tables](/engines/table-engines/integrations/iceberg#aggregate-function-states), whose data files are Parquet.
+
 ## Example usage {#example-usage}
 
 ### Inserting data {#inserting-data}

src/Core/Settings.cpp

@@ -7433,6 +7433,34 @@ Query Iceberg table using the specific snapshot id.
 )", 0) \
     DECLARE(Bool, allow_experimental_geo_types_in_iceberg, false, R"(
 Allow parsing Iceberg `geometry` and `geography` field types as ClickHouse `Geometry` (Variant) type.
+)", 0) \
+    DECLARE(Bool, allow_experimental_aggregate_function_states_in_iceberg, false, R"(
+Allow `AggregateFunction` and `SimpleAggregateFunction` columns in Iceberg tables, both when creating
+a table and when reading one. Writing additionally requires
+[`allow_experimental_aggregate_function_states_in_parquet`](/operations/settings/settings#allow_experimental_aggregate_function_states_in_parquet),
+since the data files are Parquet.
+
+An `AggregateFunction` state is stored as an opaque `binary` value with its ClickHouse type name in
+the schema field's `clickhouse.type` key. Honouring that key means the table's metadata, not the
+query, chooses the deserializer the stored bytes are handed to, so keep this disabled for tables from
+untrusted sources; a field recording such a type is then rejected rather than read as `String`.
+`SimpleAggregateFunction` is not gated on read, holding ordinary values of its storage type.
+
+The setting is read from the query that parses the table's schema. [More about aggregate function
+states in Iceberg](/engines/table-engines/integrations/iceberg#aggregate-function-states).
+)", 0) \
+    DECLARE(Bool, allow_experimental_aggregate_function_states_in_parquet, false, R"(
+Allow `AggregateFunction` states in Parquet files, both when writing and when inferring a schema.
+While disabled, writing such a column is refused with `UNKNOWN_TYPE`.
+
+A state is written as an opaque `BYTE_ARRAY`, with the ClickHouse type name - including the state
+version, which pins the serialized layout - recorded in the `clickhouse.column_types` file metadata
+key. Reconstructing the type from that key means the file, not the query, chooses the deserializer
+the stored bytes are handed to, so keep this disabled for files from untrusted sources; a file
+recording such a type is then rejected rather than read as `String`.
+
+Neither direction affects an explicitly given structure, nor `SimpleAggregateFunction`. [More about
+aggregate function states in Parquet](/interfaces/formats/Parquet#aggregate-function-states).
 )", 0) \
     DECLARE(Bool, show_data_lake_catalogs_in_system_tables, false, R"(
 Enables showing data lake catalogs in system tables.

src/Core/SettingsChangesHistory.cpp

@@ -42,6 +42,8 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory()
         addSettingsChanges(settings_changes_history, "26.6.2.20001.altinityantalya",
         {
             {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."},
+            {"allow_experimental_aggregate_function_states_in_iceberg", false, false, "New setting gating aggregate function states in Iceberg tables, on creation and on read. Disabled by default, so such a column keeps being refused with `SUPPORT_IS_DISABLED` as in versions without the feature."},
+            {"allow_experimental_aggregate_function_states_in_parquet", false, false, "New setting gating `AggregateFunction` states in Parquet files, on write and in schema inference. Disabled by default, so writing such a column keeps throwing `UNKNOWN_TYPE` as in versions without the feature."},
         });
 
         addSettingsChanges(settings_changes_history, "26.6",