-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Description
When using Paimon 1.3.1 with Iceberg REST export, a partitioned Paimon table generates Iceberg metadata with:
- spec-id = 0 (empty)
- spec-id = 1 (partitioned)
- while default-spec-id remains 0.
As a result, Iceberg readers interpret the table as unpartitioned, and the presence of multiple specs introduces partition evolution even for a freshly created table.
This behavior blocks downstream systems such as StarRocks from creating partitioned materialized views, as StarRocks rejects Iceberg tables that expose partition evolution.
Environment
- Paimon: 1.3.1
- Flink: 2.2
- Gravitino REST: 1.1.0
- S3-compatible storage
- Iceberg REST catalog
Minimal reproduction
1) Create a Paimon table with Iceberg REST export:
CREATE TABLE analytics.test_paimon_bug (
id BIGINT,
time_day DATE,
PRIMARY KEY (id, time_day) NOT ENFORCED
)
PARTITIONED BY (time_day)
WITH (
'bucket' = '1',
'metadata.iceberg.storage' = 'rest-catalog',
'metadata.iceberg.rest.uri' = 'http://<gravitino>/iceberg',
'metadata.iceberg.rest.warehouse' = 'icebergCatalog',
'metadata.iceberg.rest.clients' = '1'
);
INSERT INTO analytics.test_paimon_bug VALUES
(1, DATE '2026-01-16'),
(2, DATE '2026-01-17');
2) Check Iceberg metadata via REST:
GET /iceberg/v1/icebergCatalog/namespaces/analytics/tables/test_paimon_bug
Output (trimmed)
{
"default_spec": 0,
"specs": [
{
"spec-id": 0,
"fields": []
},
{
"spec-id": 1,
"fields": [
{
"name": "time_day",
"transform": "identity",
"source-id": 1,
"field-id": 1000
}
]
}
]
}
Expected
- One partition spec with
time_day default-spec-idpointing to that spec- No partition evolution on a fresh table
Actual
default-spec-id = 0spec-id = 0emptyspec-id = 1withtime_day
Additional context
This issue does not appear to be related to Gravitino or the Iceberg REST catalog itself.
When creating an Iceberg-native table directly in the same Gravitino Iceberg REST catalog (using the same storage and catalog configuration):
{
"default_spec": 0,
"specs": [
{
"spec-id": 0,
"fields": [
{
"name": "time_day",
"transform": "identity",
"source-id": 2,
"field-id": 1000
}
]
}
]
}
- The table is created with a single partition spec
default-spec-idcorrectly points to the partitioned spec- No partition evolution is present
This indicates that the dual-spec metadata (empty spec-id=0 + partitioned spec-id=1 with default-spec-id=0) is produced at table creation time by Paimon’s Iceberg export.
Possible cause
It appears that Paimon initializes Iceberg tables with an empty default partition spec (spec-id=0), and then adds the partitioned spec as spec-id=1 without updating default-spec-id.
Question
Is it possible for Paimon to:
- initialize the table directly with the partitioned spec as the default spec, or
- set
default-spec-idto the partitioned spec at creation time to avoid partition evolution?