Skip to content

Iceberg REST export creates spec-id=0 (empty) + spec-id=1 partitioned, default-spec-id stays 0 #7068

@anais-source

Description

@anais-source

Description

When using Paimon 1.3.1 with Iceberg REST export, a partitioned Paimon table generates Iceberg metadata with:

  • spec-id = 0 (empty)
  • spec-id = 1 (partitioned)
  • while default-spec-id remains 0.

As a result, Iceberg readers interpret the table as unpartitioned, and the presence of multiple specs introduces partition evolution even for a freshly created table.

This behavior blocks downstream systems such as StarRocks from creating partitioned materialized views, as StarRocks rejects Iceberg tables that expose partition evolution.

Environment

  • Paimon: 1.3.1
  • Flink: 2.2
  • Gravitino REST: 1.1.0
  • S3-compatible storage
  • Iceberg REST catalog

Minimal reproduction

1) Create a Paimon table with Iceberg REST export:

CREATE TABLE analytics.test_paimon_bug (
  id BIGINT,
  time_day DATE,
  PRIMARY KEY (id, time_day) NOT ENFORCED
)
PARTITIONED BY (time_day)
WITH (
  'bucket' = '1',
  'metadata.iceberg.storage' = 'rest-catalog',
  'metadata.iceberg.rest.uri' = 'http://<gravitino>/iceberg',
  'metadata.iceberg.rest.warehouse' = 'icebergCatalog',
  'metadata.iceberg.rest.clients' = '1'
);
INSERT INTO analytics.test_paimon_bug VALUES
  (1, DATE '2026-01-16'),
  (2, DATE '2026-01-17');

2) Check Iceberg metadata via REST:

GET /iceberg/v1/icebergCatalog/namespaces/analytics/tables/test_paimon_bug
Output (trimmed)

{
  "default_spec": 0,
  "specs": [
    {
      "spec-id": 0,
      "fields": []
    },
    {
      "spec-id": 1,
      "fields": [
        {
          "name": "time_day",
          "transform": "identity",
          "source-id": 1,
          "field-id": 1000
        }
      ]
    }
  ]
}

Expected

  • One partition spec with time_day
  • default-spec-id pointing to that spec
  • No partition evolution on a fresh table

Actual

  • default-spec-id = 0
  • spec-id = 0 empty
  • spec-id = 1 with time_day

Additional context

This issue does not appear to be related to Gravitino or the Iceberg REST catalog itself.

When creating an Iceberg-native table directly in the same Gravitino Iceberg REST catalog (using the same storage and catalog configuration):

{
  "default_spec": 0,
  "specs": [
    {
      "spec-id": 0,
      "fields": [
        {
          "name": "time_day",
          "transform": "identity",
          "source-id": 2,
          "field-id": 1000
        }
      ]
    }
  ]
}
  • The table is created with a single partition spec
  • default-spec-id correctly points to the partitioned spec
  • No partition evolution is present

This indicates that the dual-spec metadata (empty spec-id=0 + partitioned spec-id=1 with default-spec-id=0) is produced at table creation time by Paimon’s Iceberg export.

Possible cause

It appears that Paimon initializes Iceberg tables with an empty default partition spec (spec-id=0), and then adds the partitioned spec as spec-id=1 without updating default-spec-id.

Question

Is it possible for Paimon to:

  • initialize the table directly with the partitioned spec as the default spec, or
  • set default-spec-id to the partitioned spec at creation time to avoid partition evolution?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions