REST Catalog CTAS fails against Lakekeeper: staged create commit sends SetLocation update

[eperrella]

Summary

Dremio REST Catalog writes fail against Lakekeeper during CTAS/CREATE TABLE commit. Source registration and namespace discovery work, but the write path fails because Dremio's create-table transaction emits a table-location update that Lakekeeper rejects.

This appears to be a Dremio REST Catalog writer compatibility issue, not an S3/SeaweedFS connectivity issue.

Environment

• Dremio image: ghcr.io/devais-eng/dremio-oss:1.2
• Catalog: Lakekeeper 0.12.2
• Storage: SeaweedFS S3-compatible gateway
• Dremio namespace: polaris-seaweedfs-smoke
• Lakekeeper namespace: lakekeeper-seaweedfs-smoke

Reproduction

1. Bootstrap Lakekeeper.
2. Create an S3-compatible warehouse and namespace demo.
3. Register a Dremio RESTCATALOG source:

{
  "entityType": "source",
  "type": "RESTCATALOG",
  "name": "lakekeeper",
  "config": {
    "restEndpointUri": "http://lakekeeper.lakekeeper-seaweedfs-smoke.svc.cluster.local:8181/catalog",
    "propertyList": [
      { "name": "warehouse", "value": "warehouse" },
      { "name": "rest.auth.type", "value": "none" }
    ]
  }
}

4. Dremio reports state.status=good and discovers the namespace.
5. Run:

CREATE TABLE lakekeeper.demo.dremio_rest_smoke AS
SELECT 1 AS id, 42 AS marker;

Actual Result

Dremio fails during WRITER_COMMITTER:

SYSTEM ERROR: BadRequestException: Malformed request: Cannot change table location
Caused By: org.apache.iceberg.exceptions.BadRequestException: Malformed request: Cannot change table location
  org.apache.iceberg.rest.RESTTableOperations.commit(...)
  org.apache.iceberg.rest.DremioRESTTableOperations.commit(...)
  org.apache.iceberg.BaseTransaction.commitCreateTransaction(...)
  com.dremio.exec.store.iceberg.model.IcebergBaseCommand.endTransaction(...)
  com.dremio.exec.store.iceberg.model.IcebergTableCreationCommitter.commit(...)

Lakekeeper logs show:

error.type=SetLocationNotAllowed
error.message=Cannot change table location

Investigation Notes

Lakekeeper behavior is intentional. It assigns table UUID/location during create according to the warehouse storage profile/layout, and on commit it rejects TableUpdate::SetLocation unless it is exactly equal to the existing table metadata location.

Dremio bytecode inspection of the running fork image suggests the issue is in the create-table flow:

• RESTSessionCatalog$Builder.createTransaction() performs REST stageCreate() and receives catalog-assigned metadata.
• IcebergBaseCommand.beginCreateTableTransaction() then constructs fresh metadata with TableMetadata.newTableMetadata(..., getTableLocation(), ...).
• That produces a commit containing SetLocation, which strict catalogs like Lakekeeper reject.

Testing a Lakekeeper warehouse with name-based storage-layout still fails with the same error, so this is not only a path-layout mismatch.

Expected Result

CTAS should complete through a strict Iceberg REST Catalog implementation such as Lakekeeper.

CREATE TABLE lakekeeper.demo.dremio_rest_smoke AS SELECT 1 AS id, 42 AS marker;
SELECT COUNT(*) AS c, SUM(marker) AS marker_sum FROM lakekeeper.demo.dremio_rest_smoke;

Expected result: c=1, marker_sum=42.

Suggested Fix Direction

For RESTCATALOG creates:

• Treat the staged metadata returned by the REST catalog as authoritative.
• Do not regenerate table UUID/location in IcebergBaseCommand.beginCreateTableTransaction() for REST Catalog sources.
• Avoid emitting SetLocation during create commit unless it exactly matches the staged table metadata location after normalization.
• Verify that Dremio does not emit a new AssignUUID, because Lakekeeper also rejects UUID reassignment after stage-create.