Skip to content

fix(connector-iceberg): partition logic hardcodes dummy values and ignores table spec #3853

Description

@ashudeqode

Bug description

Description of Bug
The Apache Iceberg sink routing logic (write_data in router/mod.rs) contains severe bugs in partition key construction:

Hardcoded Zero/Dummy Partition Values: primitive_type_to_literal returns static zero/empty values (PrimitiveLiteral::Int(0), PrimitiveLiteral::String(String::new()), etc.). When passed into get_partition_type_value and PartitionKey::new(...), every batch is written under a single epoch-zero/dummy partition regardless of payload contents.

Empty PartitionSpec Construction: The code calls PartitionSpec::builder(table.current_schema_ref()).build(), which creates an unpartitioned spec (Spec ID 0) rather than retrieving the active spec via table.metadata().default_partition_spec().

Missing Record Fanout: DataFileWriter expects a single PartitionKey per writer instance. There is currently no logic to evaluate partition transforms against incoming RecordBatch data or group records across target partition writers.

Area
Connectors (iggy-connector-sdk / Iceberg connector) — High-Risk Area

Steps to Reproduce
Point the Iceberg sink to an Iceberg table partitioned by a field (e.g., identity(date) or day(event_timestamp)).

Produce JSON messages with varying timestamp/date payload fields into Iggy.

Observe sink output: all records end up in epoch 0 / default fallback partition paths.

Affected area / component

Connectors

Deployment

Docker (self-built image)

Versions

No response

Hardware / environment

No response

Sample code

No response

Logs

No response

Iggy server config

No response

Reproduction

No response

Contribution

  • I'm willing to submit a pull request to fix this bug

Good first issue

  • I think this could be a good first issue for a new contributor

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconnectorsConnectors runtime

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions