Bug description
Description of Bug
The Apache Iceberg sink routing logic (write_data in router/mod.rs) contains severe bugs in partition key construction:
Hardcoded Zero/Dummy Partition Values: primitive_type_to_literal returns static zero/empty values (PrimitiveLiteral::Int(0), PrimitiveLiteral::String(String::new()), etc.). When passed into get_partition_type_value and PartitionKey::new(...), every batch is written under a single epoch-zero/dummy partition regardless of payload contents.
Empty PartitionSpec Construction: The code calls PartitionSpec::builder(table.current_schema_ref()).build(), which creates an unpartitioned spec (Spec ID 0) rather than retrieving the active spec via table.metadata().default_partition_spec().
Missing Record Fanout: DataFileWriter expects a single PartitionKey per writer instance. There is currently no logic to evaluate partition transforms against incoming RecordBatch data or group records across target partition writers.
Area
Connectors (iggy-connector-sdk / Iceberg connector) — High-Risk Area
Steps to Reproduce
Point the Iceberg sink to an Iceberg table partitioned by a field (e.g., identity(date) or day(event_timestamp)).
Produce JSON messages with varying timestamp/date payload fields into Iggy.
Observe sink output: all records end up in epoch 0 / default fallback partition paths.
Affected area / component
Connectors
Deployment
Docker (self-built image)
Versions
No response
Hardware / environment
No response
Sample code
No response
Logs
No response
Iggy server config
No response
Reproduction
No response
Contribution
Good first issue
Bug description
Description of Bug
The Apache Iceberg sink routing logic (write_data in router/mod.rs) contains severe bugs in partition key construction:
Hardcoded Zero/Dummy Partition Values: primitive_type_to_literal returns static zero/empty values (PrimitiveLiteral::Int(0), PrimitiveLiteral::String(String::new()), etc.). When passed into get_partition_type_value and PartitionKey::new(...), every batch is written under a single epoch-zero/dummy partition regardless of payload contents.
Empty PartitionSpec Construction: The code calls PartitionSpec::builder(table.current_schema_ref()).build(), which creates an unpartitioned spec (Spec ID 0) rather than retrieving the active spec via table.metadata().default_partition_spec().
Missing Record Fanout: DataFileWriter expects a single PartitionKey per writer instance. There is currently no logic to evaluate partition transforms against incoming RecordBatch data or group records across target partition writers.
Area
Connectors (iggy-connector-sdk / Iceberg connector) — High-Risk Area
Steps to Reproduce
Point the Iceberg sink to an Iceberg table partitioned by a field (e.g., identity(date) or day(event_timestamp)).
Produce JSON messages with varying timestamp/date payload fields into Iggy.
Observe sink output: all records end up in epoch 0 / default fallback partition paths.
Affected area / component
Connectors
Deployment
Docker (self-built image)
Versions
No response
Hardware / environment
No response
Sample code
No response
Logs
No response
Iggy server config
No response
Reproduction
No response
Contribution
Good first issue