Apache Iceberg Rust version
main (8cb2ade), 0.10.1
Describe the bug
iceberg::arrow::strip_metadata_from_schema returns an error for any Arrow schema that contains a list or map column:
Unexpected => Field stack underflow in list
MetadataStripVisitor remembers each field's name and nullability by pushing onto field_stack in before_field, and pops it when rebuilding the field in list, map, struct and primitive. But visit_list and the map arm of visit_type call before_list_element, before_map_key and before_map_value for the inner fields, not before_field. The visitor doesn't override those hooks, so rebuilding a list element or a map key or value pops from an empty stack.
The only test of this function is the doctest, which uses a flat schema, so this wasn't caught.
Downstream, datafusion-iceberg calls this function to validate INSERTs into partitioned tables, so an INSERT into a partitioned table with a list or map column fails (found in apache/datafusion-iceberg#19).
To Reproduce
use std::sync::Arc;
use arrow_schema::{DataType, Field, Schema};
use iceberg::arrow::strip_metadata_from_schema;
let schema = Schema::new(vec![Field::new(
"l",
DataType::List(Arc::new(Field::new("element", DataType::Int32, true))),
true,
)]);
strip_metadata_from_schema(&schema).unwrap(); // Field stack underflow in list
Expected behavior
It returns the same schema with metadata removed at every level, including list elements and map keys and values.
Willingness to contribute
I can contribute a fix for this bug independently
Apache Iceberg Rust version
main (8cb2ade), 0.10.1
Describe the bug
iceberg::arrow::strip_metadata_from_schemareturns an error for any Arrow schema that contains a list or map column:MetadataStripVisitorremembers each field's name and nullability by pushing ontofield_stackinbefore_field, and pops it when rebuilding the field inlist,map,structandprimitive. Butvisit_listand the map arm ofvisit_typecallbefore_list_element,before_map_keyandbefore_map_valuefor the inner fields, notbefore_field. The visitor doesn't override those hooks, so rebuilding a list element or a map key or value pops from an empty stack.The only test of this function is the doctest, which uses a flat schema, so this wasn't caught.
Downstream, datafusion-iceberg calls this function to validate INSERTs into partitioned tables, so an INSERT into a partitioned table with a list or map column fails (found in apache/datafusion-iceberg#19).
To Reproduce
Expected behavior
It returns the same schema with metadata removed at every level, including list elements and map keys and values.
Willingness to contribute
I can contribute a fix for this bug independently