Skip to content

strip_metadata_from_schema fails on list and map columns #3297

Description

@NoahKusaba

Apache Iceberg Rust version

main (8cb2ade), 0.10.1

Describe the bug

iceberg::arrow::strip_metadata_from_schema returns an error for any Arrow schema that contains a list or map column:

Unexpected => Field stack underflow in list

MetadataStripVisitor remembers each field's name and nullability by pushing onto field_stack in before_field, and pops it when rebuilding the field in list, map, struct and primitive. But visit_list and the map arm of visit_type call before_list_element, before_map_key and before_map_value for the inner fields, not before_field. The visitor doesn't override those hooks, so rebuilding a list element or a map key or value pops from an empty stack.

The only test of this function is the doctest, which uses a flat schema, so this wasn't caught.

Downstream, datafusion-iceberg calls this function to validate INSERTs into partitioned tables, so an INSERT into a partitioned table with a list or map column fails (found in apache/datafusion-iceberg#19).

To Reproduce

use std::sync::Arc;

use arrow_schema::{DataType, Field, Schema};
use iceberg::arrow::strip_metadata_from_schema;

let schema = Schema::new(vec![Field::new(
    "l",
    DataType::List(Arc::new(Field::new("element", DataType::Int32, true))),
    true,
)]);
strip_metadata_from_schema(&schema).unwrap(); // Field stack underflow in list

Expected behavior

It returns the same schema with metadata removed at every level, including list elements and map keys and values.

Willingness to contribute

I can contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions