ArrowReader returns nested field values for Parquet files without field ids or a name mapping, where Iceberg Java returns NULL #3436

Apache Iceberg Rust version

main @ b154603a710d730d2f43f8af4db257cc137237ef

Describe the bug

When a Parquet data file has no field ids and the table has no name mapping (schema.name-mapping.default), ArrowReader returns the values stored in the file for every nested field. Iceberg Java returns NULL for them. This covers struct fields, including structs inside lists and maps.

Iceberg Java's fallback, ParquetSchemaUtil.addFallbackIds, assigns ids by position to the top-level fields only. Its readers then match nested struct fields by id. SparkParquetReaders.ReadBuilder.struct, for example, registers a reader only for a file field that has an id. A nested field of such a file has none, so every expected field reads as NULL. The spec's column projection rules also give null for a field id the file doesn't contain when there is no name mapping.

iceberg-rust assigns the same top-level ids (add_fallback_field_ids_to_arrow_schema). Then two things differ:

• get_arrow_projection_mask_fallback reads each projected top-level column whole, including its nested content.
• RecordBatchTransformer then matches the file's id-less nested fields to the table's by position when the counts line up. This was kept in fix(arrow): reconcile evolved nested structs by field id, not position #3255 as "a compatibility fallback until recursive name mapping (Use schema visitor in ArrowReader name mapping #1845)". fix(reader): apply name mapping to nested fields #3384 now applies a name mapping at every depth, so the positional match only matters for files with neither ids nor a mapping, and that is the case where Java returns NULL.

So the values come back even when the file's nested field names differ from the table's.

Found through Apache DataFusion Comet, whose native Iceberg scan uses ArrowReader: apache/datafusion-comet#6794. With Spark 4.1 and Iceberg 1.11, for files added with SparkTableUtil.importSparkTable (no name mapping):

The table is (id INT, s STRUCT<a: INT, b: INT>, items ARRAY<STRUCT<x: INT>>, m MAP<STRING, STRUCT<v: INT>>). With the file's struct fields named p and q instead of a and b, Spark still returns NULL for s.a and s.b, while the iceberg-rust read returns the values by position.

To Reproduce

The existing test test_read_parquet_without_field_ids_with_struct shows it. It writes a file without field ids holding person STRUCT<name, age>, reads it with no name mapping, and asserts person.name is "Alice"/"Bob" and person.age is 30/25. It passes on main. For the same file, Iceberg Java returns a person struct whose name and age are NULL.

cargo test -p iceberg --lib test_read_parquet_without_field_ids_with_struct

Expected behavior

For a file without field ids and no name mapping, top-level columns keep their positional fallback ids, and nested fields match by id. Since such a file has no nested ids, struct fields read as NULL, as in Iceberg Java. In Java's readers a list or map is matched to its element, key and value by position, so a top-level ARRAY<INT> or MAP<STRING, INT> still reads its values. Only fields inside a struct come back NULL. A list whose element is a struct, with no nested leaf read, comes back empty in Spark.

test_read_parquet_without_field_ids_with_struct encodes the current behavior and would change with a fix.

Willingness to contribute

I would be willing to contribute a fix for this bug with guidance from the Iceberg community