Repository navigation
Struct reads as NULL when all requested children are missing from the file and another column is projected #3378
Description
Apache Iceberg Rust version
main @ 7f9691fc7ed3
Describe the bug
If none of a struct's requested children exist in a data file, get_arrow_projection_mask_with_field_ids selects no leaf under the struct. When another column is also projected, the struct is treated as absent and reads as NULL for every row, including rows where the file has a non-null struct. When the struct is the only projected column, the mask falls back to ProjectionMask::all() and the struct validity is right. (On main the child values in that case are also wrong, because of the positional matching #3255 fixes.)
Same mask behavior as #3374, but a different trigger: here the file has field ids and the struct's children were all added after it was written, so recursive name mapping (#1845) won't fix it. Iceberg Java's ParquetValueReaders.structReader reads a column under the struct to get its presence for this case.
Came up in review of #3255.
To Reproduce
The file has {1: id int, 2: s struct<5: x int>}, with s null in row 1. The table has {1: id int, 2: s struct<6: x int>} (x dropped and re-added). Added to the tests in crates/iceberg/src/arrow/reader/projection.rs:
#[tokio::test]
async fn struct_with_all_children_missing() {
use arrow_array::{Array, Int32Array, StructArray};
use arrow_buffer::NullBuffer;
let with_id = |f: Field, id: &str| {
f.with_metadata(HashMap::from([(PARQUET_FIELD_ID_META_KEY.to_string(), id.to_string())]))
};
let s_col = StructArray::new(
vec![with_id(Field::new("x", DataType::Int32, true), "5")].into(),
vec![Arc::new(Int32Array::from(vec![10, 20, 30])) as ArrayRef],
Some(NullBuffer::from(vec![true, false, true])),
);
let file_schema = Arc::new(ArrowSchema::new(vec![
with_id(Field::new("id", DataType::Int32, false), "1"),
with_id(Field::new("s", s_col.data_type().clone(), true), "2"),
]));
let table_schema = Arc::new(
Schema::builder()
.with_schema_id(2)
.with_fields(vec![
NestedField::required(1, "id", Type::Primitive(PrimitiveType::Int)).into(),
NestedField::optional(2, "s", Type::Struct(StructType::new(vec![
NestedField::optional(6, "x", Type::Primitive(PrimitiveType::Int)).into(),
])))
.into(),
])
.build()
.unwrap(),
);
let tmp_dir = TempDir::new().unwrap();
let path = format!("{}/1.parquet", tmp_dir.path().to_str().unwrap());
let batch = RecordBatch::try_new(file_schema.clone(), vec![
Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef,
Arc::new(s_col) as ArrayRef,
])
.unwrap();
let mut writer = ArrowWriter::try_new(File::create(&path).unwrap(), file_schema, None).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
for projected in [vec![2], vec![1, 2]] {
let reader = ArrowReaderBuilder::new(FileIO::new_with_fs(), Runtime::current()).build();
let tasks = Box::pin(futures::stream::iter(vec![Ok(FileScanTask::builder()
.with_file_size_in_bytes(std::fs::metadata(&path).unwrap().len())
.with_start(0)
.with_length(0)
.with_data_file_path(path.clone())
.with_data_file_format(DataFileFormat::Parquet)
.with_schema(table_schema.clone())
.with_project_field_ids(projected.clone())
.with_case_sensitive(false)
.build()
.unwrap())])) as FileScanTaskStream;
let out = reader.read(tasks).unwrap().stream().try_collect::<Vec<RecordBatch>>().await.unwrap();
let s = out[0].column(projected.len() - 1).as_struct();
let validity: Vec<bool> = (0..3).map(|i| s.is_valid(i)).collect();
// Passes for [2], fails for [1, 2] with [false, false, false].
assert_eq!(validity, vec![true, false, true], "projection {projected:?}");
}
}Expected behavior
s validity is [true, false, true] for both projections, with x null in every row.
Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
Activity
fix(arrow): address final review on nested field-id promotion
fix(arrow): reconcile evolved nested structs by field id, not position (