fix: accept NOT NULL input columns in partitioned INSERT

[NoahKusaba]

Which issue does this PR close?

• Closes INSERT into a partitioned table rejects NOT NULL source columns for optional table columns #27.
• Part of the Ballista-Iceberg integration (follows feat: make PartitionExpr reconstructible #14), but the bug here affects any DataFusion user writing to a partitioned table.

What changes are included in this PR?

An INSERT into a partitioned table fails when the source has a NOT NULL column where the table's column is optional:

Error: Plan("Input schema does not match Iceberg table schema.
Expected schema: Field { "id": nullable Int32 }, Field { "category": nullable Utf8 }
Input schema: Field { "id": Int32 }, Field { "category": Utf8 }")

Every value of a non-nullable column is valid in an optional one, so the write is safe. The same INSERT into an unpartitioned table already succeeds, because project_with_partition returns before this check when the spec is unpartitioned, so today the result depends on whether the table is partitioned.

project_with_partition compared the input and table schemas with ==. It now uses Arrow's Schema::contains, which allows the input to be narrower in nullability and is otherwise as strict as before:

• Same columns, same order, same names, same types (checked recursively for nested types).
• Field metadata is still stripped from both sides before comparing, as before.
• A nullable input column into a required table column is still rejected, at any nesting depth.

The public doc of project_with_partition now states this contract. The error message changes from "does not match" to "is not compatible with", since the schemas no longer need to be equal.

The new unit tests need a partitioned table with a given schema. The three existing schema-validation tests each built one inline with the same ~50 lines, so they now share a table_partitioned_by_id helper, with their imports moved to the top of the tests module. Their assertions are unchanged apart from the new error message.

Are these changes tested?

• test_insert_not_null_source_into_partitioned_table (integration): inserts from a NOT NULL MemTable into a table partitioned on an optional column, and reads the rows back. It fails on main with the error above.
• test_schema_validation_nullability: non-nullable and nullable input into an optional column are both accepted; into a required column, non-nullable is accepted and nullable is rejected.
• test_schema_validation_struct_nullability: the same four cases for a field inside a struct.
• List and map cases are staged but commented out pending strip_metadata_from_schema fails on list and map columns iceberg-rust#3297.
• The existing matching, mismatched-name and metadata tests still pass.

cargo fmt --all -- --check, cargo clippy --workspace --locked --all-targets -- -D warnings and cargo test --workspace --locked all pass locally.

AI Disclosure

• Used Claude Code to find the bug, write the tests and draft this description. I reviewed the change and reproduced the failure on main.

🤖 Generated with Claude Code

[mbutrovich]

Thanks @NoahKusaba! The fix looks right to me, and I confirmed that test_insert_not_null_source_into_partitioned_table fails with main's project.rs and passes with this change.

While testing the partitioned and unpartitioned paths side by side, I found a gap on the unpartitioned side that this PR doesn't cause but that sits next to its motivation. Could you open an issue for it? The unpartitioned path skips project_with_partition (table/mod.rs), and IcebergWriteExec passes the input's own schema as the sink schema to execute_input_stream (write.rs). DataFusion only runs its runtime null check (check_not_null_constraints) for columns that are non-nullable in the sink schema and nullable in the input. Because both schemas are the input's here, the check never runs. At the head commit, inserting SELECT * FROM source into an unpartitioned table with a required id: int column, from a MemTable whose nullable id holds [1, NULL], succeeds and reads back 0, 1. The NULL is written as 0. The partitioned path goes the other way and rejects a nullable source at plan time even when it holds no nulls, which DataFusion's own sinks accept and check at runtime. Passing the table's Arrow schema (plus the partition column) as the sink schema would probably fix both, so the issue could cover both.

[mbutrovich]

The new doc says the rule holds "at any nesting depth", and this test covers a struct field. Could we add the same four cases for a list element and a map value? Lists and maps go through their own arms of Arrow's DataType::contains, and the map arm also compares the keys_sorted flag, so a struct test doesn't cover them.

[NoahKusaba]

Thanks, good call. Adding the list and map cases turned up a bug already on main: iceberg-rust's strip_metadata_from_schema fails on any list or map column with "Field stack underflow in list", so a partitioned INSERT with such a column errors out before the schemas are compared. MetadataStripVisitor only pushes onto its field stack in before_field, which the visitor doesn't call for list elements or map keys and values. Filed as apache/iceberg-rust#3297.

The fix and a regression test are ready. I'll open that PR once my pending iceberg-rust PRs are merged (apache/iceberg-rust#3286, apache/iceberg-rust#2904), since reviews there are bandwidth-limited.

In abee042 the four cases live in a shared assert_nested_nullability helper used by the struct test, and the list and map tests are staged but commented out until the fix lands. They pass against 665c64e with the fix applied, including the map case with keys_sorted = false (what Iceberg maps convert to), so enabling them is just uncommenting.

Two list/map limitations of contains I noticed while writing them, neither new in this PR (the old == had both):

• Field names are compared, so a list whose element is named item (Arrow's default, used by DataType::new_list) won't match Iceberg's element.
• A source map with keys_sorted = true is rejected, since Iceberg maps convert to unsorted Arrow maps.

If you think either is worth handling, I can fold them into #22 or open a separate issue.

[NoahKusaba]

@mbutrovich Opened #22 for the unpartitioned gap, with your repro. It covers both sides:

• Unpartitioned: IcebergWriteExec passes the input's schema as the sink schema to execute_input_stream, so check_not_null_constraints never runs and NULLs land in required columns as 0.
• Partitioned: a nullable source column into a required one is rejected at plan time, even when it holds no NULLs.

I'd like to keep this PR to the narrower fix, accepting NOT NULL sources into optional columns, and leave the plan-time rejection as it is. Once #22 passes the table's schema as the sink schema, that rejection can be dropped in favour of DataFusion's runtime check, and both paths will behave the same.