fix: infer missing nested Parquet fields as nullable

[rgbuilds]

Which issue does this PR close?

• Closes Parquet schema inference keeps missing nested fields non-nullable #26172.

Rationale for this change

When a required child is missing from one Parquet file but its containing struct exists in both, inference leaves the child non-nullable. Selecting the child fails, and an IS NULL predicate can incorrectly return no rows. With this change, selecting s.y returns 10 and NULL, and WHERE s.y IS NULL returns the row from the file without y.

What changes are included in this PR?

Implementation

• Extend partial-field nullability correction recursively through structs, lists, and large lists.
• Compare children only across schemas containing their parent. Missing parents become nullable without weakening required children or list elements; missing struct children within present elements become nullable.
• Preserve existing nullability and metadata. Other container types retain existing merge behavior.

Performance

• Optimized synthetic benchmarks against the exact fix: infer a Parquet column as nullable when some files lack it #26104 head measured 2.4–3.0% additional schema-inference time for evolving nested schemas: 32 files × 128 children took 1.973 → 2.033 ms (approximately 0.060 ms extra); 256 files × 128 children took 16.072 → 16.459 ms (approximately 0.387 ms extra).
• Flat-schema aggregate medians showed no consistent inference penalty.
• Registered-table query medians varied from −0.15% to +1.12%, using universally present columns because missing-child queries fail on the baseline.
• Measurements used an Apple M4 Mac Mini, release-nonlto, in-memory Parquet files, warmed metadata, and three process runs per variant. These synthetic results do not establish production query impact.
• The change adds inference traversal and allocations, with no additional footer reads or scan-time correction pass.
• Benchmark methodology and detailed results — synthetic data, experiment setup, timings, and limitations. (Benchmark preparation and report drafting assisted by Codex)

What is the testing strategy for this PR?

• Unit tests cover absent parents, independent nested paths, metadata, and required list elements.

• A Parquet integration regression checks struct/list/large-list projection and IS NULL across 12 configurations (metadata and pushdown on/off). It fails on the unfixed dependency head.

• SQL regression verifies the missing child's projected NULL and the matching IS NULL row.

• Fresh extended workspace validation passed all Rust library/integration/bin tests and all 529 SQL logic files. The 126 extended fuzz tests passed in the initial run; subsequent Rust runs skipped only those already-passing cases. All-target/all-feature Clippy with warnings denied, formatting, and every submission lint check passed. macOS emitted linker unwind-table warnings when linking large test executables.

• After rebasing onto main following the merge of fix: infer a Parquet column as nullable when some files lack it #26104: formatting, the full submission lint suite (including CI-configured Clippy), and extended workspace Rust/SQL tests passed. The all-target/all-feature Clippy command reports three mutable_key_type errors in unchanged upstream optimizer code.

Are there any user-facing changes?

Inferred nested fields missing from some containing schemas become nullable, allowing missing values to be read and IS NULL queries to return correct rows. Explicit schemas and public APIs are unchanged.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.12977% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.80%. Comparing base (b295a67) to head (0c0fe4c).

Files with missing lines	Patch %	Lines
datafusion/datasource-parquet/src/file_format.rs	93.12%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26195      +/-   ##
==========================================
- Coverage   82.80%   82.80%   -0.01%     
==========================================
  Files        1149     1149              
  Lines      452234   452347     +113     
  Branches   452234   452347     +113     
==========================================
+ Hits       374493   374581      +88     
- Misses      54946    54969      +23     
- Partials    22795    22797       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rgbuilds]

@zhuqi-lucas This builds on #26104 and addresses the nested-field case we discussed, tracked in #26172. The fix and tests are in the final commit, 278b3e5; local validation and CI passed. Would you take a look when convenient? I’ll rebase once #26104 merges.

I’ve included schema-inference benchmarks in the description and would appreciate feedback on the tradeoff.

[zhuqi-lucas]

This comment is not accurate for Union. Field::try_merge in arrow-schema/src/field.rs recurses into Struct, Union, List and LargeList; Map and FixedSizeList fall into the equality arm, so the comment holds for those. A union whose child is missing from some files merges fine and then lands here, where it is silently left alone, which is the same bug one container away.

Low risk in practice since Parquet cannot carry a union, but the comment states an invariant that does not hold, and the next person will rely on it. Either widen unions too, or name the types that are actually rejected.

[zhuqi-lucas]

The old fast path is gone. It returned early when every field was present in every file, using the precomputed counts, which is the common case of all files sharing a schema. Now only the single-source case returns early, so identical schemas still walk the whole tree, build a HashMap per struct level and allocate a fresh Fields per level. That looks like the 2.4-3.0% the description measures.

Cheap to restore: compare the field name sets across sources first, or short-circuit a level when every widen_field returned the same Arc.

[zhuqi-lucas]

Unguarded map index, so a missing key panics rather than erroring. I could not construct a reachable case, since a merged field name always comes from some source, but that leans on a try_merge invariant instead of enforcing it here. .get() with an empty slice treated as missing is both safer and the semantically right default.

Separately, and this predates the PR so take it or leave it: the loop above pushes per field rather than per source, so a file carrying two fields of the same name contributes two occurrences. occurrences.len() < sources.len() can then be false even though another file lacks the field, leaving it required. Counting distinct sources would fix it.