You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Share Parquet field-ID and name resolution between remap_physical_schema and match_struct_fields #6978
Parquet field resolution by field ID and by case-insensitive name is implemented twice, once for top-level columns and once for nested struct fields, and the two must be kept in sync by hand:
remap_physical_schema mixes four concerns in one function: duplicate-ID validation (L207-228), building the id-to-logical map (L231-241), name folding, and a per-field closure with three precedence stages (ID match, fake-name shadowing, case-insensitive name match). The renamed-field construction (Field::new(logical.name(), ...).with_metadata(..) plus name_map.insert) is copy-pasted between the ID path (L262-272) and the name path (L305-315). The closure uses early returns and mutates name_map, fake_counter and occupied_names, so the precedence rules that mirror Spark's matchIdField are hard to follow or unit test.
Because this file is still patched often for Spark-compatibility fixes, a fix to one path that misses the other could reintroduce a schema-evolution bug for nested columns only.
Extract a field resolver shared by both paths that returns an enum such as {IdMatch, Shadowed, NameMatch, Unchanged}, so each precedence stage can be unit tested against Spark's behaviour.
Extract check_duplicate_field_ids, build_id_to_logical, a rename_field helper used by both the ID and name branches, and a small FakeNameGenerator holding occupied_names and fake_counter.
Use the shared resolver from match_struct_fields.
Split create and rewrite into named steps (e.g. check_duplicate_columns, id_duplicate_roots, resolve_rebase_policies, restore_physical_names), keeping the Spark-reference comments with the extracted helpers.
Optionally move the roughly 1,900 lines of tests in schema_adapter.rs into a separate file.
Activity