You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit eee2667
Browse filesBrowse the repository at this point in the historyBrowse files
authored and
Gayathri Srividya Rajavarapu
committed
fix: include NULL in delete predicate for evolved partition fields
dynamic_partition_overwrite builds a delete predicate from the current
partition spec. After spec evolution, data files written under older
specs may carry NULL for fields that were added later. The predicate
category=A AND region=us never matches those files (region=NULL ≠ us),
so they are silently left behind after the overwrite.
Fix: detect which fields in the current spec are absent from at least one
historical spec (i.e., were added via evolution), and extend the predicate
for those fields to also accept NULL:
category=A AND (region=us OR region IS NULL)
This ensures pre-evolution data files are included in the delete pass
while files from other partitions (region=eu, etc.) are correctly kept.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
0 commit comments