Support for write default and initial default #3250

Activity

1. dannycjones

   added

   enhancementNew feature or requestNew feature or request

   on Sep 18, 2026
2. dannycjones commented on Sep 18, 2026

   

   dannycjones

   on Sep 18, 2026

   Last edited by dannycjones

   ContributorAuthor

   More actions

   Tying this to V3: #2411

   I see a bunch of this is already implemented, this is to track how much is actually implemented and what gaps remain, such that we can call V3 complete, tested, and spec compliant. Thanks all!

   Edit to document what I've found so far (2025-10-02):

   • Serde is done
   • We validate schema evolution to check required columns specify defaults
   • We appear to populate initial default only for projected primitive types, not complex types
     • I have not verified tests exist either
     • This was discovered again in fix(arrow): reconcile evolved nested structs by field id, not position #3255, which improves the error message for non-primitives
     • This change appears to change the case from error to nulls fix: fill schema-added nested columns with typed NULL arrays on read #2635
   • Correctness issue, initial default not populated for predicates: Rows are dropped when a filter references a column that is missing from the data file but has an identity partition value or initial-default #3298
   • We DO NOT populate write defaults (I don't think there's a nice way to allow nulls in the Arrow batches written for a required column yet)
   • We do not perform any validation on the defaults content
   • We need some strengthening of the validation on the values when constructing NestedField values. API: Add fallible NestedField builder #3190 refactors this area of the code to facilitate this.
   • There's no UpdateColumnDefault action. feat: Support default values in UpdateSchema #1357
   • iceberg-java does not appear to fully implement the spec. When gating what defaults we allow, we may want to be more conservative as iceberg-rust spec-compliant schema evolutions could be entirely unreadable by Java.

   To triage:

   • DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803
3. dannycjones

   changed the title [-]Support for default and initial value[/-] [+]Support for write default and initial default[/+] on Sep 18, 2026
4. dannycjones commented on Sep 21, 2026

   

   dannycjones

   on Sep 21, 2026

   ContributorAuthor

   More actions

   I'd like to take ownership of this one, I've been prototyping it a bunch. However if anyone would really like to take it, ping me, I don't want to block this.
5. dannycjones commented on Sep 22, 2026

   

   dannycjones

   on Sep 22, 2026

   ContributorAuthor

   More actions

   FYI @blackmwk @CTTY. I plan to have a lot of the validation evaluated late for default values rather than at construction. In other words, we can load the table with a bad default just fine but only on read or write might it fail.

   The reason for this being that tables with bad defaults may otherwise be working for the use case: projecting columns that don't have bad defaults, writing to files where the column is always provided so default is never needed, etc..

   I'm raising this as this feels to be heading in the opposite direction of other changes, like validating construction of types allowing them only to be constructed or deserialized if they are valid. I'm wondering what your take is here.
6. CTTY commented on Sep 22, 2026

   

   CTTY

   on Sep 22, 2026

   Collaborator

   More actions

   Hi Danny, thanks for jumping on this and the detailed analysis.

   I plan to have a lot of the validation evaluated late for default values rather than at construction

   Late validation seems ok to me. Loading a table shouldn't fail because of a bad default on a column the query never touches.

   Just to make sure I understand this correctly: we still want to validate the default values that we write in iceberg-rust, so we ensure that iceberg-rust doesn't produce metadata that we couldn't read, correct?
7. dannycjones commented on Sep 23, 2026

   

   dannycjones

   on Sep 23, 2026

   ContributorAuthor

   More actions

   Hi Danny, thanks for jumping on this and the detailed analysis.

   I plan to have a lot of the validation evaluated late for default values rather than at construction

   Late validation seems ok to me. Loading a table shouldn't fail because of a bad default on a column the query never touches.

   Just to make sure I understand this correctly: we still want to validate the default values that we write in iceberg-rust, so we ensure that iceberg-rust doesn't produce metadata that we couldn't read, correct?

   That's nuanced. I want to validate anything we change in iceberg-rust - so if a column has a bad default on load, we'll still write it into new metadata. But if we construct a TableMetadata with an updated schema or use the update schema actions, those will validate so we don't introduce bad defaults.

   The deferred defaults is going to be a bit more involved than I thought to defer all validation, since deserialization will flag things like a string instead of an int. I'll have a think on if we want to defer all of it or not.