Skip to content

Preserve Iceberg column order on schema change#672

Open
lawofcycles wants to merge 2 commits intoaws-samples:mainfrom
lawofcycles:fix/iceberg-column-order
Open

Preserve Iceberg column order on schema change#672
lawofcycles wants to merge 2 commits intoaws-samples:mainfrom
lawofcycles:fix/iceberg-column-order

Conversation

@lawofcycles
Copy link
Copy Markdown
Contributor

resolves #531

Description

When columns are added to an Iceberg table via on_schema_change='sync_all_columns' or 'append_new_columns', they are currently always appended to the end of the table regardless of their position in the source model. This PR preserves the source column order by using ALTER TABLE ADD COLUMN ... AFTER instead of ALTER TABLE ADD COLUMNS.

For Iceberg/S3Tables incremental models, a new macro glue__process_schema_changes_with_column_order replaces the call to process_schema_changes in the incremental materialization. It walks the source column list to determine the correct position for each new column and issues individual ALTER TABLE ADD COLUMN col type AFTER prev_col (or FIRST) statements. Non-Iceberg models fall back to the existing process_schema_changes behavior.

Changes:

  • dbt/include/glue/macros/adapters.sql: Add glue__process_schema_changes_with_column_order macro
  • dbt/include/glue/macros/materializations/incremental/incremental.sql: Call the new macro for Iceberg schema changes
  • tests/functional/adapter/test_iceberg.py: Add TestIcebergColumnOrderOnSchemaChange with single and multi-column addition test cases

Verified on a live AWS Glue environment that columns are inserted at the correct positions.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…change

Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
@akmalsoliev
Copy link
Copy Markdown
Contributor

hey @lawofcycles,
does it happen to also cover the issue #673?
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep the column order for Iceberg table

2 participants