Preserve Iceberg column order on schema change#672
Open
lawofcycles wants to merge 2 commits intoaws-samples:mainfrom
Open
Preserve Iceberg column order on schema change#672lawofcycles wants to merge 2 commits intoaws-samples:mainfrom
lawofcycles wants to merge 2 commits intoaws-samples:mainfrom
Conversation
…change Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>
Contributor
|
hey @lawofcycles, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #531
Description
When columns are added to an Iceberg table via
on_schema_change='sync_all_columns'or'append_new_columns', they are currently always appended to the end of the table regardless of their position in the source model. This PR preserves the source column order by usingALTER TABLE ADD COLUMN ... AFTERinstead ofALTER TABLE ADD COLUMNS.For Iceberg/S3Tables incremental models, a new macro
glue__process_schema_changes_with_column_orderreplaces the call toprocess_schema_changesin the incremental materialization. It walks the source column list to determine the correct position for each new column and issues individualALTER TABLE ADD COLUMN col type AFTER prev_col(orFIRST) statements. Non-Iceberg models fall back to the existingprocess_schema_changesbehavior.Changes:
dbt/include/glue/macros/adapters.sql: Addglue__process_schema_changes_with_column_ordermacrodbt/include/glue/macros/materializations/incremental/incremental.sql: Call the new macro for Iceberg schema changestests/functional/adapter/test_iceberg.py: AddTestIcebergColumnOrderOnSchemaChangewith single and multi-column addition test casesVerified on a live AWS Glue environment that columns are inserted at the correct positions.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.