Skip to content

fix: incremental append and partition_by support for Iceberg Python#662

Merged
yotahk merged 11 commits intoaws-samples:mainfrom
akmalsoliev:fix_incremental_append_partition_by
May 1, 2026
Merged

fix: incremental append and partition_by support for Iceberg Python#662
yotahk merged 11 commits intoaws-samples:mainfrom
akmalsoliev:fix_incremental_append_partition_by

Conversation

@akmalsoliev
Copy link
Copy Markdown
Contributor

@akmalsoliev akmalsoliev commented Feb 13, 2026

Description

  • Add incremental append strategy for Iceberg Python models using INSERT INTO instead of falling through to CREATE OR REPLACE (full refresh)
  • Support partition_by config in Python models for both full refresh (PARTITIONED BY clause in CREATE OR REPLACE) and incremental append (Iceberg partition evolution via ALTER TABLE ADD/DROP PARTITION FIELD)
  • Guard iceberg/s3tables schema change temp table creation with language != 'python' check to prevent SQL-only code from running in Python models
  • Remove silent error fallback (CREATE TABLE IF NOT EXISTS) in favor of proper error propagation with traceback

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Comment thread dbt/include/glue/macros/materializations/python/python_utils.sql Outdated
@akmalsoliev akmalsoliev force-pushed the fix_incremental_append_partition_by branch from f720454 to 48c78d2 Compare March 26, 2026 15:15
@akmalsoliev
Copy link
Copy Markdown
Contributor Author

Hey @yotahk,
took a look and understood what you mean, the fix has been implemented, also created tests, but they shouldn't be added as it doesn't render dbt/include/glue/macros/materializations/python/python_utils.sql, generated with Claude Code for TDD.
Thanks,
Akmal

@akmalsoliev akmalsoliev force-pushed the fix_incremental_append_partition_by branch from 3e56fbf to 64d8ece Compare March 27, 2026 13:11
@yotahk
Copy link
Copy Markdown
Collaborator

yotahk commented Mar 28, 2026

thanks, will take a look next week.

@akmalsoliev
Copy link
Copy Markdown
Contributor Author

thanks, will take a look next week.

Hey @yotahk,
could we run CI to see if this works? Wanted to see if integration tests work

@yotahk yotahk added the enable-functional-tests This label enable functional tests label Mar 31, 2026
@akmalsoliev
Copy link
Copy Markdown
Contributor Author

Hey @yotahk, just noticed that you run the CI, fixed the indentation error.

@yotahk yotahk added enable-functional-tests This label enable functional tests and removed enable-functional-tests This label enable functional tests labels Apr 15, 2026
@yotahk
Copy link
Copy Markdown
Collaborator

yotahk commented Apr 15, 2026

Thanks for the update, I kicked the CI again. Let's see...

@akmalsoliev
Copy link
Copy Markdown
Contributor Author

akmalsoliev commented Apr 16, 2026

Thanks for the update, I kicked the CI again. Let's see...

Hey @yotahk, everything looks ok?

If yes, I'll remove the test file and update the changelog

Copy link
Copy Markdown
Collaborator

@yotahk yotahk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding one comment. Also, would you update CHANGELOG.md?

Comment thread dbt/include/glue/macros/materializations/python/python_utils.sql
@akmalsoliev
Copy link
Copy Markdown
Contributor Author

@yotahk all done

@yotahk
Copy link
Copy Markdown
Collaborator

yotahk commented Apr 17, 2026

@akmalsoliev Would you also fix this too?

created tests, but they shouldn't be added as it doesn't render dbt/include/glue/macros/materializations/python/python_utils.sql, generated with Claude Code for TDD.

@akmalsoliev
Copy link
Copy Markdown
Contributor Author

@akmalsoliev Would you also fix this too?

created tests, but they shouldn't be added as it doesn't render dbt/include/glue/macros/materializations/python/python_utils.sql, generated with Claude Code for TDD.

yeah, mb, skipped that.

@akmalsoliev
Copy link
Copy Markdown
Contributor Author

hey @yotahk, pinging to see if there are any updates.

@yotahk
Copy link
Copy Markdown
Collaborator

yotahk commented May 1, 2026

Hey sorry I didn't have much time to review last week. Diff looks good to me, kicked CI.

@yotahk yotahk merged commit 43efb9f into aws-samples:main May 1, 2026
21 checks passed
@yotahk
Copy link
Copy Markdown
Collaborator

yotahk commented May 1, 2026

Merged, thanks for your patience.

@akmalsoliev akmalsoliev deleted the fix_incremental_append_partition_by branch May 1, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beginning-contributor enable-functional-tests This label enable functional tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants