Skip to content

Use intermediate-and-swap pattern in full-refresh incremental path #398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdebruyn wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Use intermediate-and-swap pattern in full-refresh incremental path #398

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions dbt/include/fabric/macros/materializations/models/incremental/incremental.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,33 @@

{% if existing_relation is none or full_refresh_mode or existing_relation.is_view %}

{% set tmp_vw_relation = target_relation.incorporate(path={"identifier": target_relation.identifier ~ '__dbt_tmp_vw'}, type='view')-%}
-- Dropping temp view relation if it exists
{%- set intermediate_relation = make_intermediate_relation(target_relation) -%}
{%- set backup_relation = make_backup_relation(target_relation, 'table') -%}
{% set tmp_vw_relation = intermediate_relation.incorporate(path={"identifier": intermediate_relation.identifier ~ '__dbt_tmp_vw'}, type='view')-%}

-- Dropping any leftover intermediate / backup / temp view relations from a previous failed run
{{ adapter.drop_relation(intermediate_relation) }}
{{ adapter.drop_relation(backup_relation) }}
{{ adapter.drop_relation(tmp_vw_relation) }}
-- Dropping target relation if exists
{{ adapter.drop_relation(target_relation) }}

-- Build into an intermediate relation. If this fails, the existing target is untouched.
{%- call statement('main') -%}
{{ get_create_table_as_sql(False, target_relation, sql)}}
{{ get_create_table_as_sql(False, intermediate_relation, sql)}}
{%- endcall -%}

-- Dropping temp view relation
{{ adapter.drop_relation(tmp_vw_relation) }}

-- Atomic swap: rename existing target to backup (if any), intermediate to target, then drop backup
{%- set target_existing = load_cached_relation(target_relation) -%}
{% if target_existing is not none %}
{{ adapter.rename_relation(target_existing, backup_relation) }}
{% endif %}
{{ adapter.rename_relation(intermediate_relation, target_relation) }}
{% if target_existing is not none %}
{{ adapter.drop_relation(backup_relation) }}
{% endif %}

-- Add constraints including FK relation.
{{ build_model_constraints(target_relation) }}

Expand Down
Loading