Skip to content

fabric__get_incremental_microbatch_sql ignores unique_key — always delete+insert, never MERGE #393

@sdebruyn

Description

@sdebruyn

Summary

fabric__get_incremental_microbatch_sql always performs a delete-then-insert sequence, even when unique_key is configured on the model. Fabric DW supports native MERGE, which is atomic and avoids the intermediate "empty rows" window that delete-then-insert exposes to concurrent readers.

Evidence (HEAD 0de2190, v1.10.0)

dbt/include/fabric/macros/materializations/models/incremental/merge.sql: the fabric__get_incremental_microbatch_sql macro emits a delete-by-microbatch-predicate followed by an insert, with no branch on unique_key.

User impact

  • For models with unique_key configured, what should be one atomic MERGE becomes two statements.
  • Concurrent readers see a window where the deleted rows are gone but the inserts have not yet landed. On a typical Fabric DW with snapshot isolation this is bounded but observable.
  • More warehouse work per run than necessary.

Suggested fix

When unique_key is present, dispatch to get_incremental_merge_sql so the operation runs as a single MERGE:

{% macro fabric__get_incremental_microbatch_sql(arg_dict) %}
    {% if arg_dict["unique_key"] %}
        {% do return(adapter.dispatch('get_incremental_merge_sql', 'dbt')(arg_dict)) %}
    {% endif %}
    ...
{% endmacro %}

A PR with the change is linked from this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions