Skip to content

Add merge loads to the sqlserver connector #138

Description

@victorchutw

Implements ADR-0064 (merge — widening ADR-0057's scope), the ADR-0058 gate in T-SQL, and ADR-0059's staged-terminal shape in SQL Server idiom. Part of the SQL Server destination cycle (map #106, sliced from #113).

Blocked by #136

Context

Merge is the production pattern this cycle serves — including merge keys on IDENTITY columns. It reuses the staged terminal transaction of ADR-0059 with a created-shape stage, deliberately not a clone of the target: the bulk path always writes ADR-0062 types, and any accepted lossy conversion happens server-side inside MERGE, governed by ADR-0065's opt-in.

The contract

  • begin(merge): target must pre-exist (merge never bootstraps); introspect + Accept Family with merge's rules (Validate existing SQL Server tables against the Accept Family #135: a mapped IDENTITY is acceptable iff it is a merge key; DEFAULT-carrying mapped columns are acceptable). Then BEGIN TRAN and create a real staging table data_spark_merge_stage_<uuid> in the target's schema, in the dataset's created shape (Generate SQL Server DDL and bulk rows from the dataset schema #134).
  • write_chunk: bulk_insert into the stage, inside the transaction.
  • commit, all inside the same transaction:
    1. ADR-0058 duplicate-key gate on the stage — a duplicate key tuple rolls back and fails duplicate_merge_keys with the destination untouched.
    2. Count updated as staged rows with a key match in the target; inserted = staged − updated — preserving the documented updated + inserted == written invariant.
    3. MERGE INTO [schema].[target] WITH (HOLDLOCK) USING stage ON <key equality> WHEN MATCHED THEN UPDATE SET <non-key columns> WHEN NOT MATCHED THEN INSERT <dataset columns> — the UPDATE clause omitted when every column is a key; the INSERT branch omits extra target columns (server defaults apply).
    4. When the merge key maps an IDENTITY column, wrap the MERGE in SET IDENTITY_INSERT [target] ON/OFF (requires ALTER on the target; session-scoped, cannot leak — failure paths end the session).
    5. Drop the stage, terminal COMMIT.
  • Failure: explicit ROLLBACK; the in-transaction stage vanishes on rollback or connection death — no orphan cleanup exists by construction.
  • Facts: atomicity: "atomic", strategy: "transactional_merge" (existing string), merge counts as on DuckDB. Zero survivors: empty stage, no-op merge, COMMIT — the mirror posture.
  • Target-side duplicate keys: no gate — every matching target row is replaced whole and counts stay source-perspective (the docs slice documents this in the merge guide; DuckDB's undercount is outside-map DuckDB merge replaces every duplicate-key target record but reports updated from the stage side #125).
  • Modes: supported_load_modes += merge — all three live; other destinations' decline messages unchanged. Classification stays Terminal this slice.

Acceptance criteria

  1. Gated live tests:
    • Fresh merge into an empty existing table inserts all rows (updated: 0); a second merge with changed + new records updates matched rows whole and inserts the rest — counts assert the invariant.
    • Merge key on an IDENTITY column: end-to-end merge lands inserted keys faithfully (read-back proves the source key values, not server-assigned ones).
    • Stage duplicate keys → duplicate_merge_keys, target rows byte-equal before/after (count + content probe).
    • All-columns-are-keys merge is legal (insert-only semantics).
    • Merge onto a missing target fails before any write.
    • Zero-survivor merge commits no change with atomic/transactional_merge facts.
    • Extra target columns: UPDATE never touches them; INSERT leaves them to server defaults.
  2. Non-key mapped IDENTITY still rejects via incompatible_destination_table (gated).
  3. Serverless cargo test --locked green with new tests ignored; CI runs them.
  4. CHANGELOG.md Unreleased entry: merge lands on sqlserver (ADR-0057 scope widened).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified and ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions