Skip to content

Add append loads to the sqlserver connector #137

Description

@victorchutw

Implements ADR-0064 (append) with ADR-0065's append rules live. Part of the SQL Server destination cycle (map #106, sliced from #113).

Blocked by #136

Context

Append mirrors ADR-0047 verbatim on SQL Server: independent per-chunk bulk loads, auto-committed, committed prefix persists on failure. The table must already exist — append never bootstraps.

The contract

  • begin(append): the table must pre-exist; an absent table fails destination_write_failed with before-append wording and the existing before-session posture (the DuckDB append/merge precedent). Present → introspect + Accept Family with append's rules (Validate existing SQL Server tables against the Accept Family #135): mapped IDENTITY rejected, mapped DEFAULT-carrying column with a nullable field rejected, extra columns per the extra-column rule. No transaction is opened.
  • write_chunk: one auto-committed bulk_insert per chunk (one INSERT BULK batch). The spike proved a failed batch is atomic — zero rows — so a failed load keeps exactly its committed prefix.
  • commit: finalize only; write facts atomicity: "best_effort", strategy: "bulk_insert" (new string, additive).
  • Extra columns: the stream carries NULL for them — which the bulk path documentedly replaces with the column's DEFAULT — or a typed placeholder the server discards for IDENTITY (Validate existing SQL Server tables against the Accept Family #135's rules make this reachable only where faithful).
  • Modes: supported_load_modes += append. Classification stays Terminal in this slice (the transience slice flips append's decided sites).

Acceptance criteria

  1. Gated live tests:
    • Two loads append cumulatively (counts add up; no delete/replace).
    • Append onto a missing table fails before any write with the before-append posture.
    • Committed prefix: with a small configured chunk_rows, a load whose later chunk carries a >32,767-character value fails, and exactly the earlier chunks' rows are present — report says best_effort, failure surfaced, earlier rows queryable.
    • Extra destination columns: nullable, DEFAULT-carrying, and IDENTITY extras accept the load and take NULL/default/generated values respectively.
    • Rejections: a mapped-IDENTITY table and a mapped DEFAULT+nullable-field table each fail incompatible_destination_table at begin.
  2. Report facts and exit codes as specified; serverless cargo test --locked green with the new tests ignored.
  3. CHANGELOG.md Unreleased entry: append lands on sqlserver.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified and ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions