Skip to content

Add GizmoSQL adapter (community, DuckDB-dialect, system ADBC driver) - #16289

Open
prmoore77 wants to merge 8 commits into
dbt-labs:mainfrom
gizmodata:gizmosql-adapter
Open

prmoore77 wants to merge 8 commits into
dbt-labs:mainfrom
gizmodata:gizmosql-adapter

Conversation

@prmoore77

@prmoore77 prmoore77 commented Sep 11, 2026

Copy link
Copy Markdown

Summary

Adds a GizmoSQL community adapter (type: gizmosql). GizmoSQL is an Arrow Flight SQL server backed by DuckDB, maintained by GizmoData; the Python dbt-gizmosql adapter has served dbt Core 1.x users and this brings it to dbt Core v2.

Contributed on behalf of GizmoData, following the community adapter guide and the shape of the Exasol contribution.

Design

  • DuckDB dialect. AdapterType::GizmoSQL / Backend::GizmoSQL are handled alongside DuckDB everywhere the code matches on adapter type: dialect and keywords, type rendering, DuckDBMetadataAdapter, duckdb_get_relation, column builder, constraint support, catalog relation defaults.
  • Macro inheritance. GizmoSQL inherits the dbt-duckdb macro package through the dispatch chain (gizmosqlduckdbdefault), exactly as lakecompute does. A small dbt-gizmosql package overrides only what a remote DuckDB needs: seeds use the default batched insert ... values loader (the server cannot read the client's seed file), grants warn, and get_binding_char returns %s.
  • Driver. Loaded from the system library search path as adbc_driver_gizmosql (entrypoint AdbcDriverGizmosqlInit), not the CDN, like Exasol. The driver is the gizmosql-adbc Go driver (a thin layer over the upstream Flight SQL ADBC driver); shared libraries for linux/macos/windows × amd64/arm64 are published on its releases page.
  • Auth / profile. dbt-auth builds a gizmosql://host:port URI (the driver's own scheme: TLS by default, ?transport=tcp when use_encryption: false) plus basic auth, with tls_skip_verify and auth_type: external (OAuth/SSO browser flow) options. Profile keys and aliases match the Python adapter's credentials so existing profiles.yml targets work unchanged. database (the DuckDB catalog on the server) is required, since it cannot be derived offline; it is set as the session's current catalog on connect.
  • Gating. Not added to NON_EXPERIMENTAL_ADAPTERS; runs with DBT_ALLOW_EXPERIMENTAL_ADAPTERS=true.
  • Also registers get_binding_char and convert_datetimes_to_strs as pure adapter methods in time_machine::semantic; the DuckDB non-fast seed path hit the debug assertion without them.

Profile example

my_project:
  target: dev
  outputs:
    dev:
      type: gizmosql
      host: gizmosql.example.com
      port: 31337
      username: dbt
      password: "{{ env_var('GIZMOSQL_PASSWORD') }}"
      database: my_db
      schema: main
      use_encryption: true
      tls_skip_verify: false
      threads: 8

Testing

  • cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, and cargo nextest run for every touched crate pass.
  • The dbt-loader DuckDB macro tests now run for both DuckDB and GizmoSQL; added tests for the auth module, the package inheritance chain, multi-adapter run_hooks resolution, and contract wrappers. The dbt-adbc driver test is env-gated on ADBC_GIZMOSQL_URI.
  • End to end against a GizmoSQL 1.38.4 server with driver 2.0.13: dbt build on jaffle-shop-classic (3 seeds, 5 models, 20 tests) plus an incremental model (two runs), a check-strategy snapshot (two runs), --full-refresh, docs generate, and show, all clean.

CI

For dbt Labs CI, a gizmodata/gizmosql Docker image is available (the server is a single binary; the gizmosql PyPI package can also start one as a subprocess). Happy to provide a hosted instance instead if that is easier — please ping @prmoore77.

Not in scope for this first cut

Python models and the DuckDB external materialization.

🤖 Generated with Claude Code

https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ

GizmoSQL is an Arrow Flight SQL server backed by DuckDB. This adds
`AdapterType::GizmoSQL` / `Backend::GizmoSQL`, treated as a DuckDB-dialect
adapter throughout (dialect, keywords, type rendering, metadata adapter,
get_relation, constraint support) and inheriting the `dbt-duckdb` macro
package through the dispatch chain (`gizmosql` -> `duckdb` -> `default`),
the way lake compute does. A small `dbt-gizmosql` macro package overrides
what a remote DuckDB needs: seeds use the default batched insert loader
instead of `COPY ... FROM` a local file, grants warn, and the seed binding
char is `%s`.

Driver loading mirrors Exasol: the `adbc_driver_gizmosql` library is loaded
from the system search path (entrypoint `AdbcDriverGizmosqlInit`), not the
CDN. Auth builds a `grpc+tls://` (or `grpc+tcp://`) URI plus basic auth,
and supports `tls_skip_verify` and `auth_type: external` (OAuth/SSO). The
profile schema and key aliases match the Python dbt-gizmosql 1.x adapter so
existing profiles work unchanged; `database` (the DuckDB catalog on the
server) is required and is set as the session's current catalog on connect.

Also registers `get_binding_char` and `convert_datetimes_to_strs` as pure
adapter methods for the time-machine layer; the DuckDB non-fast seed path
hit a debug assertion without them.

Verified against a local GizmoSQL 1.38.4 server with driver 2.0.13:
`dbt build` on jaffle-shop-classic (3 seeds, 5 models, 20 tests) plus
incremental, snapshot, full refresh, docs generate, and show. The DuckDB
macro tests now run for both DuckDB and GizmoSQL; multi-adapter, contract
wrapper, package-chain, auth, and driver (env-gated) tests added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ
@prmoore77
prmoore77 requested a review from a team as a code owner September 11, 2026 15:02
Copilot AI lite review requested due to automatic review settings September 11, 2026 15:02
@cla-bot

cla-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @prmoore77

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are blocking correctness/security issues in GizmoSQL auth handling and interactive profile setup that can lead to unintended credential usage or prevent configuring documented auth modes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new GizmoSQL community adapter (type: gizmosql) to dbt Core v2, treating it as a DuckDB-dialect, Arrow Flight SQL–backed system using an ADBC driver loaded from the system library path.

Changes:

  • Introduces AdapterType::GizmoSQL / Backend::GizmoSQL and wires it through dialect, relation, SQL typing, metadata, and ADBC engine integration.
  • Adds GizmoSQL auth/profile schema support plus interactive/headless profile setup and adapter config aliases to match the Python adapter.
  • Adds a dbt-gizmosql internal macro package that inherits dbt-duckdb, plus tests ensuring macro dispatch/inheritance parity.
File summaries
File Description
crates/dbt-tasks-sa/src/sql/dialect.rs Maps GizmoSQL to DuckDB sqlparser dialect.
crates/dbt-schemas/src/schemas/relations/base.rs Includes GizmoSQL in relation time filtering behavior.
crates/dbt-schemas/src/schemas/profiles.rs Adds DbConfig::GizmoSQL, config fields, and target context conversion.
crates/dbt-profile-schemas/src/lib.rs Exposes GizmoSQL profile setup module and helper.
crates/dbt-profile-schemas/src/headless.rs Enables GizmoSQL in headless profile generation.
crates/dbt-profile-schemas/src/gizmosql_config.rs Implements interactive GizmoSQL profile prompts.
crates/dbt-metricflow/src/lib.rs Accepts gizmosql as DuckDB dialect in metricflow parsing.
crates/dbt-loader/tests/macros/multi_adapter.rs Adds multi-adapter macro dispatch test coverage for GizmoSQL.
crates/dbt-loader/tests/macros/duckdb.rs Runs DuckDB macro tests for both DuckDB and GizmoSQL.
crates/dbt-loader/tests/macros/contract_wrappers.rs Includes GizmoSQL in contract wrapper macro tests.
crates/dbt-loader/tests/macros/aggregated_tests.rs Adds assertion that GizmoSQL renders batched unique like DuckDB.
crates/dbt-loader/src/load_packages.rs Ensures GizmoSQL loads dbt-duckdb via internal package chain.
crates/dbt-loader/src/dbt_macro_assets/dbt-gizmosql/macros/adapters.sql Adds GizmoSQL macro overrides (seeds/grants/binding char).
crates/dbt-loader/src/dbt_macro_assets/dbt-gizmosql/dbt_project.yml Defines the dbt-gizmosql internal macro package.
crates/dbt-jinja/minijinja/src/dispatch_object.rs Adds GizmoSQL → DuckDB dispatch prefix and internal package inheritance.
crates/dbt-init/src/profile_setup.rs Adds GizmoSQL to interactive dbt init adapter list and setup routing.
crates/dbt-df-providers/src/seed_io.rs Aligns GizmoSQL seed column-name inference with DuckDB/LakeCompute.
crates/dbt-common/src/adapter.rs Maps GizmoSQL to DuckDB dialect.
crates/dbt-auth/src/lib.rs Registers GizmoSQL auth implementation.
crates/dbt-auth/src/gizmosql/mod.rs Implements GizmoSQL auth → ADBC database options translation + tests.
crates/dbt-adbc/tests/driver/mod.rs Adds env-gated GizmoSQL driver execution tests.
crates/dbt-adbc/src/repl.rs Enables gizmosql backend selection in the ADBC REPL.
crates/dbt-adbc/src/install.rs Excludes GizmoSQL from CDN install path (system-only).
crates/dbt-adbc/src/driver.rs Adds Backend::GizmoSQL with system-library driver name/entrypoint.
crates/dbt-adapter/src/time_machine/semantic.rs Marks new adapter methods as pure for time_machine semantics.
crates/dbt-adapter/src/sql_types.rs Includes GizmoSQL in type metadata/comment handling and size limits.
crates/dbt-adapter/src/relation/factory.rs Enables static relation construction for GizmoSQL.
crates/dbt-adapter/src/metadata/get_relation.rs Routes GizmoSQL relation lookup through DuckDB metadata path.
crates/dbt-adapter/src/engine/adbc.rs Sets current catalog for GizmoSQL connections based on database.
crates/dbt-adapter/src/column/column_builder.rs Constructs columns for GizmoSQL and aligns it with postgres-like path.
crates/dbt-adapter/src/catalog_relation.rs Uses DuckDB defaults/attrs for GizmoSQL catalog relations.
crates/dbt-adapter/src/adapter/mod.rs Adds GizmoSQL to various adapter behavior match arms.
crates/dbt-adapter/src/adapter/adapter_impl.rs Reuses DuckDB metadata adapter and adds GizmoSQL feature/behavior arms.
crates/dbt-adapter/src/adapter/adapter_factory.rs Maps AdapterType::GizmoSQL to Backend::GizmoSQL.
crates/dbt-adapter-sql/src/types/tests.rs Treats GizmoSQL like DuckDB in type test scaffolding.
crates/dbt-adapter-sql/src/types/mod.rs Uses DuckDB-like rendering/units for GizmoSQL SQL types.
crates/dbt-adapter-sql/src/statements.rs Includes GizmoSQL in update-statement detection.
crates/dbt-adapter-sql/src/keywords.rs Reuses DuckDB keyword sets for GizmoSQL.
crates/dbt-adapter-sql/src/ident.rs Aligns GizmoSQL identifier quoting/validity rules with DuckDB-like backends.
crates/dbt-adapter-core/src/lib.rs Adds AdapterType::GizmoSQL and quote-char mapping + tests.
crates/dbt-adapter-core/src/config_aliases.rs Adds GizmoSQL profile key alias map matching Python adapter.
.changes/unreleased/Features-20260911-095735.yaml Adds changelog entry for the new GizmoSQL adapter.
Review details
  • Files reviewed: 42/42 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/dbt-auth/src/gizmosql/mod.rs Outdated
Comment thread crates/dbt-profile-schemas/src/gizmosql_config.rs Outdated
Comment thread crates/dbt-schemas/src/schemas/profiles.rs
The GizmoSQL ADBC driver owns the `gizmosql://` scheme (TLS by default,
`?transport=tcp` for plaintext) and rewrites it to the underlying Flight SQL
URI itself. Building that instead of `grpc+tls://` / `grpc+tcp://` keeps the
transport a driver concern, so it can change without touching dbt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ
@cla-bot

cla-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @prmoore77

…s range

- `auth_type: external` no longer sends profile `username`/`password` as
  Flight SQL basic auth; the OAuth flow supplies its own credentials. If
  either is present it is ignored with a warning. Test added.
- Interactive/headless profile setup gains an `auth_type` selector
  (password or external); `username`/`password` are only asked for
  password auth, and `external_root` is an optional field.
- `threads` given as an integer is range-checked with `u16::try_from`
  instead of `as u16`, matching the string path's error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a new adapter/backend and threads that behavior through many core crates (auth, profiles, macro dispatch, ADBC engine/driver), so it needs final human review for cross-cutting correctness and compatibility.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

crates/dbt-adapter/src/engine/adbc.rs:608

  • new_connection_with_config adds GizmoSQL-specific connection setup using an if self.adapter_type == ... chain; this is the same target-specific control-flow pattern used just above for ClickHouse and is easier to extend/scan when handled via a single match self.adapter_type dispatch.
  • Files reviewed: 42/42 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot suggested folding the ClickHouse and GizmoSQL connection-option
branches in `new_connection_with_config` into one `match self.adapter_type`,
matching the adapters guidance to dispatch on adapter type rather than
chain `if` checks. No behavior change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

TargetContext construction for GizmoSQL does not currently validate required connection fields (notably host), which can allow invalid profiles to proceed until later failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 42/42 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread crates/dbt-schemas/src/schemas/profiles.rs
Comment thread crates/dbt-adapter/src/adapter/adapter_impl.rs Outdated
… comment

- `TargetContext` construction for GizmoSQL now fails at profile load when
  `host` is missing or empty, with the same "field is required" error as
  `database`, instead of leaving `target.host` unset until the connection
  fails. `GizmoSQLTargetEnv.host` is a `String` accordingly.
- The note about GizmoSQL not batching multi-statement SQL now sits on the
  splitter arm it describes, not on the DuckDB/LakeCompute batching arm.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

GizmoSQL columns are currently built via a path that hard-codes AdapterType::Postgres in ColumnBuilder, which can cause GizmoSQL columns to be tagged/parsed with Postgres rules instead of GizmoSQL/DuckDB behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 42/42 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread crates/dbt-adapter/src/column/column_builder.rs
`ColumnBuilder::build_postgres_like` hard-coded `AdapterType::Postgres`
when constructing the `Column`, so GizmoSQL columns built from Arrow
fields were tagged as Postgres while `build_from_parts` tagged them as
GizmoSQL. The builder now takes the adapter type to tag the column with.
Postgres, Salesforce, DuckDB and lake compute keep the Postgres tag they
always had; only GizmoSQL passes its own type. Numeric precision/scale
still follow the Postgres rules for all of them (integers get scale 0).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ZP2QGdMHXGr5VeCMjPgfQ

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A couple of required GizmoSQL fields are not validated against empty-string values (notably username and database), which can allow invalid profiles to pass and fail later in confusing ways.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

crates/dbt-auth/src/gizmosql/mod.rs:156

  • username is treated as present even when it is an empty string, so a profile like username: "" will pass validation (unless auth_type: external) and then attempt to authenticate with an empty username. Consider trimming/filtering empty usernames so the existing "requires 'username'" check also rejects empty values.
    crates/dbt-schemas/src/schemas/profiles.rs:2293
  • database is required for GizmoSQL, but an explicitly empty string (e.g., database: "") will currently pass ok_or_else(missing("database")) and become the session catalog, which is likely invalid and can lead to confusing runtime behavior. Treat empty strings the same as missing for consistency with the host check above.
    crates/dbt-adapter/src/adapter/adapter_impl.rs:314
  • This match duplicates the same DuckDBMetadataAdapter::new(engine) construction for DuckDB, LakeCompute, and GizmoSQL. These can be combined into a single pattern arm to reduce repetition and keep future edits consistent.
  • Files reviewed: 42/42 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

… metadata arms

- Treat an empty `username` the same as a missing one in GizmoSQL auth,
  so `username: ""` fails the required-username check (unless
  `auth_type: external`) instead of attempting basic auth with an empty
  name.
- Treat an empty `database` the same as a missing one when building the
  GizmoSQL target context, matching the existing `host` check.
- Collapse the identical DuckDB / LakeCompute / GizmoSQL metadata adapter
  arms into one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@prmoore77

Copy link
Copy Markdown
Author

Addressed the three suppressed Copilot comments in 5d2134e: an empty username is now rejected like a missing one (unless auth_type: external), an empty database is rejected the same way host already was, and the identical DuckDB / LakeCompute / GizmoSQL metadata-adapter arms are collapsed into one. Covered by test_empty_username_returns_error and test_gizmosql_target_context_rejects_empty_host_and_database.

@prmoore77
prmoore77 requested a lite review from Copilot September 15, 2026 17:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces a new adapter type and threads it through many cross-cutting code paths (auth/ADBC, macro dispatch, profiles, adapter logic), which warrants final human review despite targeted tests.

Review details
  • Files reviewed: 42/42 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants