Skip to content

feat(athena): AthenaDbConfig profile schema - #16274

Open
fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/athena-profile-schema
Open

fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/athena-profile-schema

Conversation

@fpiped

@fpiped fpiped commented Sep 10, 2026

Copy link
Copy Markdown

Resolves #16252
Related: #13822

Problem

type: athena in profiles.yml fails at parse time (InvalidConfig (dbt1005): Failed to parse profiles.yml: unknown variant 'athena'), before the experimental-adapter gate and before dbt-auth are reached. The Athena pieces already in the repo (AdapterType::Athena, Backend::Athena, the ADBC option keys in crates/dbt-adbc/src/athena.rs, AthenaAuth in crates/dbt-auth/src/athena/mod.rs) are therefore unreachable from a profile: DbConfig has no Athena variant (crates/dbt-schemas/src/schemas/profiles.rs, // Athena,).

Solution

The profile-schema slice, same shape as ClickHouse "Part 2" (b468efcd6), plus one arm in the relation factory:

  • AthenaDbConfig, with the field set of dbt-athena's AthenaCredentials (connections.py) and the catalog alias for database. Every dbt-athena field is declared, including the ones AthenaAuth currently rejects (assume_role_*, s3_data_dir, poll_interval, ...): the auth layer reads the profile through to_mapping(), so a field absent from the struct would be dropped silently instead of producing the existing "not yet supported" error. A test pins this.
  • DbConfig::Athena wired through adapter_type, get_unique_field (s3_staging_dir, as in dbt-athena), get_database / get_database_or_default (defaults to awsdatacatalog, the default AthenaAuth applies), get_schema, get_threads / set_threads, to_yaml_value.
  • connection_keys_for(AdapterType::Athena): dbt-athena's _connection_keys restricted to what AthenaAuth honours today; aws_secret_access_key and aws_session_token stay out, so dbt debug never prints them (covered by the existing connection_keys_never_expose_a_credential).
  • TargetContext::Athena / AthenaTargetEnv (region_name, s3_staging_dir, work_group, aws_profile_name plus the common database / schema / type / threads). Threads parsing goes through a small threads_from_profile helper instead of a fourth inline copy of the same match.
  • relation/factory.rs: Athena joins the generic RelationStatic arm, so dbt parse / dbt ls stop at a configuration error instead of todo!("Athena").
  • dbt-test-utils::profiles: schema / database overrides for Athena, as for the other adapters.

Athena is not added to NON_EXPERIMENTAL_ADAPTERS, so it stays behind DBT_ALLOW_EXPERIMENTAL_ADAPTERS. The remaining todo!("Athena") arms in dbt-adapter (adapter_impl.rs metadata adapter, column_builder.rs, sql_types.rs) and the dbt-athena macro package under dbt_macro_assets are the next slices and are not touched here.

Behaviour with this branch, debug build, against the profile below:

Command Result
dbt debug, DBT_ALLOW_EXPERIMENTAL_ADAPTERS=false The 'athena' adapter is not yet supported by dbt. Supported adapters: snowflake, ... (gate reached; before: unknown variant 'athena')
dbt debug, gate open AuthenticationFailed: Driver not found: adbc_driver_athena (profile → AthenaAuth → driver load)
dbt debug, catalog: my_catalog same as above; alias accepted
dbt debug, assume_role_arn: ... Athena profile field 'assume_role_arn' is not yet supported by the dbt Athena backend (the AthenaAuth rejection now fires)
dbt debug, no schema Athena requires 'schema' in profile configuration
dbt parse / dbt ls, gate open Missing dbt_project.yml in embedded package 'dbt-athena' (before this branch: panic at relation/factory.rs:28)
athena_probe:
  target: dev
  outputs:
    dev:
      type: athena
      region_name: eu-west-1
      s3_staging_dir: s3://example-bucket/athena-results/
      schema: analytics
      work_group: primary

Not in scope, and a dbt Labs step: the Athena ADBC driver is not in INSTALLABLE_DRIVERS, so loading falls back to LoadStrategy::System (a locally built adbc_driver_athena from dbt-labs/athena).

Checks run locally: cargo fmt --check, cargo nextest run -p dbt-schemas -E 'test(/profiles::tests/)' (36 passed), cargo check -p dbt-test-utils -p dbt-profile-schemas -p dbt-init -p dbt-loader -p dbt-compilation, cargo clippy --all-targets --all-features on dbt-schemas, dbt-test-utils and dbt-adapter, cargo build -p dbt-sa-cli for the table above.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX. — adds the type: athena profile type, gated as experimental; flagging for Product/DX review.

@fpiped
fpiped requested a review from a team as a code owner September 10, 2026 08:11
Add the `athena` variant of `DbConfig` so Athena profiles parse and reach
the experimental-adapter gate and `dbt-auth`, which already carry the
Athena pieces (`AdapterType::Athena`, `Backend::Athena`, `AthenaAuth`).

`AthenaDbConfig` declares every field of dbt-athena's `AthenaCredentials`,
including the ones `AthenaAuth` rejects today: the auth layer reads the
profile through `to_mapping()`, so an undeclared field would be dropped
silently instead of raising the "not yet supported" error. `catalog` is
accepted as an alias of `database`, matching dbt-athena.

`connection_keys_for(Athena)` lists the honoured, non-secret keys.
`TargetContext::Athena` exposes `region_name`, `s3_staging_dir`,
`work_group`, `aws_profile_name` and the common fields, with `database`
defaulting to `awsdatacatalog` as in `AthenaAuth`; threads parsing goes
through `threads_from_profile`.

The relation factory builds the generic `RelationStatic` for Athena, as
for every other adapter, so `dbt parse` and `dbt ls` stop at a
configuration error naming the missing `dbt-athena` macro package instead
of a `todo!` panic.

Athena stays outside `NON_EXPERIMENTAL_ADAPTERS`. The remaining
`todo!("Athena")` arms in `dbt-adapter` and the macro package are left
for follow-ups.

Resolves dbt-labs#16252
@fpiped
fpiped force-pushed the feat/athena-profile-schema branch from 75ee2dc to 391a3c8 Compare September 10, 2026 08:14
@cla-bot

cla-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @fpiped

@fpiped

fpiped commented Sep 10, 2026

Copy link
Copy Markdown
Author

@cla-bot check

@cla-bot cla-bot Bot added the cla:yes label Sep 10, 2026
@cla-bot

cla-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

The cla-bot has been summoned, and re-checked this pull request!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Athena adapter for dbt Core 2.0

1 participant