dbt for Apache Doris

dbt-for-apache-doris enables Python dbt Core projects to transform data in Apache Doris through the Doris MySQL protocol. It is maintained by the VeloDB community.

Installation · Quickstart · Examples · Compatibility · PyPI · dbt docs · Doris docs · Issues · Releases

Supported capabilities

Status: ✅ Supported · ❌ Not supported

Feature status and database-version compatibility are separate contracts. Supported means the documented scope is implemented and tested; explicit platform boundaries are described alongside each capability.

Materializations

Capability	Status	Current support and boundaries
Table	✅ Supported	Duplicate Key CTAS; configurable HASH distribution, integer buckets, RANGE/LIST partitions, properties, contracts, docs, grants, and hooks. Unique Key creation belongs to incremental merge
View	✅ Supported	Standard lifecycle, contracts, docs, grants, and hooks; relation-type switching is not zero-downtime
Incremental	✅ Supported	Four strategies and every on_schema_change mode; boundaries are listed below
Snapshot	✅ Supported	check/timestamp, hard-delete modes, schema evolution, atomic replacement, and recovery; same-target runs must be serialized by the scheduler
Materialized view	✅ Supported	Standard dbt materialized_view, implemented with Doris Async MV; build/refresh lifecycle, task waiting, configuration changes, atomic replacement, and recovery. Same-target dbt runs must be serialized by the scheduler
Seed	✅ Supported	CSV loading, type inference, column_types, and ref
Ephemeral	✅ Supported	Compiled and inlined by dbt Core

dbt capabilities

Capability	Status	Current support and boundaries
Sources and freshness	✅ Supported	loaded_at_field, filter, and loaded_at_query; sources may use Internal or External Catalog relations
Data tests	✅ Supported	Singular, generic, ephemeral, and store_failures paths
dbt Unit tests	✅ Supported	Inline-row and CSV fixtures, case-insensitive columns, invalid-input validation, quoted reserved words, Doris-adapted data-type fixtures, and non-truncating VARCHAR fixtures
Model contracts	✅ Supported	Column names/types for Table, View, and Incremental; not database PK/NOT NULL constraints
Persisted docs	✅ Supported	Relation and column comments for Table, View, Incremental, Snapshot, Seed, and Async MV; updating View comments or comment text containing both quote delimiters may require recreation/full refresh
Grants	✅ Supported	Reconciles supported Doris table privileges for user and user@host principals on Table, View, Incremental, Seed, Snapshot, and Async MV; role principals are not reconciled
Hooks	✅ Supported	Pre-hooks and post-hooks across adapter materializations; Doris does not provide transactional rollback for hook side effects
Metadata and dbt docs catalog	✅ Supported	Relation and column discovery for Internal and External Catalogs; Internal Catalog Async MV detection
Cross-database and cross-catalog sources	✅ Supported	Two-part database.table within Internal Catalog and three-part catalog.database.table for configured External Catalogs
Advanced metadata APIs	❌ Not supported	Catalogs V2, metadata-by-relation, single-relation catalog, and last-modified metadata are not declared

Compatibility

Component	Declared or runtime constraint	Current evidence or status
Python	>=3.10	Unit CI covers 3.10 and 3.14; the distribution-build job uses 3.12
dbt Core	>=1.12,<1.13	Declared lower bound is 1.12.0; Python dbt Core v1 only. Fusion/v2 compatibility is not claimed
MySQL connector	>=8.0.33	Installed automatically with the adapter
Apache Doris	No package-wide minimum is declared	Validate the adapter against the Doris release and topology used in production
Async MV	Doris 2.x >=2.1.5; Doris 3.x except 3.0.0; Doris 4.x+	This runtime gate applies to Async MV. Identifiable source builds are accepted for development testing only
VeloDB	No release range is declared	Validate the adapter against the VeloDB release and topology used in production

Before production use, validate the adapter against your exact database release and deployment topology.

Installation

Install the VeloDB-maintained distribution from PyPI:

python -m venv .venv
source .venv/bin/activate
python -m pip install "dbt-for-apache-doris==1.1.0"
dbt --version

On Windows, create the environment with py -m venv .venv, activate it using .venv\Scripts\Activate.ps1, and run the same pip install command.

The adapter declares dbt Core and the MySQL connector as dependencies, so they are installed automatically. You do not need to install dbt Core separately or download a standalone binary.

Quickstart

Add a Doris output to ~/.dbt/profiles.yml. Keep credentials outside version control; this example reads the password from an environment variable:

doris_demo:
  target: dev
  outputs:
    dev:
      type: doris
      host: 127.0.0.1
      port: 9030
      username: root
      password: "{{ env_var('DORIS_PASSWORD') }}"
      schema: analytics
      threads: 4

On Doris, dbt schema is a Doris Database. Omit dbt database for the Internal Catalog.

Create a new doris-demo directory with a models subdirectory, then add:

# dbt_project.yml
name: doris_demo
version: 1.0.0
config-version: 2
profile: doris_demo
model-paths: ["models"]

-- models/example.sql
{{ config(materialized='table', replication_num=1) }}
select 1 as id, 'hello from dbt-for-apache-doris' as message

# models/schema.yml
version: 2
models:
  - name: example
    columns:
      - name: id
        data_tests: [not_null, unique]

replication_num=1 is only for a local single-BE Quickstart.

export DORIS_PASSWORD='<your-password>'
dbt debug
dbt build

On Windows PowerShell, set the password with $env:DORIS_PASSWORD = '<your-password>', then run the same dbt commands.

External Catalog sources

dbt relation fields map to Doris as database.schema.identifier → catalog.database.table. Set database to an existing Doris Catalog and schema to the Database inside that Catalog:

sources:
  - name: lakehouse
    database: hive_catalog
    schema: ods
    tables:
      - name: orders

{{ source('lakehouse', 'orders') }} renders as `hive_catalog`.`ods`.`orders`. The adapter discovers its tables and columns from the selected Catalog and includes them in dbt docs generate. The External Catalog must already exist in Doris. DDL and write support depend on the corresponding Doris Catalog connector.

End-to-end examples

The examples tree contains five runnable Doris projects and a single user-facing entry point at examples/doris-demos. Open its README and notebooks/ directory; the project directories and runner scripts are implementation details. Five focused Jupyter Notebooks cover Table, View, Seed, Data Test, cross-database Source, incremental merge, Snapshot, and Async MV workflows.

Follow the examples quick start to configure Doris, create the pinned dbt environment, start JupyterLab, and run any of the five demos.

Doris-specific highlights

Incremental strategies

Strategy	Doris target	Behavior and boundaries
append	Duplicate Key table	Appends rows with INSERT INTO
merge	MOW or MOR Unique Key table	Full-row INSERT INTO upsert using Doris Unique Key semantics; requires unique_key and does not emit SQL MERGE INTO
insert_overwrite	Writable Doris table	Whole-table, named-partition, or dynamic-partition INSERT OVERWRITE; unique_key is rejected
microbatch	Duplicate Key table with exact RANGE partitions	One named-partition overwrite per dbt Core UTC window; hour/day/month/year windows; static or dynamic partitions; batches run serially

Without an explicit strategy, unique_key selects merge; otherwise dbt uses append.

Materialized views

Use dbt's standard materialized_view materialization. The adapter implements it with Doris Async MV and exposes Doris-specific refresh configuration:

{{ config(
    materialized='materialized_view',
    refresh_trigger='manual',
    wait_for_refresh=true
) }}

select order_date, sum(amount) as sales
from {{ ref('orders') }}
group by order_date

Supported lifecycles include immediate/deferred build, manual/schedule/commit refresh, task waiting, configuration changes, docs, grants, and recovery. Overlapping dbt runs against the same MV target must be serialized, and a dbt wait timeout does not cancel a submitted Doris task.

Known limitations

• Aggregate Key table modeling and secondary-index configuration are not supported.
• Catalogs V2 and connector-specific External Catalog write guarantees are not supported.
• SSL configuration, timeout/retry, multi-FE failover, server-side cancellation, and complete query telemetry are not implemented.
• Some Table/View/MV type changes have a short canonical-name availability window rather than a zero-downtime switch.

Development and testing

Install development dependencies, then run local checks:

python -m pip install -r dev-requirements.txt
python -m pip install -e .
make lint
make test-unit

Functional tests need a dedicated non-production cluster. Edit test/doris_test.env; use an external file for private credentials:

make test
make test DORIS_TEST_CONFIG=/secure/path/doris_test.env

Preflight records live FE/BE versions, checks replication against live BEs, and requires cross_db_test to be absent. Tests create/drop databases, relations, users, and grants, so the account needs those permissions. Never use a shared or production cluster or run Functional sessions concurrently.

python scripts/run_doris_functional_tests.py --preflight-only
python scripts/run_doris_functional_tests.py -- -k snapshot -vv

The runner records evidence about the connected cluster but does not certify a release compatibility matrix.

License

The code is licensed under Apache License 2.0. See the license, notice, and migration provenance.