ETT Data — A Databricks Asset Bundle Example

An end-to-end reference project for learning Databricks Asset Bundles (DAB). It uses the public ETT (Electricity Transformer Temperature) dataset as a concrete example to show how to structure, configure, and deploy a real data pipeline on Databricks — not a toy example.

What this repo covers

By reading and running this project you will understand how to:

• Structure a DAB project with databricks.yml, resource files, and a Python package
• Define a DLT pipeline and Workflow jobs as bundle resources
• Use variables and targets to manage dev/prod configuration without duplicating code
• Build and deploy a Python wheel as part of the bundle
• Chain a DLT pipeline and Python wheel tasks in a single orchestrated workflow
• Run a job standalone, independently of the full workflow

The pipeline

The ETT dataset contains time-series readings from electricity transformers (temperature, load, etc.). This project processes it through the classic bronze → silver → gold medallion architecture:

Raw CSVs (Volume)
    ↓  [DLT Pipeline]
Bronze tables  — raw ingestion, everything as string
    ↓
Silver tables  — typed columns, quality checks, invalid rows dropped
    ↓
Gold tables    — daily averages per feature column
    ↓  [Wheel Job: compute_splits]
Gold train/test tables  — time-ordered 80/20 split
    ↓  [Wheel Job: export_gold_tables]
Parquet files (Volume)  — exported for downstream use

Project structure

.
├── databricks.yml                    # Bundle entry point: variables, targets, artifacts
├── pyproject.toml                    # Python package definition and entry points
├── resources/
│   ├── ett_data_etl.pipeline.yml     # DLT pipeline resource (bronze → silver → gold)
│   ├── ett_full_workflow.job.yml     # Orchestrated workflow: pipeline + jobs in sequence
│   └── ett_split.job.yml            # Standalone train/test split job
└── src/ett_data/
    ├── config.py                     # Dataset variants and feature column names
    ├── compute_train_test_splits.py  # Entry point: time-ordered train/test split
    ├── export_gold_tables.py         # Entry point: export gold tables to parquet
    └── transformations/
        ├── build_bronze_tables.py    # DLT: ingest CSVs from volume
        ├── build_silver_tables.py    # DLT: cast types, drop invalid rows
        └── build_gold_tables.py     # DLT: compute daily averages

Key DAB concepts illustrated

databricks.yml — the bundle root

All configuration lives here: variable declarations, artifact build commands, and target definitions. Variables without a default (like catalog and schema) must be set explicitly per target — this prevents accidental deployments to the wrong environment.

artifacts:
  python_artifact:
    type: whl
    build: uv build --wheel   # DAB runs this before deploying

When you run databricks bundle deploy, DAB automatically builds the wheel and uploads it to the workspace before deploying any resources.

Variables and targets

Variables are declared once and resolved per target. Derived paths reference other bundle values so you never hardcode environment-specific strings:

variables:
  whl_path:
    default: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}/artifacts/.internal/ett_data-0.0.1-py3-none-any.whl

targets:
  dev:
    variables:
      catalog: workspace
      schema: ett
      volume_path: /Volumes/${var.catalog}/${var.schema}/ett_raw
  prod:
    variables:
      catalog: workspace
      schema: ett_prod
      volume_path: /Volumes/${var.catalog}/${var.schema}/ett_raw

Bundle key vs. display name

Every resource has two names that serve different purposes:

resources:
  pipelines:
    ett_pipeline:              # bundle key — used in CLI commands and cross-references
      name: ETT Data pipeline  # display name — shown in the Databricks UI

The bundle key is used to reference the resource elsewhere (e.g. ${resources.pipelines.ett_pipeline.id}) and in CLI commands (databricks bundle run ett_pipeline). The display name can contain spaces and can be overridden per target.

Mixing DLT and wheel tasks in one workflow

The orchestrated workflow chains a DLT pipeline and two Python wheel tasks with explicit dependencies:

tasks:
  - task_key: run_pipeline
    pipeline_task:
      pipeline_id: ${resources.pipelines.ett_pipeline.id}   # cross-reference by bundle key

  - task_key: compute_splits
    depends_on:
      - task_key: run_pipeline   # runs only after the pipeline succeeds
    python_wheel_task:
      package_name: ett_data
      entry_point: compute_splits

  - task_key: export_to_parquet
    depends_on:
      - task_key: compute_splits
    python_wheel_task:
      package_name: ett_data
      entry_point: export_gold_tables

Passing configuration to DLT vs. wheel tasks

DLT pipelines and wheel tasks receive configuration differently:

• DLT pipeline: use the configuration: block; values are read at runtime via spark.conf.get("key")
• Wheel tasks: pass values as CLI parameters; read via argparse

# DLT pipeline — spark config
configuration:
  volume_path: ${var.volume_path}

# Wheel task — CLI args
parameters:
  - "--catalog"
  - "${var.catalog}"
  - "--split_ratio"
  - "${var.train_split_ratio}"

Entry points

Python wheel tasks are not called by filename — they use named entry points declared in pyproject.toml:

[project.scripts]
compute_splits    = "ett_data.compute_train_test_splits:main"
export_gold_tables = "ett_data.export_gold_tables:main"

The entry_point field in the job YAML must match these keys exactly.

Prerequisites

• Databricks workspace with Unity Catalog enabled
• Databricks CLI installed and configured (databricks auth login)
• uv installed locally
• Two Volumes created in your catalog/schema:
  • ett_raw — place the raw ETT CSV files here (download from ETDataset)
  • ett_processed — used for parquet export output

Getting started

1. Install dev dependencies:

uv sync

2. Edit databricks.yml: set your catalog, schema, and warehouse_id under the dev target. You can find your warehouse_id in the Databricks UI under SQL Warehouses — it's the ID in the warehouse URL.

3. Deploy:

First validate to check for any error,

databricks bundle validate --target dev

then deploy,

databricks bundle deploy --target dev

4. Run the full workflow:

databricks bundle run ett_full_workflow --target dev

Or run just the train/test split:

databricks bundle run ett_split_job --target dev

5. Repeat steps 3 and 4 for production using the target prod Prod will have its own catalog and schema, make sure you update them in databricks.yml; go to targets -> prod -> variables -> and you can find catalog, schema.

Dataset

The ETT dataset (Electricity Transformer Temperature) is a public benchmark dataset widely used in time-series forecasting research. It includes four variants — ETTh1, ETTh2 (hourly) and ETTm1, ETTm2 (15-minute) — each with readings of oil temperature and six electrical load features.