databricks-tools

Databricks utilities behind one AI-friendly CLI: dbtools.

dbtools
├── follow-run [RUN_ID] [--notebook PATH]   stream a job run's logs until it finishes
├── submit      NOTEBOOK --like CLUSTER     one-off notebook run on an inline cluster
│               [--docker-image IMG]        cloned from an existing one, then followed
├── runs        list / show / export        query runs, anomaly flags, HTML export
│               logs / diff / cancel        one-shot log fetch, run-vs-run comparison,
│                                           cancel an in-flight run (mutating)
├── notebook    pull / tail / triage        export with cell outputs, poll-and-diff tail
│               diff / create               of a live notebook, scan for problems,
│                                           diff workspace vs local, create a notebook
├── backup                                  workspace + jobs/pipelines/catalog metadata
├── jobs        list / show / search        job metadata + full-text notebook search
│               history                     success rate + duration stats per job
├── sql         exec / warehouses           run SQL on a warehouse (read-only by default)
│               history                     recent warehouse queries, incl. failures;
│                                           --statement-id drills into one with metrics
├── cost                                    DBUs by day/SKU from system.billing.usage
├── catalog     ls / describe / sample      browse Unity Catalog, peek at table rows
│               history / volumes / grants  Delta history, UC volumes, effective privileges
├── clusters    list / show / events        cluster states, sizes, owners, event log,
│               create                      start a cluster cloned from an existing one
├── policies    list / show                 cluster policies and their definitions
├── pools       list / show                 instance pools with used/idle/pending stats;
│                                           list --watch live-refreshes utilization
├── pipelines   list / show / events        DLT pipeline state, event log, and
│               updates                     update history (state, cause, full refresh)
├── fs          ls / get                    browse/download dbfs:/ and /Volumes/ paths
├── secrets     scopes / keys               secret names only — values are never fetched
├── workspace   ls / check                  list workspace dirs, readability sanity check
├── repos       list / show                 workspace repo checkouts: branch, head commit
├── alerts      list / show                 SQL alerts and their evaluation state
├── queries     list / show                 saved SQL queries (incl. the SQL text)
├── permissions TYPE ID                     who holds which permission on an object
├── profiles                                list ~/.databrickscfg profiles
└── tui                                     interactive resource browser (also: dbtui)

Install

uv sync                 # library + CLI in .venv
uv run dbtools --help

To publish dbtools to ~/bin (via spg, the per-project command publisher): run spg install in the repo root once; spg sync after pulling changes to spg.toml. The same step links this repo's Claude Code skill (.claude/skills/databricks-tools) into ~/.claude/skills/, so agent sessions in any directory pick up the checked-in version rather than a hand-copied one.

Authentication

Everything authenticates through the standard databricks-sdk chain: pass --profile/-p to pick a ~/.databrickscfg profile, or set DATABRICKS_CONFIG_PROFILE for a sticky default, or DATABRICKS_HOST/DATABRICKS_TOKEN env vars. No credentials are ever read or stored by this tool itself. dbtools profiles shows what's configured.

Examples

# Follow a run you just submitted; exit code mirrors the run result
dbtools follow-run 8632745 -p <profile>

# Follow whatever is currently running for a notebook path
dbtools follow-run --notebook "/Users/<you>/etl/daily load" -p <profile>

# Did that notebook run cleanly? Pull it (with outputs), then triage
dbtools notebook pull "/Users/<you>/etl/daily load" --out /tmp/daily.ipynb -p <profile>
dbtools notebook triage /tmp/daily.ipynb          # exit 1 + findings if anything looks wrong

# Run metadata with anomaly flags (retries, cold starts, slow cleanup)
dbtools runs show 8632745 --json -p <profile>

# What's running right now, and whose is it?
dbtools runs list --active-only -p <profile>
dbtools runs list --mine -p <profile>              # only your runs
dbtools runs list --user alice --active-only -p <profile>

# Live dashboard of active runs, refreshed until Ctrl-C
# (survives network drops: keeps the last snapshot on screen and retries)
dbtools runs list --active-only --watch -p <profile>
dbtools-watch -p <profile>                         # spg shorthand for --mine --active-only --watch

# Which jobs' notebooks mention a module?
dbtools jobs search -s my_module --json -p <profile>       # refreshes the notebook cache
dbtools jobs search -s my_module --cache-only --json       # fast, reuses the cache

# Back up a workspace (dry-run first)
dbtools backup -p <profile> --output-dir ~/backups/myworkspace --exclude-path mlruns --dry-run
dbtools backup -p <profile> --output-dir ~/backups/myworkspace --exclude-path mlruns

# Ad-hoc SQL on a warehouse (read-only unless --allow-write); --json/--csv for pipes
dbtools sql warehouses -p <profile>
dbtools sql exec "SELECT * FROM main.core.events LIMIT 10" -p <profile>
dbtools cost --days 14 -p <profile>                # DBUs by day/SKU (system tables access)

# Browse Unity Catalog and sample a table
dbtools catalog ls main.core -p <profile>
dbtools catalog describe main.core.events -p <profile>
dbtools catalog sample main.core.events --limit 5 -p <profile>

# Compute + pipelines at a glance
dbtools clusters list --running-only -p <profile>
dbtools pipelines events my_pipeline --limit 20 -p <profile>

# Instance pools: capacity + live utilization (used/idle/pending), watchable
dbtools pools list -p <profile>
dbtools pools list --watch -p <profile>            # refreshed until Ctrl-C
dbtools pools show "worker pool" -p <profile>

# Fetch a finished run's logs without re-following it; compare two runs
dbtools runs logs 8632745 -p <profile>
dbtools runs diff 8632745 8639999 -p <profile>

# Is this job healthy lately? (exit 1 if its latest completed run failed)
dbtools jobs history "daily etl" -p <profile>

# Has the workspace notebook drifted from my local copy?
dbtools notebook diff "/Users/<you>/etl/daily load" ./daily_load.py -p <profile>

# Someone is running a notebook interactively — tail its outputs as cells finish
# (job runs never write back to the workspace path; use follow-run for those)
dbtools notebook tail "/Users/<you>/etl/daily load" -p <profile>              # Ctrl-C to stop
dbtools notebook tail "/Users/<you>/etl/daily load" --until-idle 10 --json    # agent-friendly

# Create a workspace notebook (empty, or from a local file); overwrite is opt-in
dbtools notebook create "/Users/<you>/scratch/probe" --file ./probe.py -p <profile>

# Spin up a standalone cluster cloned from an existing one's spec
dbtools clusters create "my dev box" --like "shared etl" --autotermination 30 -p <profile>

# One-off notebook run on an inline cluster cloned from an existing one,
# optionally with a different docker image; follows to terminal by default
dbtools submit "/Users/<you>/etl/daily load" --like "shared etl" \
  --docker-image repo/image@sha256:<digest> --param date=2026-07-07 -p <profile>
dbtools submit "/Users/<you>/smoke test" --existing-cluster dev-box --no-follow --json

# Why did the cluster resize or terminate?
dbtools clusters events "shared etl" --limit 20 -p <profile>

# What ran (or failed) on the warehouses lately?
dbtools sql history --failed-only -p <profile>
dbtools sql history --mine --limit 20 --json

# Browse the workspace tree itself (fs ls covers dbfs:/ and /Volumes/)
dbtools workspace ls "/Users/<you>" -p <profile>

# What changed this Delta table recently, and who did it?
dbtools catalog history main.core.events --limit 10 -p <profile>

TUI

dbtui (equivalently dbtools tui, both take -p PROFILE) opens a read-only interactive browser: an expandable tree (alphabetical) of jobs, runs, active runs (everything in flight right now, workspace-wide), clusters, job clusters (the ephemeral job/pipeline ones, newest first, capped), cluster policies, pipelines, instance pools, SQL warehouses, alerts, saved queries, Unity Catalog (schemas expand to tables and volumes), the workspace tree, repos, DBFS/volumes, secret scopes, and profiles. The right side is split: expanding a node lists resources in the upper table, and selecting one (or pressing enter / clicking a row in that table) shows the same detail the CLI's show commands print in the lower pane — so the list stays in view while you inspect items (namespace listings — catalog, workspace, files, secrets — are sorted, directories first). Watch refreshes happen in place: the tree only repaints when the listing actually changed.

Resources expand into sub-views too: jobs → history + permissions + recent runs; runs → logs; clusters/pipelines → event logs (pipelines also update history); tables → sample rows + grants; and jobs/clusters/pipelines/pools/warehouses/repos/policies each carry a Permissions child.

Keys: arrows / hjkl traverse the tree (→ or double-click expands, ← collapses; a single click only selects) · enter select · enter / click on a table row opens its detail below · click a column header to sort · f follow a run live (streaming logs) · w watch: toggle auto-refresh of the highlighted node — several at once if you like; watched nodes get a 👀 marker and the header lists them · W stop all watches · m load more (raises the Jobs/Runs/Job clusters caps) · o open in the browser · c copy (tree: the node's id; table: the cell's full value) · r refresh node · [ / ] resize the tree/right split · { / } resize the table/detail split · p switch profile · / search loaded entries (esc cancels) · tab cycle focus: tree → table → detail · ? help · q quit.

The TUI browses only — mutating operations (submit, create, sql exec) and live follow views stay CLI-only for now. It shares all Databricks access with the CLI (src/databricks_tools/registry.py over the feature modules), and tests/test_parity.py fails if either surface gains a resource the other lacks.

AI-friendliness

This CLI is designed to be driven by coding agents as well as humans:

• --plain everywhere (and auto-enabled when stdout isn't a TTY): no live panels, no styling, no mid-message line wrapping.
• --json on every query command: stdout is pure JSON, human chrome goes to stderr.
• Uniform exit codes: 0 success · 1 domain failure (failed run, not found, triage findings) · 2 usage/auth/API error · 130 interrupted.
• A repo-local Claude skill (.claude/skills/databricks-tools/) documents the intent-to-command mapping for agents.

Development

asdf install            # python + just from .tool-versions
uv sync --extra dev
just hooks              # install pre-commit
just verify             # ruff check + format check + ty + pytest (the CI gate)
just run --help         # run the CLI from source

Conventions and architecture: see AGENTS.md.

Roadmap

• dbtools repos list|status sync-status details beyond branch/head commit.
• TUI: in-table row filtering; true pagination beyond the m cap-doubling.