dmt

High-performance tool for database migrations between SQL Server, PostgreSQL, and MySQL — driven from a CLI, an interactive terminal UI, or a browser (--webui), all from a single binary.

Quick Start

dmt runs deterministically with no AI provider required — type mapping, error diagnosis, DB tuning, and runtime parameter adjustment all use built-in rule catalogs. AI features remain available as an opt-in enhancement for vendor-specific edge cases (see Optional AI Enhancements below).

# 1. Create secrets file (no AI section by default; pass --with-ai to opt in)
./dmt init-secrets

# 2. Create a config (interactive)
./dmt init -o my-migration.yaml

# 3. Run the migration
./dmt run --config my-migration.yaml

For interactive use, launch without arguments:

./dmt

Interactive Mode

Launch the tool without arguments to enter the Interactive Shell, a modern TUI designed for ease of use.

./dmt

Features

Slash Commands: Type / to see all available commands (e.g., /run, /preflight, /diagnose, /analyze). The full CLI / TUI / WebUI parity table lives in docs/TUI_COMMANDS.md.
CLI parity: /run --dry-run previews the plan, /preflight --ai-review checks readiness, /validate --ai-triage and /diagnose triage failures, /ai config-review generates patch recommendations and a runbook — the same renderers the CLI uses.
Session defaults: /session KEY VALUE keeps sticky per-session defaults (config, profile, state-file, observability and audit settings) so they don't need repeating on every command.
Resume: Use /resume to continue interrupted migrations (--force-resume after config changes).
Auto-Completion:
- Commands: Tab-complete commands like /validate or /history.
- Files: Type @ to browse and select configuration files from your current directory (e.g., /run @conf<TAB>).
Configuration Wizard: Type /wizard to interactively create or edit your config.yaml. It guides you through connection details, SSL settings, and performance tuning.
Live Monitoring: Watch migration progress with real-time logs and visual status indicators.
Git Integration: View your current branch and repository status directly in the status bar.

Web UI

Launch a browser-based operator console — the same command surface as the TUI, served as an authenticated single-page app straight from the binary (no separate install, no build step):

./dmt --webui        # loopback bind; prints a one-click URL with an auto-generated token
./dmt --gui          # same thing, but opens a browser window automatically and acts like a desktop app

It runs the live migration dashboard (progress streamed over Server-Sent Events), readiness checks (preflight/validate/diagnose), the guided setup wizard, profile and session management, and a ⌘K command palette. It calls the same orchestrator the CLI and TUI do — no forked logic.

--gui turns it into a desktop app with no native shell and no new runtime dependency (dmt stays a single pure-Go, CGO_ENABLED=0 binary): it auto-opens a browser, hands off to an already-running instance instead of failing to bind, exits shortly after the window closes (never mid-migration), and is installable as a PWA (Chrome/Edge "Install", Safari "Add to Dock"). See docs/WEBUI.md for details.

Loopback-only by default. A remote/server bind (--webui-addr 0.0.0.0:8484) requires an auth token plus TLS — either native (--webui-tls-cert/--webui-tls-key) or a TLS-terminating reverse proxy (--webui-insecure behind it, optionally with --webui-trusted-proxy for per-client rate-limiting). Brute-force throttling, session renewal, and a strict CSP are on by default. The setup wizard stores DB passwords in ${file:} secret files rather than plaintext.

See docs/WEBUI.md for the security model, remote-deployment guide (nginx/Caddy), and maturity notes (ready for local single-operator use; beta for remote/team).

Security

Credential redaction - passwords are never stored in the state database or logs
DSN injection protection - connection strings URL-encode credentials to prevent injection
SQL injection protection - internal SQLite queries use whitelist validation for table names
Secret templates - use ${env:VAR} or ${file:/path} instead of plaintext passwords in config
Secure permissions - state files (0600) and data directories (0700) enforced automatically

Incremental Sync

For large databases with frequent updates, use date-based incremental loading to dramatically reduce sync times. Instead of transferring all rows every time, only rows modified since the last sync are transferred.

Recommended Workflow

# Step 1: Initial load (fast bulk copy, creates tables)
./dmt -c config.yaml run   # target_mode: drop_recreate

# Step 2: Incremental syncs (uses highwater marks)
./dmt -c config.yaml run   # target_mode: upsert

Performance Impact

Scenario	Time
Full sync (19M rows)	1m 47s
Incremental (no changes)	12 seconds

Configuration

migration:
  target_mode: upsert     # Required for incremental sync

  # Date columns for incremental loading (tries each in order)
  # Supported types: datetime, datetime2, timestamp, timestamptz, date
  date_updated_columns:
    - UpdatedAt           # Common convention
    - ModifiedDate        # SQL Server convention
    - LastModified
    - CreationDate        # Fallback for append-only tables

How It Works

First run: Full load of all rows, records sync timestamp per table
Subsequent runs: Only fetches rows where date_column > last_sync_timestamp
Highwater marks: Stored automatically in the state database (~/.dmt/migrate.db)
Upsert logic: INSERTs new rows, UPDATEs changed rows, preserves target-only rows

Important Notes

Upsert requires existing tables - Run drop_recreate first for initial load
Primary keys required - Both source and target tables must have PKs
Tables without date columns - Fall back to full table comparison (slower)

AI-Driven Real-Time Parameter Adjustment

Continuously monitor migration performance and automatically adjust parameters in real-time. The tuner receives live system resource data (CPU cores, available RAM, connection limits) and makes informed decisions without hard-coded safety guards — an effectiveness tracker measures each adjustment's impact and pauses tuning if consecutive changes hurt performance.

Quick Start

Enable in config.yaml:

migration:
  runtime_tuning: true
  runtime_tuning_interval: 5s

ai:
  api_key: ${ANTHROPIC_API_KEY}
  provider: anthropic

Pre-#211 these fields were named ai_adjust / ai_adjust_interval. The old names are still accepted but emit a deprecation warning; rename to runtime_tuning to silence it. The runtime tuner is deterministic and rule-based — it does not call out to the AI provider.

How It Works

Continuous Monitoring: Every 30 seconds, collects performance metrics:
- Windowed throughput (rows/sec), memory usage, CPU utilization
- Transfer time breakdown (source query/scan vs. target write percentages)
- Connection pool utilization (active, idle, wait counts per pool)
- Queue depth, active workers, error count
Resource-Aware Controller: The rule engine receives live system state:
- CPU cores, available/used RAM, max database connections
- Connection pool saturation (active/idle/waits for source and target)
- Data profile: total tables, total rows, avg row size, estimated pipeline memory
- Table-level progress (complete/failed/remaining)
- Current parameter values and adjustment history
- Effectiveness of previous adjustments
Rule-Based Decisions: Analyzes trends and recommends adjustments:
- Scale up: Increase workers or chunk_size if resources available
- Scale down: Reduce workers to minimize lock contention
- Reduce chunk: Decrease batch size if memory pressure detected
- Continue: Maintain current parameters if performance optimal
Safety Mechanisms:
- Post-adjustment cooldown (90s) — waits for metrics to stabilize before next adjustment
- Effectiveness tracking — measures throughput change after each adjustment
- Consecutive-negative breaker — pauses tuning after 3 adjustments that hurt performance
- Completion skip — no adjustments when transfer is >90% complete
- Updates applied at chunk boundaries, never mid-transfer

Performance

Tested on Stack Overflow 2013 dataset (106.5M rows, MSSQL to PostgreSQL):

Configuration                     Transfer    Overall    Throughput
──────────────────────────────────────────────────────────────────────
Smartconfig + runtime tuning      2m 33s      3m 08s     697K rows/sec
Runtime tuning only               5m 14s      5m 50s     339K rows/sec

Smartconfig analyzes source schema, system resources, driver profiles, and completed-run history to set initial parameters. Runtime tuning monitors live metrics (throughput, CPU, memory, pool utilization, transfer time breakdown) and adjusts parameters mid-migration through the rule-based controller.

Configuration

migration:
  runtime_tuning: true             # Enable/disable rule-based runtime controller (default: true)
  runtime_tuning_interval: 5s      # Evaluation interval (default: 5s)

  # Initial parameters (runtime tuner adjusts from here)
  chunk_size: 10000
  workers: 4
  read_ahead_buffers: 8
  write_ahead_writers: 2
  parallel_readers: 2

ai:
  api_key: ${ANTHROPIC_API_KEY}
  provider: anthropic              # Also: openai, gemini, ollama, lmstudio
  model: claude-sonnet-5           # Recommended default
  timeout_seconds: 30

Cost

API Calls: ~1-2 per minute (60s cache + 90s cooldown between adjustments)
Cost: Model-dependent; AI calls are sparse and cached
Fallback: Heuristic rules apply if AI unavailable

Troubleshooting

Runtime tuning not happening:

Check logs for the controller-startup message
Verify runtime_tuning: true in config (legacy alias ai_adjust still works)
Tuner skips adjustments when >90% complete or during the post-adjustment cooldown

Performance degradation:

The effectiveness tracker will automatically pause after consecutive negative adjustments
Check --verbosity debug logs for adjustment-effect measurements
Disable with runtime_tuning: false to use fixed parameters

Encrypted Profiles (SQLite)

You can store full configuration profiles (including secrets) encrypted at rest inside the same SQLite database used for run history.

Master key

Set DMT_MASTER_KEY to a base64-encoded 32-byte key.
Example key generation (POSIX):
```
openssl rand -base64 32
```
Without this key, profile operations will fail, but YAML-based workflows continue to work.

CLI workflow

# Save a profile from YAML (encrypts and stores in SQLite)
DMT_MASTER_KEY=... ./dmt profile save --name prod --config config.yaml

# List profiles
DMT_MASTER_KEY=... ./dmt profile list

# Run using a profile
DMT_MASTER_KEY=... ./dmt run --profile prod

# Export a profile back to YAML
DMT_MASTER_KEY=... ./dmt profile export --name prod --out config.yaml

YAML profile name (optional)

profile:
  name: prod
  description: |
    Production profile for nightly migrations.
    Uses MSSQL source and PostgreSQL target.

If profile.name is present, profile save can infer the name when --name is omitted. Descriptions are shown in profile list.

TUI workflow

/profile save prod @config.yaml
/profile save @config.yaml      # infers name from profile.name or filename
/profile list
/run --profile prod
/profile export prod @config.yaml

Airflow note

Profiles are stored as encrypted blobs in the same SQLite DB (~/.dmt/migrate.db by default).
In Airflow, you can set DMT_MASTER_KEY via your secrets backend and run profile save at deploy time, or stick with YAML + env vars for CI/CD.
You can relocate the SQLite DB by setting migration.data_dir in your config (e.g., to a shared volume).
On first run, the default data directory (~/.dmt) is created automatically if it does not exist.

Optional AI Enhancements

dmt's core migration path is fully deterministic — type mapping, error diagnosis, DB tuning, and runtime parameter adjustment all work without AI. The features described here are optional enhancements for cases the deterministic catalog doesn't cover (vendor-specific column types like Oracle hierarchyid or MSSQL geography).

To opt in (two safe paths):

Brand-new install — run dmt init-secrets --with-ai to seed the AI provider section at template-creation time.
Existing secrets file — APPEND an ai: block to your ~/.secrets/dmt-config.yaml. Do NOT run --force --with-ai; that would overwrite your master_key and Slack webhook values.

Example snippet to append:

ai:
  default_provider: anthropic
  providers:
    anthropic:
      api_key: ""  # Get from https://console.anthropic.com/
      model: "claude-sonnet-5"

All AI features share common configuration under the ai section.

AI Quick Start

Simply add your API key to enable AI features:

ai:
  api_key: ${ANTHROPIC_API_KEY}   # or OPENAI_API_KEY or GEMINI_API_KEY

This auto-enables AI type mapping. Provider defaults to Anthropic.

Supported Providers

Provider	Config Value	Default Model	API Key Variable
Anthropic (default)	`anthropic`	`claude-sonnet-5`	`ANTHROPIC_API_KEY`
OpenAI	`openai`	`gpt-5.5`	`OPENAI_API_KEY`
Google Gemini	`gemini`	`gemini-2.0-flash`	`GEMINI_API_KEY`
Ollama (local)	`ollama`	-	-
LM Studio (local)	`lmstudio`	-	-

Full Configuration

ai:
  api_key: ${ANTHROPIC_API_KEY}  # Required - your API key
  provider: anthropic            # Optional - anthropic (default), openai, gemini, ollama, lmstudio
  model: claude-sonnet-5         # Optional - uses provider default if not set

  type_mapping:
    enabled: true                # Auto-enabled when api_key is set
    cache_file: ~/.dmt/type-cache.json

AI Type Mapping

Automatically infers the best target type for unknown or complex source types.

How it works:

Data Sampling: Samples up to 5 rows from each table for context
Intelligent Inference: AI analyzes column metadata plus sample values
Cross-Engine Awareness: Understands encoding differences (e.g., PG varchar → MSSQL nvarchar)
Caching: Mappings cached to minimize API calls

When to use:

Custom domains, user-defined types, or database-specific types
Cross-engine migrations with different type systems
Unicode handling (AI correctly infers nvarchar for UTF-8 text)

Smart Config Detection

Analyze your source database and get optimal configuration suggestions:

./dmt -c config.yaml analyze

Output example:

# AI-detected configuration suggestions

migration:
  date_updated_columns:
    - UpdatedAt
    - ModifiedDate
    - LastModified

  exclude_tables:
    - temp_imports
    - audit_log
    - __EFMigrationsHistory

  chunk_size: 150000

What it detects:

Date columns: Columns suitable for incremental sync (UpdatedAt, ModifiedDate, etc.)
Exclude tables: Tables that should probably be excluded (temp, log, archive, etc.)
Chunk size: Optimal chunk size based on average row sizes

Error Diagnosis

When a table transfer fails, DMT uses its deterministic diagnosis catalog to provide actionable suggestions. Unmatched errors are counted in observability as catalog-growth candidates; they are not sent to an AI provider.

Example output:

Table Orders failed: pq: invalid input syntax for type integer: "abc"

  Diagnosis:
    Cause: Data type mismatch - column contains non-numeric values being inserted into integer column
    Suggestions:
      - Check source data for non-numeric values in numeric columns
      - Use TEXT type instead of INTEGER for this column
      - Add data transformation to filter/convert invalid values
    Confidence: high

Features:

Automatic: Runs automatically when a transfer fails
Context-aware: Uses table schema, column types, and source/target DB info when available
Local: No error text is sent to an AI provider
Categorized: Errors classified as type_mismatch, constraint, permission, connection, or data_quality

Common diagnoses:

Error Type	Diagnosis
Type mismatch	Identifies incompatible column types and suggests mappings
NULL constraint	Detects NULL values in NOT NULL columns
Foreign key	Identifies missing parent records or ordering issues
Permission	Suggests required grants or role assignments
Connection	Diagnoses timeout, authentication, or network issues

Requirements:

AI must be configured (ai.api_key set)
Works with all supported providers (Claude, OpenAI, Gemini)

Cost Considerations

Each unique type mapping requires one API call (cached for future runs)
Typical migration: 20-50 API calls on first run, zero on subsequent runs
Smart config analysis: 0 API calls (uses pattern matching, not AI)
Error diagnosis: 1 API call per unique error (cached to avoid duplicates)

State File Backend (Airflow/Kubernetes)

For headless environments like Airflow or Kubernetes where SQLite may be impractical, you can use a YAML-based state file instead.

# Use a YAML state file instead of SQLite
./dmt -c config.yaml --state-file /tmp/migration-state.yaml run

# Resume using the same state file
./dmt -c config.yaml --state-file /tmp/migration-state.yaml resume

# Check status
./dmt -c config.yaml --state-file /tmp/migration-state.yaml status

# View history
./dmt -c config.yaml --state-file /tmp/migration-state.yaml history

State file features:

Portable - Single YAML file, easy to store in cloud storage or shared volumes
Human-readable - Inspect and debug migration state directly
Chunk-level resume - Same resume granularity as SQLite backend
Error tracking - Failed runs store the error message for debugging

Example state file:

run_id: a1b2c3d4
started_at: 2025-01-15T10:30:00Z
completed_at: 2025-01-15T10:45:00Z
status: success
source_schema: dbo
target_schema: public
config_hash: 2bd314ff9b5251d5
config_path: /path/to/config.yaml
tables:
  transfer:dbo.Users:
    status: success
    last_pk: 2465713
    rows_done: 2465713
    rows_total: 2465713
    task_id: 1001
  transfer:dbo.Posts:
    status: success
    last_pk: 17142169
    rows_done: 17142169
    rows_total: 17142169
    task_id: 1002

When to use state file vs SQLite:

Feature	SQLite (default)	State File (`--state-file`)
History	Full run history	Current run only
Profiles	Encrypted storage	Not supported
Best for	Desktop, TUI	Airflow, Kubernetes, CI/CD
Persistence	Local database	Any storage (S3, NFS, etc.)

Airflow Integration

The CLI provides first-class support for Airflow with machine-readable outputs and deterministic run IDs.

Airflow CLI Flags

Flag	Description
`--run-id <id>`	Explicit run ID (default: auto-generated UUID). Use `{{ dag_run.run_id }}` in Airflow.
`--output-json`	Output JSON result to stdout on completion (logs go to stderr)
`--output-file <path>`	Write JSON result to file on completion
`--log-format=json`	Structured JSONL logging (one JSON object per line)
`status --json`	Output current status as JSON (for Airflow sensors)
`--force-resume`	Bypass config hash validation on resume
`resume --abandon`	Fenced, explicit abandonment of the resumable run for the configured target
`resume --abandon-reason <text>`	Persist the operator reason with `--abandon`

BashOperator Example

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('mssql_to_pg_migration', start_date=datetime(2025, 1, 1)) as dag:

    migrate = BashOperator(
        task_id='migrate_data',
        bash_command='''
            /opt/dmt \
                --run-id "{{ dag_run.run_id }}" \
                --output-json \
                --output-file /tmp/{{ dag_run.run_id }}_result.json \
                --log-format=json \
                --state-file /tmp/{{ dag_run.run_id }}_state.yaml \
                -c /opt/configs/migration.yaml \
                run
        ''',
        do_xcom_push=True,  # Captures stdout JSON for downstream tasks
    )

KubernetesPodOperator Example

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

migrate = KubernetesPodOperator(
    task_id='migrate_data',
    name='dmt',
    image='your-registry/dmt:latest',
    cmds=['/dmt'],
    arguments=[
        '--run-id', '{{ dag_run.run_id }}',
        '--output-json',
        '--log-format', 'json',
        '--state-file', '/data/state.yaml',
        '-c', '/config/migration.yaml',
        'run'
    ],
    volumes=[...],
    volume_mounts=[...],
    get_logs=True,
    do_xcom_push=True,
)

JSON Output Format

Migration Result (--output-json / --output-file):

{
  "run_id": "dag_2025_01_15",
  "status": "success",
  "started_at": "2025-01-15T10:00:00Z",
  "completed_at": "2025-01-15T10:01:34Z",
  "duration_seconds": 94,
  "tables_total": 9,
  "tables_success": 9,
  "tables_failed": 0,
  "rows_transferred": 19310703,
  "rows_per_second": 205432,
  "failed_tables": [],
  "table_stats": [
    {"name": "Users", "rows": 299398, "status": "success"},
    {"name": "Posts", "rows": 3729195, "status": "success"}
  ]
}

Status Result (status --json - for Airflow sensors):

{
  "run_id": "dag_2025_01_15",
  "status": "running",
  "phase": "transferring",
  "started_at": "2025-01-15T10:00:00Z",
  "tables_total": 9,
  "tables_complete": 5,
  "tables_running": 2,
  "tables_pending": 2,
  "rows_transferred": 12500000,
  "progress_percent": 65
}

JSON Log Format (--log-format=json - JSONL to stderr):

{"ts":"2025-01-15T10:00:01Z","level":"info","msg":"Starting migration run: dag_2025_01_15"}
{"ts":"2025-01-15T10:00:02Z","level":"info","msg":"Found 9 tables"}
{"ts":"2025-01-15T10:00:03Z","level":"info","msg":"Transferring data..."}

Config Hash Validation

When using --state-file, the tool stores a hash of the config at run start. On resume:

If the config has changed, resume is blocked with an error showing both hashes
Use --force-resume to bypass this check (useful for intentional config tweaks)
This prevents accidentally resuming with mismatched source/target settings

# Config changed error
Error: config changed since run started (hash 6abfe692 != 1cddb8e0), use --force-resume to override

# Force resume anyway
./dmt --state-file state.yaml -c config.yaml resume --force-resume

Airflow Sensor Pattern

Poll migration status from a separate task:

from airflow.sensors.python import PythonSensor
import subprocess
import json

def check_migration_status(**context):
    result = subprocess.run([
        '/opt/dmt',
        '--state-file', f"/tmp/{context['dag_run'].run_id}_state.yaml",
        '-c', '/opt/configs/migration.yaml',
        'status', '--json'
    ], capture_output=True, text=True)

    status = json.loads(result.stdout)
    if status['status'] == 'success':
        return True
    elif status['status'] == 'failed':
        raise Exception(f"Migration failed: {status.get('error')}")
    return False  # Still running

sensor = PythonSensor(
    task_id='wait_for_migration',
    python_callable=check_migration_status,
    poke_interval=60,
    timeout=3600,
)

Performance

222K-697K rows/sec depending on direction and row width
MSSQL → PG: 697K rows/sec with smartconfig + runtime tuning (106.5M rows in 2m33s)
PG → MSSQL: 645K rows/sec (PG streaming + TDS bulk copy)
PG → PG: 563K rows/sec (COPY protocol both ends)
MSSQL → MSSQL: 222K rows/sec (TDS both ends)
Auto-tuning based on CPU cores, available RAM, completed-run history, and rule-based runtime adjustments
Single binary - no runtime dependencies, no CGO

Supported Databases

Database	As Source	As Target	Write Method	Auth
PostgreSQL	✓	✓	COPY protocol (fastest)	Password
SQL Server	✓	✓	TDS bulk copy	Password
MySQL	✓	✓	Multi-row INSERT	Password

Kerberos / SPNEGO is not currently supported — the DSN-building plumbing exists but is not wired to the runtime drivers, and we don't yet have a Kerberized integration environment to verify against. See #251 for the re-enable plan.

All combinations are supported, including same-engine migrations (PG→PG, MSSQL→MSSQL, MySQL→MySQL).

Same-Engine Migrations

Use cases: database cloning, environment sync (dev → staging → prod), disaster recovery, data center migrations.

source:
  type: postgres
  host: source-pg.example.com
target:
  type: postgres
  host: target-pg.example.com
migration:
  target_mode: upsert  # or drop_recreate

PostGIS Spatial Data Support

Cross-engine migrations (PG→MSSQL) preserve spatial reference systems:

SRID preservation - reads SRID from PostGIS geometry_columns/geography_columns metadata
Automatic conversion - WKT text converted to SQL Server geography/geometry with correct SRID
Default fallback - uses SRID 4326 (WGS84) when source SRID is 0 or unset

Features

Database Support

PostgreSQL, SQL Server, MySQL - migrate between any combination
Bulk copy protocols - PostgreSQL COPY, TDS bulk copy, MySQL LOAD DATA for maximum throughput
SSL/TLS encryption - configurable per connection

Transfer Engine

Pipelined I/O - read-ahead buffering with parallel writers (222K-697K rows/sec)
Keyset pagination - efficient partitioning for integer PKs (no OFFSET degradation)
ROW_NUMBER pagination - automatic fallback for composite/varchar PKs
Parallel partitioning - large tables split via NTILE for concurrent transfer
Auto-tuning - workers and pools sized from CPU, with target-aware chunk sizing constrained by the memory budget
Memory-bounded - configurable memory cap (default: 70% available RAM)

AI Integration

Multi-provider - Claude, OpenAI, Gemini, Ollama, LM Studio
Type mapping - LLM-powered cross-database type inference with caching
Smart config analysis - analyze source database and recommend optimal parameters (analyze command)
Real-time parameter tuning - monitor performance and auto-adjust workers, chunk size, buffers mid-migration
Error diagnosis - AI-powered root cause analysis with remediation suggestions

Checkpoint & Resume

Chunk-level resume - progress saved every N chunks, resume from exact position
Table-level resume - skip already-completed tables on restart
Idempotent retries - partition cleanup on retry prevents duplicates
State backends - SQLite (default) or YAML file for Airflow/headless environments
Config validation - prevents resume with mismatched configuration

Incremental Sync

Upsert mode - INSERT new rows, UPDATE changed rows, preserve target-only data
Date-based highwater marks - only transfer rows modified since last sync
Configurable date columns - tries multiple column names in order

Interactive Mode (TUI)

Slash commands - /run (incl. --dry-run), /resume, /preflight, /validate, /diagnose, /analyze, /ai config-review, /session, /init-secrets, /cache clear, and more — see docs/TUI_COMMANDS.md for the CLI parity table
Auto-completion - tab-complete commands and @ file browser for config selection
Configuration wizard - interactive setup for source, target, and tuning parameters
Live monitoring - real-time migration progress with log capture

CLI & Automation

Commands - run, resume, status, validate, history, health-check, analyze, init, profile
Dry-run mode - preview migration plan without execution
JSON output - structured results to stdout or file for orchestration tools
Exit codes - semantic codes for retry logic (success, transient, config error, cancelled)
Airflow integration - YAML state files, explicit run IDs, BashOperator/KubernetesPodOperator support
Graceful shutdown - SIGINT/SIGTERM handling with configurable timeout and checkpoint save

Security

Encrypted profiles - AES-encrypted configs stored in SQLite with master key
Secret templates - ${env:VAR}, ${file:/path} for credentials (no plaintext in config)
Credential redaction - passwords never stored in state database or logs
Secure permissions - 0600 files, 0700 directories enforced automatically

Schema Management

Full schema transfer - tables, primary keys, indexes, foreign keys, check constraints
Identity/sequence reset - preserves auto-increment values after transfer
Table filtering - include/exclude tables with glob patterns
Strict consistency mode - table locks instead of NOLOCK for consistent reads

Observability

Progress bar - real-time throughput stats with ETA
Slack notifications - start, completion, and failure alerts
JSON progress - streaming updates to stderr for monitoring dashboards
Verbosity levels - debug, info, warn, error with text or JSON log format
Row count validation - automatic post-transfer verification
Sample data validation - random row verification with composite PK support

Deployment

Single binary - no runtime dependencies, no CGO
Cross-platform - Linux, macOS (Intel + Apple Silicon), Windows
YAML configuration - with environment variable expansion

Installation

Download pre-built binaries

Download from GitHub Releases. Releases ship raw binaries per platform plus a checksums.txt:

# Linux x64
curl -LO https://github.com/johndauphine/dmt/releases/download/v5.4.0/dmt-v5.4.0-linux-amd64
chmod +x dmt-v5.4.0-linux-amd64 && ./dmt-v5.4.0-linux-amd64 --version

# macOS Apple Silicon
curl -LO https://github.com/johndauphine/dmt/releases/download/v5.4.0/dmt-v5.4.0-darwin-arm64
chmod +x dmt-v5.4.0-darwin-arm64 && ./dmt-v5.4.0-darwin-arm64 --version

# macOS Intel
curl -LO https://github.com/johndauphine/dmt/releases/download/v5.4.0/dmt-v5.4.0-darwin-amd64
chmod +x dmt-v5.4.0-darwin-amd64

# Windows (PowerShell)
Invoke-WebRequest -Uri https://github.com/johndauphine/dmt/releases/download/v5.4.0/dmt-v5.4.0-windows-amd64.exe -OutFile dmt.exe

# Verify a download (any platform)
curl -LO https://github.com/johndauphine/dmt/releases/download/v5.4.0/checksums.txt
shasum -a 256 -c --ignore-missing checksums.txt

Build from source

Requires Go 1.25.7+

git clone https://github.com/johndauphine/dmt.git
cd dmt
CGO_ENABLED=0 go build -o dmt ./cmd/dmt

Go install

go install github.com/johndauphine/dmt/v5/cmd/dmt@latest

The /v5 module suffix makes @latest resolve to v5 release tags, and this installs a binary named dmt. Immutable tags published before this fix cannot be repaired; use the first v5 release published after this change (or a newer one). For a pinned, reproducible install, prefer a specific version or the release binaries.

Quick Start

SQL Server to PostgreSQL

Create a config.yaml:

source:
  type: mssql              # optional, default for source
  host: sqlserver.example.com
  port: 1433
  database: MyDatabase
  user: sa
  password: ${MSSQL_PASSWORD}
  schema: dbo

target:
  type: postgres           # optional, default for target
  host: postgres.example.com
  port: 5432
  database: mydb
  user: postgres
  password: ${PG_PASSWORD}
  schema: public

migration:
  workers: 8
  chunk_size: 200000

PostgreSQL to SQL Server

source:
  type: postgres
  host: postgres.example.com
  port: 5432
  database: mydb
  user: postgres
  password: ${PG_PASSWORD}
  schema: public

target:
  type: mssql
  host: sqlserver.example.com
  port: 1433
  database: MyDatabase
  user: sa
  password: ${MSSQL_PASSWORD}
  schema: dbo

migration:
  workers: 8
  chunk_size: 200000

SQL Server target uses TDS Bulk Copy protocol (mssql.CopyIn) for optimal performance (~130,000 rows/sec).

Run the migration:

./dmt -c config.yaml run

If interrupted, resume:

./dmt -c config.yaml resume

Configuration Reference

The configuration file uses YAML format. Environment variables can be used with ${VAR_NAME} syntax.

Source Database Settings

The source section configures the database to migrate FROM.

Parameter	Required	Default	Description
`type`	No	`mssql`	Database type: `mssql`, `postgres`, or `mysql`
`host`	Yes	-	Database server hostname or IP address
`port`	No	Auto	Database server port (1433/5432/3306)
`database`	Yes	-	Database name
`user`	Yes	-	Username for authentication
`password`	Yes	-	Password for authentication. Supports `${ENV_VAR}` syntax
`schema`	No	Auto	Schema containing tables to migrate

SSL/TLS Settings (source):

Parameter	Required	Default	Description
`ssl_mode`	No	`require`	PostgreSQL SSL mode: `disable`, `require`, `verify-ca`, `verify-full`. MySQL accepts the same names plus `preferred` for explicit downgradeable TLS.
`encrypt`	No	`true`	SQL Server encryption: `true` or `false`
`trust_server_cert`	No	`false`	SQL Server: Skip certificate validation (use only for testing)
`packet_size`	No	`32767`	SQL Server TDS packet size in bytes (max: 32767). Larger packets improve throughput.

Kerberos / SPNEGO auth is not currently supported (see #251). Setting auth: kerberos is rejected at config-load. The auth, krb5_conf, keytab, realm, spn, and gssencmode fields are reserved in the schema for the eventual re-enable.

Target Database Settings

The target section configures the database to migrate TO. It uses the same parameters as source.

Parameter	Required	Default	Description
`type`	No	`postgres`	Database type: `mssql`, `postgres`, or `mysql`
`host`	Yes	-	Database server hostname or IP address
`port`	No	Auto	Database server port (5432/1433/3306)
`database`	Yes	-	Database name
`user`	Yes	-	Username for authentication
`password`	Yes	-	Password for authentication
`schema`	No	Auto	Target schema for migrated tables

The same SSL/TLS settings are available for target.

Migration Settings

The migration section controls how data is transferred.

Connection Pool Settings:

Parameter	Required	Default	Description
`max_source_connections`	No	Auto-sized	Maximum source database connection pool size
`max_target_connections`	No	Auto-sized	Maximum target database connection pool size

Parallelism Settings:

Parameter	Required	Default	Description
`workers`	No	CPU-based (4-12)	Number of parallel transfer workers. The load-time formula uses 4 workers through 6 cores, `cores - 2` through 13 cores, and 12 workers above that.
`chunk_size`	No	RAM-shaped (50,000-200,000 before safety clamps)	Rows per chunk. The load-time formula starts at 75,000 and scales with available memory; the retained memory envelope may reduce it, and pre-transfer tuning may replace generated values after schema/history analysis.
`max_partitions`	No	Same as effective `workers`	Maximum partitions for large table parallelism
`large_table_threshold`	No	5,000,000	Tables with more rows than this are partitioned

Omitted performance values are initialized by the legacy load-time formulas, then the schema- and history-aware tuner may replace generated values before a run. Explicit user or secrets values remain pinned against policy tuning. Before transfer, representative-width memory and target-protocol limits may reduce the global chunk, including a pinned value. During steady transfer, measured chunk bytes share one admission budget and MemoryGuard is the backstop. The complete-inventory table model gates runtime writer-count transitions and ratchets later chunk or batch growth only after a transition is applied.

Table Filtering:

Parameter	Required	Default	Description
`include_tables`	No	All tables	List of glob patterns for tables to include (e.g., `Users`, `Order*`)
`exclude_tables`	No	None	List of glob patterns for tables to exclude (e.g., `temp_`, `__`)

Target Table Handling:

Parameter	Required	Default	Description
`target_mode`	No	`drop_recreate`	How to handle existing tables: `drop_recreate` (drop and recreate) or `upsert` (incremental sync). Note: `upsert` requires target tables to already exist - run `drop_recreate` first for initial load.
`date_updated_columns`	No	None	List of column names to check for last-modified date (e.g., `UpdatedAt`, `ModifiedDate`). Enables incremental sync - only rows modified since last sync are transferred.
`data_dir`	No	`~/.dmt`	Directory for state database and temporary files

Schema Object Creation:

Parameter	Required	Default	Description
`create_indexes`	No	`true`	Create non-primary key indexes after data transfer
`create_foreign_keys`	No	`true`	Create foreign key constraints after data transfer
`create_check_constraints`	No	`true`	Create CHECK constraints after data transfer
`fail_on_schema_drift`	No	`false`	Abort before transfer if the source schema differs from the last successful run's snapshot. By default, dmt reports drift and continues.

Consistency Settings:

Parameter	Required	Default	Description
`strict_consistency`	No	`false`	Read source data from an engine-native stable view. PostgreSQL, MySQL/MariaDB, and SQL Server support parallel readers; SQLite intentionally uses one reader.
`strict_consistency_scope`	No	`table`	`table` creates an independent view per table. `migration` shares one PostgreSQL exported snapshot or SQL Server database snapshot across every table and partition. Requires `strict_consistency: true`.

Consistency scope (#640). strict_consistency starts a source transaction before target preparation, then routes every page of that table through one stable view. PostgreSQL imports an exported MVCC snapshot into parallel reader transactions without blocking writers. InnoDB MySQL/MariaDB starts parallel repeatable-read sessions inside a brief LOCK TABLES window, which requires LOCK TABLES privilege and pauses table writes only while the sessions start. SQL Server holds one shared table lock while parallel readers copy the table, so writes to that table wait for its transfer to finish. SQLite uses one serializable reader by design. Table scope keeps one unpartitioned job per table because these views do not span independent jobs; eligible integer and composite keysets can still use parallel readers inside that job. Unsupported sources fail before target mutation. The guarantee is per table, not one snapshot shared across independently transferred tables.

Migration scope (#663, #683). PostgreSQL exports one MVCC snapshot before the transfer phase and imports it into every table reader and partition. SQL Server creates one run-scoped database snapshot and routes table reads and partition planning through it; live writers remain unblocked while changed pages consume copy-on-write disk. Both mechanisms provide one cross-table source instant and safely enable strict partition jobs. PostgreSQL resume in a new process opens a new epoch; a SQL Server snapshot survives a crash and is reused on resume, which fails closed if that snapshot is missing. MySQL and SQLite do not support migration scope. See Restartability for lifecycle, prerequisite, and blocking details.

Strict validation (#664). For a full-table strict transfer, dmt captures COUNT(*) through that same pinned source transaction before it prepares the target. Validation compares the target with this persisted snapshot count, not with a later live source count. If the live source has changed since the snapshot, dmt reports the signed drift as informational and still passes when the target matches the snapshot. dmt validate uses the latest matching run's stored strict evidence. Incremental jobs with a date filter transfer a window rather than a whole table, so they retain the normal live-count validation behavior.

Validation Settings:

Parameter	Required	Default	Description
`sample_validation`	No	`false`	Enable random row sampling to verify data integrity
`sample_size`	No	100	Number of random rows per table to verify

Performance Tuning:

Parameter	Required	Default	Description
`read_ahead_buffers`	No	Memory-shaped (4-32)	Number of chunks to buffer ahead of writers
`write_ahead_writers`	No	Target- and CPU-scaled (minimum 1)	Parallel writers per job; pre-transfer tuning may replace the generated value
`parallel_readers`	No	CPU-scaled (minimum 2)	Parallel readers per job. Pin to 1 for local databases when appropriate
`source.chunk_size`	No	Same as `migration.chunk_size`	Batch size for reading from source database
`target.chunk_size`	No	Same as `migration.chunk_size`	Batch size for writing to target database

AI Settings

The ai section configures AI-powered features.

Parameter	Required	Default	Description
`ai.api_key`	Yes (if using AI)	-	API key for the AI provider
`ai.provider`	No	`anthropic`	AI provider: `anthropic`, `openai`, `gemini`, `ollama`, or `lmstudio`
`ai.model`	No	Provider default	Model to use (e.g., `claude-sonnet-5`, `gpt-5.5`, `gemini-2.0-flash`)
`ai.timeout_seconds`	No	`30`	API request timeout
`ai.type_mapping.enabled`	No	Auto	Enable AI type mapping (auto-enabled when api_key is set)
`ai.type_mapping.cache_file`	No	`~/.dmt/type-cache.json`	Path to cache AI type mappings
`ai.smart_config.enabled`	No	`false`	Enable smart config detection
`ai.smart_config.detect_date_columns`	No	`true`	Detect date_updated_columns candidates
`ai.smart_config.detect_exclude_tables`	No	`true`	Detect tables to exclude
`ai.smart_config.suggest_chunk_size`	No	`true`	Suggest optimal chunk size

Slack Notification Settings

Parameter	Required	Default	Description
`enabled`	No	`false`	Enable Slack notifications
`webhook_url`	Yes (if enabled)	-	Slack incoming webhook URL
`channel`	No	Webhook default	Channel to post to (e.g., `#data-engineering`)
`username`	No	`dmt`	Bot username for messages

Kerberos Authentication

Not currently supported — tracked in #251. The auth, krb5_conf, keytab, realm, spn, and gssencmode fields are reserved in the YAML schema for the eventual re-enable; auth: kerberos is rejected at config-load until a verifiable Kerberized test environment is in place.

Example Configurations

Ready-to-use example configuration files are available in the examples/ directory:

File	Description
`config-mssql-to-pg.yaml`	SQL Server → PostgreSQL with password auth
`config-pg-to-mssql.yaml`	PostgreSQL → SQL Server with password auth
`config-local.yaml`	Minimal config for local Docker development
`config-production.yaml`	Full production config with all options

Example 1: SQL Server to PostgreSQL (Password Authentication)

Basic migration from SQL Server to PostgreSQL using username/password:

# config-mssql-to-pg.yaml
source:
  type: mssql
  host: sqlserver.example.com
  port: 1433
  database: SourceDatabase
  user: sa
  password: ${MSSQL_PASSWORD}        # Set via: export MSSQL_PASSWORD="your-password"
  schema: dbo
  encrypt: "true"                    # Enable encryption (recommended)
  trust_server_cert: false           # Validate server certificate

target:
  type: postgres
  host: postgres.example.com
  port: 5432
  database: target_db
  user: postgres
  password: ${PG_PASSWORD}           # Set via: export PG_PASSWORD="your-password"
  schema: public
  ssl_mode: require                  # Enable SSL (recommended)

migration:
  workers: 8                         # Parallel workers
  chunk_size: 200000                 # Rows per chunk
  create_indexes: true               # Recreate indexes
  create_foreign_keys: true          # Recreate foreign keys
  target_mode: drop_recreate         # Drop and recreate tables
  fail_on_schema_drift: false        # Report source schema drift and continue

Example 2: PostgreSQL to SQL Server (Password Authentication)

Reverse migration from PostgreSQL to SQL Server:

# config-pg-to-mssql.yaml
source:
  type: postgres
  host: postgres.example.com
  port: 5432
  database: source_db
  user: postgres
  password: ${PG_PASSWORD}
  schema: public
  ssl_mode: require

target:
  type: mssql
  host: sqlserver.example.com
  port: 1433
  database: TargetDatabase
  user: sa
  password: ${MSSQL_PASSWORD}
  schema: dbo
  encrypt: "true"
  trust_server_cert: false

migration:
  workers: 8
  chunk_size: 200000
  write_ahead_writers: 8             # Use 8 writers for PG→MSSQL (faster)
  parallel_readers: 1                # Single reader per job
  create_indexes: true
  create_foreign_keys: true

Example 3: Minimal Configuration (Local Development)

Simplest config for local Docker databases:

# config-local.yaml
source:
  host: localhost
  port: 1433
  database: MyDatabase
  user: sa
  password: ${MSSQL_PASSWORD}
  encrypt: "false"                   # Disable encryption for local dev
  trust_server_cert: true            # Trust self-signed certs

target:
  host: localhost
  port: 5432
  database: mydb
  user: postgres
  password: ${PG_PASSWORD}
  ssl_mode: disable                  # Disable SSL for local dev

Example 4: Production Configuration with All Options

Full production configuration with Slack notifications and validation:

# config-production.yaml
source:
  type: mssql
  host: sqlserver-prod.example.com
  port: 1433
  database: ProductionDB
  user: migrate_user
  password: ${MSSQL_PASSWORD}
  schema: dbo
  encrypt: "true"
  trust_server_cert: false

target:
  type: postgres
  host: postgres-prod.example.com
  port: 5432
  database: production_db
  user: migrate_user
  password: ${PG_PASSWORD}
  schema: public
  ssl_mode: verify-full              # Full certificate verification

migration:
  # Connection pools
  max_source_connections: 20
  max_target_connections: 40

  # Parallelism
  workers: 16
  chunk_size: 250000
  max_partitions: 16
  large_table_threshold: 10000000

  # Table filtering
  exclude_tables:
    - temp_*
    - staging_*
    - __*
    - audit_log

  # Schema objects
  create_indexes: true
  create_foreign_keys: true
  create_check_constraints: true

  # Consistency
  strict_consistency: true           # Pin each table to one stable source transaction

  # Validation
  sample_validation: true            # Verify random samples
  sample_size: 500                   # Check 500 rows per table

  # State persistence
  data_dir: /var/lib/dmt

slack:
  enabled: true
  webhook_url: ${SLACK_WEBHOOK_URL}
  channel: "#data-migrations"
  username: dmt

Usage

Commands

# Run a new migration
./dmt -c config.yaml run

# Dry-run (preview plan without executing)
./dmt -c config.yaml run --dry-run

# Resume an interrupted migration (continues from last checkpoint)
./dmt -c config.yaml resume

# Check status of current/last run
./dmt -c config.yaml status

# Validate row counts between source and target
./dmt -c config.yaml validate

# View migration history
./dmt -c config.yaml history

# View details for a specific run
./dmt -c config.yaml history --run <run-id>

# Test database connections
./dmt -c config.yaml health-check

# Analyze source database and get AI configuration suggestions
./dmt -c config.yaml analyze

# Create a new config file interactively
./dmt init

# Create a secrets file template
./dmt init-secrets

Headless Mode (Airflow/Kubernetes)

For headless environments, use --state-file to store state in a YAML file instead of SQLite:

# Run with state file
./dmt -c config.yaml --state-file state.yaml run

# Resume with state file
./dmt -c config.yaml --state-file state.yaml resume

# All commands support --state-file
./dmt -c config.yaml --state-file state.yaml status
./dmt -c config.yaml --state-file state.yaml history

Example Output

Starting migration run: a1b2c3d4
Connection pools: MSSQL=12, PostgreSQL=12
Extracting schema...
Found 11 tables
Pagination: 9 keyset, 1 ROW_NUMBER, 1 no PK
Creating target tables (drop and recreate)...
Transferring data...
Transferring 100% |███████████| (106534570/106534570, 697K rows/s)
Transferred 106534570 rows in 2m33s (697008 rows/sec)

Transfer Profile (per table):
------------------------------
Votes                     query=2.8s (3%), scan=50.6s (55%), write=38.9s (42%), rows=52928720
Comments                  query=1.6s (0%), scan=87.1s (20%), write=346.5s (80%), rows=24534730
Posts                     query=3.2s (0%), scan=2183.3s (73%), write=823.5s (27%), rows=17142169
...

Validation Results:
-------------------
Badges                         OK 8042005 rows
Comments                       OK 24534730 rows
Posts                          OK 17142169 rows
Users                          OK 2465713 rows
Votes                          OK 52928720 rows

How It Works

Extract schema - Reads table structure, PKs, indexes, FKs, and check constraints from source
Create tables - Generates target DDL with proper type mapping and identity columns
Transfer data - Uses optimal pagination strategy per table:
- Keyset pagination for single-column integer PKs (fastest)
- ROW_NUMBER pagination for composite/varchar PKs
Save progress - Checkpoints every 20 chunks by default to SQLite for resume capability
Finalize - Resets identity sequences, creates primary keys
Create indexes - Non-PK indexes (if enabled)
Create foreign keys - FK constraints (if enabled)
Create check constraints - CHECK constraints (if enabled)
Validate - Compares row counts and optionally samples random rows

Resume Capability

The tool saves progress to enable efficient resume after failures. By default, state is stored in SQLite (~/.dmt/migrate.db). For headless environments, use --state-file for a portable YAML state file.

Table-level resume

Completed tables are skipped entirely on resume
Verified by comparing row counts between source and target
Partial outcomes remain resumable by default; outcome and resumability are exposed separately in status/history JSON
migration.allow_partial: true accepts a partial as non-resumable; use resume --abandon --abandon-reason <text> to explicitly stop retrying any other recoverable run without deleting its checkpoint history

Chunk-level resume

Progress saved every 20 chunks by default during transfer
On resume, continues from the exact last successful chunk
Partial data from interrupted chunks is cleaned up automatically

Stale progress detection

If target table has fewer rows than saved progress, the tool detects data loss
Automatically clears stale progress and restarts the table transfer
Prevents resuming with incorrect last_pk values

# Resume shows what's being skipped/continued
./dmt -c config.yaml resume

# Or with state file for Airflow/Kubernetes
./dmt -c config.yaml --state-file state.yaml resume

# Output:
# Resuming run: a1b2c3d4 (started 2025-01-15T10:30:00Z)
# Skipping 5 already-complete tables: [Users, Posts, Comments, Badges, Votes]
# Resuming transfer of 2 tables
# Resuming Orders from chunk (lastPK=1234567, rows=5000000)

Pagination Strategies

The tool automatically selects the best pagination strategy per table:

PK Type	Strategy	Performance
Single integer (int, bigint)	Keyset (`WHERE pk > @last`)	Fastest
Composite PK	ROW_NUMBER	Good
VARCHAR PK	ROW_NUMBER	Good
No PK	Rejected	-

Tables without primary keys are rejected to ensure data correctness.

Type Mapping

SQL Server → PostgreSQL

SQL Server	PostgreSQL
int	integer
bigint	bigint
smallint	smallint
tinyint	smallint
bit	boolean
decimal/numeric	numeric
float	double precision
real	real
money	numeric(19,4)
char/nchar	char
varchar/nvarchar	varchar
text/ntext	text
date	date
time	time
datetime/datetime2	timestamp
datetimeoffset	timestamptz
uniqueidentifier	uuid
varbinary/image	bytea
xml	xml

Identity columns are mapped to GENERATED BY DEFAULT AS IDENTITY with proper sequence reset.

PostgreSQL → SQL Server

PostgreSQL	SQL Server
integer	int
bigint	bigint
smallint	smallint
boolean	bit
numeric/decimal	decimal
double precision	float
real	real
char	char
varchar/character varying	nvarchar
text	nvarchar(max)
date	date
time	time
timestamp	datetime2
timestamptz	datetimeoffset
uuid	uniqueidentifier
bytea	varbinary(max)
json/jsonb	nvarchar(max)

Serial/identity columns are mapped to IDENTITY(1,1) with proper seed reset.

Unknown Types

For types not in the built-in mappings (custom domains, user-defined types, etc.), enable AI-assisted type mapping to automatically infer the best target type.

Benchmarks

Small Dataset (WideWorldImporters, 701K rows)

Hardware: WSL2 on Windows, 32GB RAM, 16 cores
Databases: PostgreSQL 15 and SQL Server 2022 (Docker)

Direction	Transfer	Overall
PG → MSSQL	688K rows/sec	645K rows/sec
PG → PG	605K rows/sec	563K rows/sec
MSSQL → PG	302K rows/sec	248K rows/sec
MSSQL → MSSQL	280K rows/sec	222K rows/sec

Large Dataset (Stack Overflow 2013, 106.5M rows)

Hardware: macOS, Apple Silicon, 36GB RAM
Databases: SQL Server 2022 and PostgreSQL 17 (Docker, 16GB limit)

Configuration	Transfer	Overall	Throughput
Smartconfig + runtime tuning	2m 33s	3m 08s	697K rows/sec
Runtime tuning only	5m 14s	5m 50s	339K rows/sec

Smartconfig analyzes source schema, system resources, driver profiles, and completed-run history to set initial parameters, delivering a 2x speedup over runtime-only tuning.

Performance varies based on network latency, table width, data types, and available CPU/memory.

Development

Running Tests

The project includes comprehensive unit tests for CLI parsing, orchestrator logic, and profile encryption.

# Run all tests
make test

# Run tests with short flag (faster, skips slow tests)
make test-short

# Generate coverage report
make test-coverage
# Open coverage.html in browser to view results

Pre-Commit Hooks

To ensure tests pass before committing:

# Set up git hooks (one-time setup)
make setup-hooks

# This configures git to use .githooks/pre-commit which:
# - Checks code formatting
# - Runs all tests
# - Blocks commit if any check fails

Building

# Build binary
make build

# Build for all platforms
make build-all

# Run all checks (format + tests)
make check

Test Databases (Docker)

For integration testing:

# Start local SQL Server and PostgreSQL
make test-dbs-up

# Stop and remove test databases
make test-dbs-down

Known Limitations

Tables Without Primary Keys

All tables must have primary keys for migration. Tables without primary keys are automatically skipped with a warning. This is required for:

Chunked pagination during data transfer
Change detection in upsert mode

License

MIT

Name		Name	Last commit message	Last commit date
.claude/agents		.claude/agents
.githooks		.githooks
.github/workflows		.github/workflows
cmd/dmt		cmd/dmt
docs		docs
examples		examples
internal		internal
scripts		scripts
tools/genicons		tools/genicons
.gitignore		.gitignore
.golangci.yml		.golangci.yml
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
Makefile		Makefile
README.md		README.md
RECREATE_DMT.md		RECREATE_DMT.md
RELEASE.md		RELEASE.md
SMT_CANONICAL_ADOPTION_PLAN.md		SMT_CANONICAL_ADOPTION_PLAN.md
VERSIONING.md		VERSIONING.md
config.yaml.example		config.yaml.example
go.mod		go.mod
go.sum		go.sum

Latest commit

History

Folders and files

Repository files navigation

dmt

Quick Start

Interactive Mode

Features

Web UI

Security

Incremental Sync

Recommended Workflow

Performance Impact

Configuration

How It Works

Important Notes

AI-Driven Real-Time Parameter Adjustment

Quick Start

How It Works

Performance

Configuration

Cost

Troubleshooting

Encrypted Profiles (SQLite)

Optional AI Enhancements

AI Quick Start

Supported Providers

Full Configuration

AI Type Mapping

Smart Config Detection

Error Diagnosis

Cost Considerations

State File Backend (Airflow/Kubernetes)

Airflow Integration

Airflow CLI Flags

BashOperator Example

KubernetesPodOperator Example

JSON Output Format

Config Hash Validation

Airflow Sensor Pattern

Performance

Supported Databases

Same-Engine Migrations

PostGIS Spatial Data Support

Features

Database Support

Transfer Engine

AI Integration

Checkpoint & Resume

Incremental Sync

Interactive Mode (TUI)

CLI & Automation

Security

Schema Management

Observability

Deployment

Installation

Download pre-built binaries

Build from source

Go install

Quick Start

SQL Server to PostgreSQL

PostgreSQL to SQL Server

Configuration Reference

Source Database Settings

Target Database Settings

Migration Settings

AI Settings

Slack Notification Settings

Kerberos Authentication

Example Configurations

Example 1: SQL Server to PostgreSQL (Password Authentication)

Example 2: PostgreSQL to SQL Server (Password Authentication)

Example 3: Minimal Configuration (Local Development)

Example 4: Production Configuration with All Options

Usage

Commands

Headless Mode (Airflow/Kubernetes)

Example Output

How It Works