Featherflow

A lightweight dbt-like CLI tool built in Rust for SQL templating, compilation, and execution against DuckDB.

Architecture

For a deep dive into what Feather-Flow is, why it works the way it does, and how the schema validation framework and static analysis engine fit together, see HOW_FEATHERFLOW_WORKS.md.

Features

SQL Templating: Jinja-style templating with config() and var() functions
Custom Macros: Reusable SQL macros loaded from macro_paths directories
AST-based Dependencies: Automatically extracts dependencies from SQL using sqlparser-rs - no need for ref() or source() functions
Dependency-aware Execution: Builds a DAG and executes models in topological order
Schema Testing: Built-in support for 10 test types (unique, not_null, positive, non_negative, accepted_values, min_value, max_value, regex, relationship, custom) with sample failing rows
Source Definitions: Document and test external data sources
Documentation Generation: Generate markdown, JSON, or HTML docs from schema files, with an interactive docs server
DuckDB Backend: In-memory or file-based DuckDB database execution
Static Analysis: DataFusion-based SQL analysis passes for type checking and best practices
SQL Formatting: Built-in SQL formatter with Jinja support
Column Lineage: Trace column-level lineage across models

Installation

Quick install (macOS and Linux)

curl -fsSL https://raw.githubusercontent.com/datastx/Feather-Flow/main/install.sh | bash

This detects your OS and architecture, downloads the correct binary from the latest GitHub Release, verifies the SHA256 checksum, and installs to ~/.local/bin.

Options:

# Install a specific version
curl -fsSL https://raw.githubusercontent.com/datastx/Feather-Flow/main/install.sh | FF_VERSION=0.1.0 bash

# Install to a custom directory
curl -fsSL https://raw.githubusercontent.com/datastx/Feather-Flow/main/install.sh | INSTALL_DIR=/usr/local/bin bash

Manual download

Pre-built binaries are available on the Releases page:

Platform	Artifact
Linux x86_64	`ff-x86_64-linux-gnu`
macOS x86_64	`ff-x86_64-apple-darwin`
macOS ARM (Apple Silicon)	`ff-aarch64-apple-darwin`

# Example: download latest for Linux x86_64
curl -fsSL https://github.com/datastx/Feather-Flow/releases/latest/download/ff-x86_64-linux-gnu -o ff
chmod +x ff
sudo mv ff /usr/local/bin/

Docker

docker pull ghcr.io/datastx/feather-flow:latest

# Run any ff command
docker run --rm -v "$(pwd)":/workspace -w /workspace ghcr.io/datastx/feather-flow dt compile
docker run --rm -v "$(pwd)":/workspace -w /workspace ghcr.io/datastx/feather-flow run
docker run --rm -v "$(pwd)":/workspace -w /workspace ghcr.io/datastx/feather-flow run --mode test

From source

git clone https://github.com/datastx/Feather-Flow.git
cd Feather-Flow
make build-release
# Binary is at target/release/ff

# Or install directly
cargo install --path crates/ff-cli

Quickstart

Create a project directory with a featherflow.yml configuration:

name: my_project
version: "1.0.0"

database:
  type: duckdb
  path: ":memory:"

node_paths:
  - nodes
macro_paths:
  - macros

vars:
  env: dev
  start_date: "2024-01-01"

Create SQL models in the nodes/ directory. Each node lives in its own directory with a .sql file and a .yml schema file:

-- nodes/stg_orders/stg_orders.sql
{{ config(materialized='view', schema='staging') }}

SELECT
    id AS order_id,
    user_id AS customer_id,
    created_at AS order_date,
    amount,
    status
FROM raw_orders
WHERE created_at >= '{{ var("start_date") }}'

Create a schema file alongside your model (1:1 convention):

# nodes/stg_orders/stg_orders.yml
version: 1
kind: sql
description: "Staged orders from raw source"
owner: data-team
tags:
  - staging
  - orders

columns:
  - name: order_id
    tests:
      - unique
      - not_null
  - name: customer_id
    tests:
      - not_null
  - name: order_date
  - name: amount
  - name: status

Run Featherflow commands:

# Load seed data
ff dt deploy seeds

# Compile models (renders Jinja, extracts dependencies, builds manifest)
ff dt compile

# List models and sources
ff dt ls

# Execute models in dependency order
ff run

# Run schema tests
ff run --mode test

# Validate project without execution
ff dt compile --parse-only

# Generate documentation
ff dt docs

Commands

Featherflow commands are organized into runtime commands (top-level) and developer tooling (ff dt):

Runtime Commands

`ff run`

Executes models, runs tests, or both depending on the --mode flag.

ff run [--mode <MODE>] [--nodes <NODES>] [--full-refresh] [--fail-fast]

Run modes:

build (default) - Execute models then run their tests in DAG order
models - Execute models only
test - Run tests only against existing tables

Node selectors:

model_name - Run a specific model
+model_name - Run model and all ancestors
model_name+ - Run model and all descendants
+model_name+ - Run model, ancestors, and descendants
tag:X - Run models with a specific tag
path:X - Run models in a specific path

`ff run-macro`

Execute a standalone SQL macro.

ff run-macro <MACRO_NAME> [--args <JSON>]

Developer Tooling (`ff dt`)

`ff dt compile`

Compiles SQL models by rendering Jinja templates, extracting dependencies, and generating a manifest.

ff dt compile [--nodes <NODES>] [--parse-only] [--strict] [--output <FORMAT>]

Use --parse-only to validate without writing output files. Use --strict to treat warnings as errors.

`ff dt ls`

Lists models and sources with their dependencies and materialization settings.

ff dt ls [--output <FORMAT>] [--nodes <NODES>] [--resource-type <TYPE>]

Output formats: table (default), json, tree, path

Resource types: model, source, seed, test, function

`ff dt deploy`

Deploy seeds or functions to the database.

# Load CSV seed files
ff dt deploy seeds [--seeds <NAMES>] [--full-refresh]

# Deploy user-defined functions
ff dt deploy functions deploy [--functions <NAMES>]

# List, validate, show, or drop functions
ff dt deploy functions list|validate|show|drop

`ff dt docs`

Generates documentation from schema files.

ff dt docs [--output <PATH>] [--format <FORMAT>] [--nodes <NODES>]

# Launch interactive documentation server
ff dt docs serve [--port <PORT>] [--no-browser] [--static-export <DIR>]

`ff dt lineage`

Trace column-level lineage across models.

ff dt lineage --node <NODE> --column <COLUMN> [--direction <DIR>] [--output <FORMAT>]

`ff dt analyze`

Run static analysis passes on SQL models.

ff dt analyze [--nodes <NODES>] [--output <FORMAT>] [--severity <LEVEL>]

# Query the metadata database
ff dt analyze query <SQL> [--json]

# Export metadata as JSON
ff dt analyze export [--output <FILE>]

# List metadata tables
ff dt analyze tables

`ff dt fmt`

Format SQL source files.

ff dt fmt [<PATHS>] [--nodes <NODES>] [--check] [--diff]

`ff dt init`

Initialize a new Featherflow project.

ff dt init [--name <NAME>] [--database_path <PATH>]

`ff dt clean`

Remove generated artifacts.

ff dt clean [--dry-run]

Global Options

--project-dir, -p <DIR>: Project directory (default: current directory)
--config, -c <FILE>: Config file path
--target, -t <TARGET>: Override target (database connection)
--verbose, -v: Enable verbose output

Configuration

featherflow.yml

name: project_name          # Project name
version: "1.0.0"            # Project version

database:
  type: duckdb              # Database type (duckdb, snowflake)
  path: ":memory:"          # Database path (":memory:" for in-memory)
  name: main                # Logical database name

dialect: duckdb             # SQL dialect (duckdb, snowflake)
materialization: view       # Default materialization (view, table, incremental, ephemeral)
schema: main                # Default schema

node_paths:                 # Directories containing all node types (models, seeds, functions, sources)
  - nodes
macro_paths:                # Directories containing macro files
  - macros
target_path: target         # Output directory for compiled files

on_run_start:               # SQL executed before each run
  - "CREATE TABLE IF NOT EXISTS audit (id INTEGER)"
on_run_end:                 # SQL executed after each run
  - "INSERT INTO audit (id) VALUES (1)"

vars:                       # Variables accessible via var()
  env: dev
  start_date: "2024-01-01"

Model Configuration

Use the config() function in your SQL models:

{{ config(materialized='table', schema='staging') }}

SELECT * FROM raw_data

Supported config options:

materialized: 'view', 'table', 'incremental', or 'ephemeral'
schema: Target schema name

Variables

Access variables with var():

SELECT * FROM {{ var('schema') }}.users
WHERE env = '{{ var('env', 'prod') }}'

Custom Macros

Create reusable macros in macros/:

-- macros/date_utils.sql
{% macro date_trunc(date_col, granularity) %}
DATE_TRUNC('{{ granularity }}', {{ date_col }})
{% endmacro %}

Use in models:

{% from "date_utils.sql" import date_trunc %}

SELECT
    {{ date_trunc('order_date', 'month') }} AS order_month,
    SUM(amount) AS total
FROM orders
GROUP BY 1

Project Structure

my_project/
├── featherflow.yml
├── nodes/
│   ├── stg_orders/
│   │   ├── stg_orders.sql      # SQL model
│   │   └── stg_orders.yml      # Schema + tests
│   ├── stg_customers/
│   │   ├── stg_customers.sql
│   │   └── stg_customers.yml
│   ├── fct_orders/
│   │   ├── fct_orders.sql
│   │   └── fct_orders.yml
│   └── raw_orders/
│       ├── raw_orders.csv      # Seed data (kind: seed)
│       └── raw_orders.yml
├── macros/
│   └── date_utils.sql
└── target/
    ├── compiled/
    ├── manifest.json
    └── run_results.json

Development

Building

# Build all crates
make build

# Run tests
make test

# Run end-to-end tests
make ci-e2e

# Run CI checks (format, clippy, test, doc)
make ci

Crate Structure

ff-cli: CLI binary and commands
ff-core: Core library (config, project, DAG, node types)
ff-sql: SQL parsing and dependency extraction
ff-jinja: Jinja templating layer with macro support
ff-db: Database abstraction and DuckDB backend
ff-test: Test generation and execution
ff-analysis: DataFusion-based static SQL analysis
ff-meta: Metadata database (DuckDB) for schema, population, and rules

License

MIT

Name		Name	Last commit message	Last commit date
crates		crates
docs		docs
scripts		scripts
testdata		testdata
vscode-featherflow		vscode-featherflow
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
Dockerfile.release		Dockerfile.release
Dockerfile.release.dockerignore		Dockerfile.release.dockerignore
HOW_FEATHERFLOW_WORKS.md		HOW_FEATHERFLOW_WORKS.md
Makefile		Makefile
README.md		README.md
install.sh		install.sh
rust-toolchain.toml		rust-toolchain.toml

Latest commit

History

Folders and files

Repository files navigation

Featherflow

Architecture

Features

Installation

Quick install (macOS and Linux)

Manual download

Docker

From source

Quickstart

Commands

Runtime Commands

ff run

ff run-macro

Developer Tooling (ff dt)

ff dt compile

ff dt ls

ff dt deploy

ff dt docs

ff dt lineage

ff dt analyze

ff dt fmt

ff dt init

ff dt clean