{{ TENANT_NAME }} Data Pipeline

This repository contains the data pipeline configuration for your Scy-Data workspace.

Structure

├── databricks.yml          # DAB bundle configuration
├── src/
│   ├── notebooks/          # Databricks notebooks
│   ├── jobs/               # Job definitions
│   └── pipelines/          # DLT pipeline definitions
├── tests/                  # Unit tests
└── config/                 # Environment-specific configs
    ├── dev.yml
    └── prod.yml

Getting Started

Prerequisites

• Databricks CLI installed
• Access to your Databricks workspace
• Python 3.10+

Setup

1. Configure Databricks CLI:

   databricks configure

2. Validate the bundle:

   databricks bundle validate

3. Deploy to development:

   databricks bundle deploy -t dev

Deployment

Development

databricks bundle deploy -t dev
databricks bundle run -t dev <job_name>

Production

databricks bundle deploy -t prod
databricks bundle run -t prod <job_name>

Resources

• Databricks Asset Bundles
• DLT Pipelines
• Scy-Data Documentation