🏢 Enterprise Databricks Asset Bundles

A production-ready, enterprise-scale Databricks Asset Bundles framework featuring intelligent CI/CD workflows, advanced data quality validation, and enterprise security patterns.

🚀 Overview

This project delivers a complete enterprise-grade framework for managing Databricks workloads using Asset Bundles with cutting-edge DevOps practices:

🎯 Core Features

• 🧠 Intelligent CI/CD Workflows - Path-based execution that runs only relevant jobs
• 🔐 Enterprise Security - OAuth Service Principal authentication, no more personal tokens
• 📊 Advanced Data Quality - Statistical anomaly detection, schema evolution validation
• 🔄 Multi-Environment Support - Seamless dev/staging/production deployments
• ⚡ Performance Optimized - Secret caching, retry mechanisms, serverless computing
• 🛡️ Production Ready - Comprehensive error handling, backup strategies, monitoring

🌟 What Makes This Different

• Intelligent Workflows: Automatically detects Python vs YAML changes and runs only necessary jobs (~13s for no relevant changes)
• Enterprise Authentication: Uses Service Principals for all environments, following security best practices
• Advanced Data Platform: Includes statistical validation, anomaly detection, and schema evolution checks
• Developer Experience: Pre-commit hooks, comprehensive testing, and detailed troubleshooting guides

📁 Project Structure

databricks-enterprise-bundles/
├── .databricks/                # Bundle variable overrides for each environment
│   └── bundle/
│       ├── dev/               # Development environment variables
│       ├── staging/           # Staging environment variables
│       └── prod/              # Production environment variables
├── .github/                   # Intelligent CI/CD workflows and actions
│   ├── actions/               # Reusable composite actions
│   └── workflows/             # Modular workflow definitions
│       ├── ci-dev.yml         # Development CI with intelligent detection
│       ├── pr-validation.yml  # PR validation with multi-env checks
│       ├── deploy-staging.yml # Staging deployment with prod validation
│       └── deploy-production.yml # Production deployment on releases
├── src/                       # Enterprise-grade source code
│   ├── libraries/            # Enhanced Python libraries with caching, retry logic
│   │   ├── data_quality.py   # Statistical validation & anomaly detection
│   │   └── secret_manager.py # Intelligent secret caching & management
│   └── notebooks/            # Production-ready notebooks with error handling
├── tests/                     # Comprehensive test suites
│   ├── unit/                 # Unit tests with mocked dependencies
│   └── integration/          # Integration tests for full pipeline
├── presentation/              # Executive presentation materials
├── monitoring/                # Enterprise monitoring and observability
├── security/                  # Security policies and configurations
└── Documentation Files        # Comprehensive guides and troubleshooting
    ├── TROUBLESHOOTING.md     # Complete troubleshooting guide
    ├── DATABRICKS_CLI_SETUP.md
    ├── GITHUB_SETUP.md
    ├── SLACK_INTEGRATION.md
    └── CONTRIBUTING.md

🏗️ Architecture

Environment Strategy

The framework follows a three-tier environment strategy:

• Development: Individual developer workspace for testing and experimentation
• Staging: Pre-production environment for integration testing
• Production: Live production environment with enhanced monitoring and reliability

Data Architecture

Implements the medallion architecture pattern:

• Bronze Layer: Raw data ingestion with minimal transformation
• Silver Layer: Cleaned and standardized data with business rules applied
• Gold Layer: Aggregated and enriched data ready for consumption

⚡ Quick Start

🔧 Prerequisites

• Python 3.9+ with pip
• Modern Databricks CLI (installed via setup script)
• Git for version control
• GitHub CLI (optional, for PR management)
• Databricks workspace with appropriate permissions

🚀 Lightning Setup (5 Minutes)

1. Clone and setup:

   git clone https://github.com/merca/databricks-enterprise-bundles.git
   cd databricks-enterprise-bundles
   pip install -r requirements.txt

2. Install Databricks CLI (modern version):

   # The setup script installs the correct CLI version
   curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

3. Configure authentication (OAuth Service Principal recommended):

   # Set environment variables (see ENVIRONMENT_VARIABLES.md)
   export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
   export DATABRICKS_CLIENT_ID="your-service-principal-id"
   export DATABRICKS_CLIENT_SECRET="your-service-principal-secret"

4. Test the setup:

   # Validate bundle configuration
   databricks bundle validate --target dev
   
   # Run tests
   pytest tests/unit/

5. Deploy to development:

   databricks bundle deploy --target dev
   databricks bundle run data_processing_job --target dev

📖 Need help? Check TROUBLESHOOTING.md for common issues and solutions.

Deployment

Development Deployment

cd bundles/dev
databricks bundle deploy

Staging/Production Deployment

Use the automated CI/CD pipeline by pushing to the appropriate branches:

• Push to next branch for dev deployment
• Push to current branch for staging and production deployment

📋 Configuration

Environment Configuration

Each environment has its own configuration file in configs/environments/:

# configs/environments/dev.yml
variables:
  service_principal_id: "dev-databricks-sp@company.com"
  notification_email: "dev-team@company.com"
  workspace_url: "https://dev-workspace.cloud.databricks.com"
  catalog_name: "dev_catalog"
  schema_name: "dev_schema"
  cluster_node_type: "i3.xlarge"
  min_workers: 1
  max_workers: 4
  # ... additional configuration

Bundle Configuration

Each environment has a databricks.yml file defining:

• Resource definitions (jobs, pipelines, clusters)
• Environment-specific variables
• Deployment targets and permissions

🧠 Intelligent CI/CD System

Our cutting-edge CI/CD system features intelligent path-based execution that dramatically improves developer productivity:

🎯 Smart Workflow Execution

• Python Changes: Triggers lint, test, and Databricks validation jobs
• YAML/Config Changes: Runs only Databricks validation jobs
• No Relevant Changes: Ultra-fast ~13 second feedback
• Mixed Changes: Executes all necessary jobs intelligently

🔐 Enterprise Security First

• OAuth Service Principals: No more personal access tokens
• Multi-Environment Secrets: Separate credentials for dev/staging/prod
• Secure Secret Caching: 5-minute TTL with intelligent invalidation
• Retry Mechanisms: Exponential backoff for transient failures

🚀 Deployment Pipeline

1. PR Validation - Comprehensive checks on every pull request
2. Development CI - Automatic deployment on next branch
3. Staging Deployment - Full environment testing on current branch
4. Production Release - Triggered by release tags with rollback capability
5. Monitoring & Alerts - Slack notifications and health monitoring

Pipeline Stages

graph TD
    A[Code Push] --> B[Lint & Test]
    B --> C[Validate Bundles]
    C --> D{Branch?}
    D -->|next| E[Deploy to Dev]
    D -->|current| F[Deploy to Staging]
    F --> G[Integration Tests]
    G --> H[Deploy to Production]
    H --> I[Smoke Tests]
    I --> J[Monitor Health]

Loading

📊 Data Quality Framework

The framework includes a comprehensive data quality validation system:

Features

• Completeness checks: Null value validation, empty string detection
• Validity checks: Format validation, range checks, allowed values
• Consistency checks: Uniqueness validation, referential integrity
• Custom business rules: Configurable validation logic

Usage Example

from src.libraries.data_quality import DataQualityValidator

# Define validation rules
rules = {
    "completeness": {
        "customer_id": {"null_threshold": 0.0},
        "transaction_amount": {"null_threshold": 0.05}
    },
    "validity": {
        "transaction_amount": {"min_value": 0.01, "max_value": 100000.00},
        "transaction_type": {"allowed_values": ["PURCHASE", "REFUND", "TRANSFER"]}
    },
    "consistency": {
        "customer_id": {"unique": True}
    }
}

# Validate DataFrame
validator = DataQualityValidator(rules)
results = validator.validate(df)

# Generate quality report
report = validator.generate_quality_report(df, "/path/to/report")

🛠️ Development Workflow

Creating New Jobs

1. Copy job template:

   cp templates/jobs/job_template.yml bundles/dev/resources/jobs/my_new_job.yml

2. Customize job configuration

3. Add notebook or library code to src/

4. Test locally in development environment

5. Commit and push for automated deployment

Adding New Notebooks

1. Use notebook templates from src/notebooks/
2. Follow established patterns for:
   • Configuration management
   • Error handling and logging
   • Data quality validation
   • Performance optimization

Environment Promotion

Code promotion follows this pattern:

1. Development → Feature branch → next branch
2. Integration testing in staging
3. Production deployment via current branch

📈 Monitoring and Observability

Health Monitoring

The deployment includes automatic health monitoring:

• Job execution status
• Pipeline health checks
• Workspace accessibility
• Unity Catalog connectivity
• Performance metrics

Alerting

Configure alerts for:

• Failed job executions
• Pipeline failures
• Data quality issues
• Performance degradation

Logging

Comprehensive logging across all components:

• Structured logging format
• Centralized log aggregation
• Error tracking and alerting

🔐 Security Best Practices

Access Control

• Service principals for production workloads
• Role-based access control (RBAC)
• Principle of least privilege
• Secret management via Azure Key Vault/AWS Secrets Manager

Data Security

• Unity Catalog integration
• Column-level security
• Data lineage tracking
• Audit logging

📚 Related Repositories

Template Repository

For reusable bundle templates and starter patterns, see the companion template repository:

• Bundle Templates: Standardized project structures and configurations
• Best Practice Patterns: Pre-configured jobs, notebooks, and pipelines
• Quick Start Templates: Ready-to-use templates for common use cases

This repository focuses on the production enterprise implementation with advanced features.

Best Practices

1. Use parameterized notebooks for flexibility
2. Implement comprehensive error handling
3. Follow naming conventions consistently
4. Document all business logic
5. Include data quality checks at every layer

🚀 Scaling Considerations

Performance Optimization

• Cluster autoscaling configuration
• Photon acceleration for pipelines
• Delta table optimizations (Z-ordering, auto-compaction)
• Caching strategies

Cost Management

• Automatic cluster termination
• Spot instance utilization
• Resource right-sizing
• Budget alerts and monitoring

Multi-Region Support

• Cross-region replication strategies
• Disaster recovery planning
• Data residency compliance

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Follow coding standards and templates
4. Add tests for new functionality
5. Update documentation
6. Submit pull request

Code Standards

• Follow PEP 8 for Python code
• Use type hints
• Write comprehensive docstrings
• Include unit tests
• Maintain test coverage above 80%

📞 Support and Troubleshooting

Common Issues

• Bundle validation failures: Check YAML syntax and required fields
• Permission errors: Verify service principal permissions
• Resource conflicts: Ensure unique resource names across environments
• Network connectivity: Verify workspace access and firewall rules

Getting Help

• Check the docs/ directory for detailed documentation
• Review GitHub Issues for known problems
• Contact the Data Engineering team for support

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🏷️ Version History

• v1.0.0: Initial enterprise framework release
• v1.1.0: Added advanced monitoring and alerting
• v1.2.0: Enhanced security and compliance features

---

Maintained by: Data Engineering Team Last Updated: 2025-10-10