ETL Pipeline using ADF, Delta Lake, Databricks, Spark, Data Governance (UnityCatalog).
End-to-end Azure data engineering pipeline using Spotify dummy data with Medallion Architecture, Databricks, Delta Live Tables, and incremental loading.
An end-to-end Netflix data engineering pipeline built on Microsoft Azure. This project ingests raw Netflix data, applies PySpark transformations , enforces data quality with Delta Live Tables, and orchestrates workflows via Azure Data Factory and Databricks.
Development and implementation of an ETL pipeline for processing oil well data using Databricks, Delta Live Tables, Spark Structured Streaming and PySpark. The project focused on automating the ingestion, transformation, and validation of large volumes of well log data.
Add a description, image, and links to the deltalivetables topic page so that developers can more easily learn about it.
To associate your repository with the deltalivetables topic, visit your repo's landing page and select "manage topics."