Azure AI for Data Engineering

Overview

Practical AI patterns for Data Engineers using Azure OpenAI, LangChain, and Azure AI Foundry. These are production patterns built from real work — not toy demos. Covers RAG for data catalogues, Text-to-SQL for self-service analytics, LLM-powered data quality explanations, and AI-assisted pipeline monitoring.

---

Repository Structure

azure-ai-data-engineering/
│
├── rag-pipeline/
│   ├── data_catalog_rag.py            # RAG over Unity Catalog metadata
│   └── document_ingestion.py          # Ingest docs into Azure AI Search
│
├── text-to-sql/
│   ├── text_to_sql_agent.py           # NL → SQL with Azure OpenAI + schema context
│   └── schema_context_builder.py      # Build schema context from Delta tables
│
├── data-catalog-ai/
│   └── catalog_metadata_enricher.py   # Auto-generate table/column descriptions
│
└── azure-openai-utils/
    └── openai_client.py               # Shared Azure OpenAI client wrapper

---

Use Cases

Use Case	File	Description
RAG over data catalogue	rag-pipeline/data_catalog_rag.py	Ask questions about your Delta tables in natural language
Text-to-SQL	text-to-sql/text_to_sql_agent.py	"Show me top 10 products by revenue last quarter" → SQL
Metadata enrichment	data-catalog-ai/catalog_metadata_enricher.py	Auto-generate descriptions for Unity Catalog tables
Self-healing pipelines	Coming soon	LLM explains pipeline failures and suggests fixes

---

Tech Stack

Component	Technology
LLM	Azure OpenAI (GPT-4o)
Embeddings	Azure OpenAI (text-embedding-3-large)
Vector Store	Azure AI Search
Orchestration	LangChain / LangGraph
Data Platform	Databricks + Unity Catalog
AI Platform	Azure AI Foundry