I started out building dashboards. Somewhere around the fiftieth "can you just add a filter?", I noticed the pattern: the dashboard was almost never the problem. Two reports disagreed because two systems defined active customer differently. A supplier's spend looked halved because the same vendor existed twice under slightly different keys. A number couldn't be defended because nobody could say where it came from. So I kept moving one layer down.

Down into the pipelines first. SQL and PL/SQL, then Python and PySpark; ETL and ELT over Oracle, Teradata, Snowflake and Databricks; watermarks, incremental loads and the unglamorous discipline of making a job safe to re-run at 3am. A report is only ever as good as the load that fed it.

Then into the models. Dimensional first, then Data Vault — because a warehouse serving eight source systems needs somewhere for their disagreements to live rather than be averaged away. Then into architecture proper, and TOGAF, because the design that survives contact with an organisation is the one that fits a target state somebody actually signed.

Then into governance, which turned out to be the layer I'd been circling all along. A glossary term with a named owner. A data-quality rule with the teeth to stop a load. Lineage you can put in front of an auditor. This is the machinery that makes a number mean something — and it's why DAMA CDMP, Collibra and Chief Data Office operating models are how I spend my time now.

Today I work as a data engineering architect — designing and building the foundation itself: ingestion and pipeline design, the canonical and Data Vault models underneath, the lakehouse they land in, and the governance that makes any of it defensible. On top of that I still build the BI and analytics layer, because knowing exactly what a dashboard will ask of a model is what stops you designing one it cannot answer. Increasingly the consumer is an agent rather than a person, and an agent answering questions about enterprise data is exactly as trustworthy as the certified datasets, entitlements and lineage underneath it. That work has run through Chief Data Office, Credit Risk & Regulatory Reporting, Procurement & Supply Chain, Merchandising & Customer Analytics, Risk Analytics and Data & Analytics Centre of Excellence functions.

Fifteen years in, the through-line hasn't changed: make the number defensible, then make it easy. Everything in the repositories below is that belief in runnable form.

Primary — Data Engineering & Architecture. Python · PySpark · SQL/PL-SQL · Databricks · Snowflake · Microsoft Fabric · Starburst/Trino · ETL/ELT · Data Vault 2.0 · dimensional modelling · lakehouse · TOGAF · DAMA CDMP · Collibra · data quality · lineage · MDM

Secondary — BI & Analytics. 5× Tableau certified · Tableau Server/Cloud architecture · 500+ workbook estates · RLS and certified data sources · Power BI · Tableau Pulse · plus agentic AI on governed data (Agentforce · Data 360 · IAPP AIGP)

Open-source reference implementations of the patterns above, ordered the way I'd want them read — engineering and architecture first, BI last. Everything here was executed before it was published: the pipelines run, the tests pass, and the numbers in each README came from a real run rather than an estimate.

Three of these reproduce failure modes I have actually hit and fixed — a future-dated row poisoning a watermark so a job silently ingests nothing forever; SAP purchase-order numbers colliding across instances because they are not globally unique; a quality suite reporting 100% pass on broken data because NULL > 0 is NULL, not false. Reproducing a bug is worth more than describing one.

Data Engineering & Pipelines

Data Platforms, Lakehouse & Cloud

BI & Analytics

Governance, Salesforce & Delivery

Data platforms are only as good as the domain understanding behind them. Where I have delivered:

🏛️ "House of Data" — Governed Data Foundation + Agentic AI · Chief Data Office
Engineered the Chief Data Office's single-source data foundation — ingestion, harmonisation and modelling across Snowflake, Azure & Microsoft Fabric, federated through Starburst/Trino, governed in Collibra — then built agentic-AI experiences on Agentforce, Prompt Builder & Data 360 so leadership can ask natural-language questions on governance, quality and lineage, feeding senior risk dashboards and regulatory (FED) reporting.

📦 Data-as-a-Service at Scale · Data & Analytics Centre of Excellence
Owned the full pipeline — data marts, dimensional models, ETL with Tableau Prep · Alteryx · Oracle SQL · Python (incl. Scrapy web-scraping of alternative data) — deploying 150+ governed dashboards globally; embedded MDM & anomaly monitoring that helped identify $20M in supply-chain savings.

🏦 Credit-Risk Data & Analytics — Consumer & Corporate Banking · Credit Risk & Regulatory Reporting
Built governed data foundations on Snowflake with enterprise ETL/ELT pipelines (Python — pandas/NumPy, Alteryx) over Oracle & Big Data back ends, powering exposure, delinquency, vintage and portfolio-risk reporting for credit-card operations with audited source-to-target lineage.

🚢 COVID-19 Live Global Ports & Shipping-Routes Command Center · Procurement & Supply Chain
Python + REST-API ingestion pipelines feeding a live Tableau dashboard with real-time status of countries, borders and ports — best/most-affordable route recommendations across Sea, Air & Land with live climate-warning overlays. The procurement team's daily command center as global supply lines changed by the hour.

🛒 Omnichannel Retail Analytics · Merchandising & Customer Analytics
Market-basket analysis, cross-sell/upsell recommendations, loyalty segmentation, Python-based competitor price-index pipelines for dynamic pricing, and store-traffic prediction on Teradata-backed data.

Fifteen years, one direction of travel — each step a layer further down the stack, then a layer further out in scope.

Roles are described by the function they sat in rather than by employer — the work is the same wherever the letterhead came from. Happy to talk specifics in a conversation.

Primary — data engineering architecture

• ⚡ Pipelines & platform — lakehouse pipelines with Databricks & PySpark (certified Data Engineer Professional), Microsoft Fabric/OneLake, Snowflake, Starburst/Trino federation; incremental loads and jobs that are safe to re-run
• 🏗️ Modelling & target state — Data Vault 2.0 for multi-source landscapes, canonical domain models, TOGAF-driven target-state design, conformed dimensions and governed semantic layers at CDO scale
• 🛡️ Governance in the pipeline, not beside it — data quality with the teeth to stop a load, lineage captured as data, policy conformance that fails a build

Secondary — BI & analytics on top of it

• 📊 BI at scale — Tableau Server/Cloud architecture, certified data sources, row-level security, estate rationalisation; Power BI
• 🤖 Agentic analytics — Agentforce agents grounded in governed data, Tableau Pulse proactive insights, and extending DAMA-style controls to AI pipelines (IAPP AIGP)

Always happy to compare notes with people working on the same problems. Particularly interested in conversations about:

• Data engineering architecture — Data Engineering Architect, Data Architect, Principal / Lead Data Engineer, Data Platform Lead
• Lakehouse and warehouse design — Data Vault 2.0, canonical domain models, migration off legacy estates
• Chief Data Office and governance transformation — operating models, DAMA-aligned frameworks, Collibra adoption
• Enterprise BI at scale — Tableau architecture, migration and rationalisation
• Agentic AI on governed data — and where the governance actually has to sit

Open to consulting, advisory and speaking on any of the above.

📫 LinkedIn · aniketdwivedi.com · aniket.dwivedi@icloud.com

The tests live in the repositories, not in that badge — clone any of them and run pytest.