From 4adcba7cb1f6f0a3ac317726925a826c4bb8f014 Mon Sep 17 00:00:00 2001 From: "DESKTOP-691E36A\\Administrator" Date: Fri, 21 Aug 2026 15:54:18 -0600 Subject: [PATCH] docs: ADR 0002 schema registry design and full-stack demo notes --- README.md | 17 +++ contracts/CHANGELOG.md | 18 +++ contracts/registry.yml | 13 ++ ...schema-registry-and-contract-versioning.md | 124 ++++++++++++++++++ docs/architecture.md | 3 +- 5 files changed, 174 insertions(+), 1 deletion(-) create mode 100644 contracts/CHANGELOG.md create mode 100644 contracts/registry.yml create mode 100644 docs/adr/0002-schema-registry-and-contract-versioning.md diff --git a/README.md b/README.md index 0fb8b0a..4f939ac 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,9 @@ Scheduling options are documented in [`docs/scheduling.md`](docs/scheduling.md). - [x] Unit and integration tests with CI - [x] Airflow DAG `dqo_contract_checks` for scheduled contract runs - [x] Webhook alert integration tests against mock server +- [x] Contract registry catalog (`contracts/registry.yml`) — [ADR 0002](docs/adr/0002-schema-registry-and-contract-versioning.md) +- [ ] CLI resolves `--contract orders` via registry (phase 2) +- [ ] Run history stores `contract_version` metadata (phase 3) ## Technology stack @@ -100,6 +103,18 @@ Inspect recent runs: python -m src.dqo.cli history --contract orders ``` +### Full stack demo (with [production-data-pipeline](https://github.com/br413/production-data-pipeline)) + +After landing data through the companion ingestion pipeline (including quarantine/DLQ in v0.2.1), run dataset contracts against the same sample fixtures this repo ships: + +```bash +# From production-data-pipeline: ingest sample events, then return here +python -m src.dqo.cli run --contract contracts/orders.yml --data data/samples/orders.csv --references data/samples +python -m src.dqo.cli run --contract contracts/customers.yml --data data/samples/customers.csv --references data/samples +``` + +See the [Data Quality Contracts article](https://dev.to/bobby_ray_581732c715283b2/data-quality-contracts-in-production-pipelines-without-a-separate-platform-team-f3) for the full ingestion + quarantine + contract stack narrative. + ## Project structure ```text @@ -111,6 +126,8 @@ python -m src.dqo.cli history --contract orders ├── dags/ │ └── dqo_contract_checks.py ├── contracts/ +│ ├── registry.yml +│ ├── CHANGELOG.md │ ├── orders.yml │ └── customers.yml ├── data/samples/ diff --git a/contracts/CHANGELOG.md b/contracts/CHANGELOG.md new file mode 100644 index 0000000..74c8cdf --- /dev/null +++ b/contracts/CHANGELOG.md @@ -0,0 +1,18 @@ +# Contract changelog + +Version bumps follow semver policy in [ADR 0002](../docs/adr/0002-schema-registry-and-contract-versioning.md). + +## orders + +### 1.0 (initial) + +- Columns: `order_id`, `customer_id`, `order_total`, `status`, `updated_at` +- Freshness: `updated_at` max 48 hours +- RI: `customer_id` → `customers.customer_id` + +## customers + +### 1.0 (initial) + +- Columns: `customer_id`, `email`, `created_at` +- Uniqueness: `customer_id`, `email` diff --git a/contracts/registry.yml b/contracts/registry.yml new file mode 100644 index 0000000..1772fb4 --- /dev/null +++ b/contracts/registry.yml @@ -0,0 +1,13 @@ +# Contract registry — canonical catalog of dataset contracts +# +# See docs/adr/0002-schema-registry-and-contract-versioning.md for versioning policy. + +contracts: + orders: + current: "1.0" + path: orders.yml + description: Order facts for retail analytics + customers: + current: "1.0" + path: customers.yml + description: Customer dimension for referential integrity checks diff --git a/docs/adr/0002-schema-registry-and-contract-versioning.md b/docs/adr/0002-schema-registry-and-contract-versioning.md new file mode 100644 index 0000000..8d1149c --- /dev/null +++ b/docs/adr/0002-schema-registry-and-contract-versioning.md @@ -0,0 +1,124 @@ +# ADR 0002: Schema registry and contract versioning (design) + +## Status + +Proposed — design accepted; implementation tracked for a future release + +## Context + +[ADR 0001](0001-contract-driven-checks.md) established YAML contracts as the quality boundary. Each contract already carries a `version` field (e.g. `orders.yml` → `"1.0"`), but today: + +- **No registry** — consumers must know the file path; there is no canonical index of datasets and active versions +- **No compatibility rules** — adding a required column or tightening a freshness window is indistinguishable from a safe docs edit in review +- **No run attribution** — history stores check outcomes but not which contract version was evaluated +- **No cross-repo pin** — [production-data-pipeline](https://github.com/br413/production-data-pipeline) quarantine handles row-level failures; dataset contracts in this repo are not yet referenced by name/version from ingestion configs + +Production teams solve this with schema registries (Confluent, AWS Glue Schema Registry, Data Contract CLI ecosystems). This portfolio project needs a **lightweight, git-native** design that demonstrates senior platform thinking without operating a separate registry service. + +## Decision + +Introduce a **file-based contract registry** and **semver versioning policy** before any HTTP registry or warehouse integration. + +### 1. Registry index + +Add `contracts/registry.yml` as the canonical catalog: + +```yaml +contracts: + orders: + current: "1.0" + path: orders.yml + owners: [data-platform] + customers: + current: "1.0" + path: customers.yml + owners: [data-platform] +``` + +The CLI resolves `--contract orders` to `registry.yml → orders.yml` instead of requiring a file path. `current` is the default version pin for scheduled runs and CI. + +### 2. Versioning policy (semver) + +| Change type | Version bump | Example | +|-------------|--------------|---------| +| Add optional column | PATCH (`1.0` → `1.0.1`) | New nullable `discount_code` | +| Tighten freshness / add optional check | MINOR (`1.0` → `1.1`) | `max_age_hours: 48 → 24` with grace period documented | +| Add required column, remove column, rename | MAJOR (`1.0` → `2.0`) | `order_total` becomes required | +| Referential integrity change | MAJOR | New FK to a new dimension | + +Breaking changes require: + +1. Major version bump in contract YAML +2. `registry.yml` `current` update in the **same PR** as the contract change +3. Entry in `contracts/CHANGELOG.md` (new file) describing consumer impact + +### 3. Run history attribution + +Extend `RunSummary` metadata (future implementation) with: + +- `contract_name` +- `contract_version` +- `registry_revision` (git SHA or optional content hash of registry + contract file) + +This makes quality regressions auditable: “orders failed freshness on **v1.1** starting 2026-08-12.” + +### 4. Compatibility with ingestion quarantine + +Registry design keeps **row-level** and **dataset-level** gates separate: + +| Boundary | Project | Responsibility | +|----------|---------|----------------| +| Row-level poison pills | `production-data-pipeline` | Quarantine during API ingestion ([ADR 0004](https://github.com/br413/production-data-pipeline/blob/main/docs/adr/0004-failed-record-quarantine.md)) | +| Dataset contract | `data-quality-observability` | Schema, freshness, RI after landing / before promote | + +Future optional integration: pipeline config pins `quality_contract: orders@1.0` and exports landed CSV/Parquet for `dqo.cli run --contract orders --version 1.0`. Not required for registry v1. + +### 5. CI enforcement (future) + +When implemented: + +- `pytest` loads registry and validates every `path` exists +- PR check fails if contract `version` changes without registry or CHANGELOG update +- `validate --strict` mode rejects datasets that do not match `current` pin + +## Consequences + +**Positive** + +- Contracts become discoverable and pin-able by name/version +- Breaking changes are explicit in review, not silent schema drift +- History can answer “which contract version failed?” — critical for on-call +- Design scales to a real registry service later without rewriting contracts + +**Negative** + +- Extra files to maintain (`registry.yml`, `CHANGELOG.md`) +- Semver discipline adds friction for solo devs — acceptable for portfolio signal +- Full cross-repo pinning with `production-data-pipeline` is follow-on work + +## Alternatives considered + +| Alternative | Why not (for this portfolio) | +|-------------|--------------------------------| +| **Confluent Schema Registry** | Heavy ops; Avro/Protobuf focus; obscures readable YAML contract story | +| **dbt exposures / metrics only** | Warehouse-native; does not cover pre-load CSV/API validation | +| **Version field only, no registry** | Current state — does not scale past two contracts | +| **Git tags as versions** | Poor discoverability; hard to pin in Airflow/CLI | + +## Implementation phases + +| Phase | Scope | Deliverable | +|-------|-------|-------------| +| **1** | Registry + docs | `contracts/registry.yml`, ADR 0002, README update | +| **2** | CLI resolution | `--contract orders` resolves via registry; `--version` override | +| **3** | History metadata | Persist `contract_version` in run history | +| **4** | CI guards | Registry consistency tests; CHANGELOG requirement | +| **5** | Pipeline pin (optional) | Config reference from `production-data-pipeline` | + +Phases 1–2 are the minimum viable registry story for portfolio reviewers. + +## References + +- [Data Quality Contracts in Production Pipelines (Dev.to)](https://dev.to/bobby_ray_581732c715283b2/data-quality-contracts-in-production-pipelines-without-a-separate-platform-team-f3) +- [production-data-pipeline ADR 0004 — quarantine](https://github.com/br413/production-data-pipeline/blob/main/docs/adr/0004-failed-record-quarantine.md) +- [architecture.md](../architecture.md) diff --git a/docs/architecture.md b/docs/architecture.md index 12ee2d7..4e9509f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -8,6 +8,7 @@ Data platforms need a repeatable way to detect contract violations before bad da | Component | Responsibility | |-----------|----------------| +| `contracts/registry.yml` | Canonical catalog — dataset name → current version + file path | | `contracts/*.yml` | Versioned schema, freshness, and FK expectations | | `src/dqo/checks/*` | Individual validation rules | | `src/dqo/runner.py` | Orchestrates checks into a run summary | @@ -52,4 +53,4 @@ Contract YAML + CSV dataset + reference tables - **Optional PostgreSQL** — shared history for team dashboards - **Pluggable alert channels** — console for dev, file/webhook for ops -See [`docs/adr/0001-contract-driven-checks.md`](adr/0001-contract-driven-checks.md). +See [`docs/adr/0001-contract-driven-checks.md`](adr/0001-contract-driven-checks.md) and [`docs/adr/0002-schema-registry-and-contract-versioning.md`](adr/0002-schema-registry-and-contract-versioning.md) (registry + semver design).