diff --git a/ASSETS.md b/ASSETS.md index 99ab02f..9985b93 100644 --- a/ASSETS.md +++ b/ASSETS.md @@ -22,7 +22,7 @@ Run the command from the root of your bundle project. The CLI will prompt for an | `monitoring-sql-warehouse` | Small, dedicated serverless SQL warehouse (2X-Small, `auto_stop_mins: 1`) for scheduled Databricks Alerts and monitoring queries. Keeps cost proportional to actual query time instead of idle warm-up. | Stable | [README](assets/monitoring-sql-warehouse/README.md) | | `sdp-quarantine-pattern` | Lakeflow SDP pipeline demonstrating the inverse-expectations quarantine pattern on `samples.nyctaxi.trips`: critical (drop) expectations route bad rows into a separate quarantine table (silver schema), valid rows flow to silver, warn expectations log to a queryable event log. NULL-safe predicates keep the silver/quarantine split a clean partition. Ships a companion agent skill (`SKILL.md`) that adapts the pattern to your own dataset and self-verifies it. | Stable | [README](assets/sdp-quarantine-pattern/README.md) | | `pyspark-test-runner` | Single-file Python wrapper around `pytest` for local PySpark suites that prints a bounded, agent-friendly digest (counts, runnable failing node ids, failures deduplicated by signature) and keeps full output in a log file, so a suite that floods with repetitive failures does not burn a coding agent's context window. Ships a `SKILL.md` for agent integration. | Stable | [README](assets/pyspark-test-runner/README.md) | -| `sdp-expectation-notifications` | Per-expectation data-quality notification for Lakeflow SDP as a validated pair: a native event hook notifies the moment a WARN expectation result is logged (fast, best-effort by platform design, throttled to at most one notification per expectation per window via a two-layer time-aware de-dup with optional UC Volume marker state), and one DABs-managed Alert v2 sweeps the published pipeline event log over a past-time window on a schedule (guaranteed, one email per state change). Webhook payloads for Slack, Teams, or generic receivers, with secret-scope URL resolution. Demo pipeline on `samples.nyctaxi.trips` fires both paths on the first run. Ships a companion agent skill (`SKILL.md`) that wires the pattern into your own SDP pipelines. | Stable | [README](assets/sdp-expectation-notifications/README.md) | +| `sdp-expectation-notifications` | Per-expectation data-quality notification for Lakeflow SDP as a validated pair: a native event hook notifies the moment an expectation result with failed records is logged, covering `warn` and `drop` alike (fast, best-effort by platform design, throttled to at most one notification per expectation per window via a two-layer time-aware de-dup with optional UC Volume marker state), and one DABs-managed Databricks SQL alert sweeps the published pipeline event log over a past-time window on a schedule (guaranteed, one email per state change). Webhook payloads for Slack, Teams, or generic receivers, with secret-scope URL resolution. Demo pipeline on `samples.nyctaxi.trips` fires both paths on the first run. Ships a companion agent skill (`SKILL.md`) that wires the pattern into your own SDP pipelines. | Stable | [README](assets/sdp-expectation-notifications/README.md) | ## What an asset is not diff --git a/CHANGELOG.md b/CHANGELOG.md index 90b181c..0317392 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [1.12.1] - 2026-07-26 + +### Changed +- **Asset `sdp-expectation-notifications` (docs only)**: seven corrections grounded in two live verification sessions (2026-07-26 drop/destination testing and the 2026-07-17/18 hygiene build), none changing behavior. + - **Drop coverage documented.** The hook filters purely on `failed_records > 0` and never inspects the expectation's action type, so `warn` and `drop` rules notify identically; the README previously framed the pattern as WARN-only. Live 2026-07-26: a `drop` expectation on the same source and tripwire produced the same passed/failed shape as the shipped WARN row (18929/3003 on 21,932 rows), the hook fired with a matching driver-log line, the update reported `COMPLETED` while discarding 13.7% of rows, `dropped_records` (DROP-only) read 3003 vs 0 on WARN, and the backstop's sweep summed both expectations additively (6,006). + - **Naming corrected.** "Alert v2" and "Alerts v2" replaced with "Databricks SQL alert(s)" throughout the README, in-bundle doc, skill files, and install prompt text, matching current Databricks docs (legacy alerts is the prior version); `alerts-v2` is kept only where it names an actual CLI command. + - **Notification destinations clarified.** Destinations serve only Databricks SQL and jobs, not the event hook, and a destination is a wrapper around a webhook, not a substitute for one (Slack needs a webhook URL, OAuth token, and channel id plus Slack app-install rights; Teams needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID plus Microsoft Copilot Studio and Entra ID permissions). Measured live: a destination alongside `user_email` on the same address sends two independent, non-deduplicated notifications. + - **No-webhook guidance added.** Readers and the companion skill are now steered to deploy the backstop alone when there is no Slack, Teams, or webhook target, since print-only hook output has no advantage over the event log the backstop already sweeps. + - **Backstop cron tied to pipeline mode.** The shipped daily cron fits a triggered pipeline; guidance now flags that a continuous pipeline (10-seconds-to-a-few-minutes freshness) needs a tighter cron and window, at the cost of more warehouse wake-ups, pointing at the `monitoring-sql-warehouse` asset. + - **Grace-period claim softened.** "Terminates compute within roughly 20 seconds" overclaimed precision two runs never measured; replaced with the honest bound (compute alive at ~8s in one run, dead before ~23s in another, neither designed to measure the timer) and the actionable rule: keep hook work in single-digit seconds. + - **Skill gains a CLI gotcha.** `databricks notification-destinations create` prints a spurious `unknown field` warning and echoes back a config that looks empty even though the destination was created correctly; documented in `adapt-the-pattern.md` since the skill is what would run this command. + ## [1.12.0] - 2026-07-18 ### Changed diff --git a/ROADMAP.md b/ROADMAP.md index ab4f073..9efee3b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -51,7 +51,7 @@ Modular sub-templates installable via `databricks bundle init --templ - `monitoring-sql-warehouse`: Dedicated serverless SQL warehouse tuned for bursty workloads (scheduled Alerts, monitoring queries) with `auto_stop_mins: 1` - `sdp-quarantine-pattern`: Lakeflow SDP pipeline demonstrating the inverse-expectations quarantine pattern (drop expectations route bad rows to a separate quarantine table) on `samples.nyctaxi.trips`, with medallion schema separation, a queryable event log, and NULL-safe predicates. Ships a companion agent skill that adapts the pattern to the user's own dataset and self-verifies it - `pyspark-test-runner`: Single-file Python wrapper around `pytest` for local PySpark suites that prints a bounded, agent-friendly digest (counts, runnable failing node ids, failures deduplicated by signature) and keeps full output in a log file, so a suite flooding with repetitive failures does not burn a coding agent's context window. Ships a `SKILL.md` for agent integration -- `sdp-expectation-notifications`: Per-expectation data-quality notification for Lakeflow SDP as a pair: an in-pipeline event hook fires the moment a WARN expectation result is logged (since v1.12.0 throttled to at most one notification per expectation per window via a two-layer time-aware de-dup with optional UC Volume marker state, with Slack/Teams/generic webhook formats and secret-scope URL resolution), backed by one DABs-managed Alert v2 sweeping the published event log on a schedule (hook delivery is best-effort by platform design; the alert is the guarantee, one email per state change). Ships a companion agent skill that wires the pattern into the user's own pipelines +- `sdp-expectation-notifications`: Per-expectation data-quality notification for Lakeflow SDP as a pair: an in-pipeline event hook fires the moment an expectation result with failed records is logged, covering `warn` and `drop` alike (since v1.12.0 throttled to at most one notification per expectation per window via a two-layer time-aware de-dup with optional UC Volume marker state, with Slack/Teams/generic webhook formats and secret-scope URL resolution), backed by one DABs-managed Databricks SQL alert sweeping the published event log on a schedule (hook delivery is best-effort by platform design; the alert is the guarantee, one email per state change). Ships a companion agent skill that wires the pattern into the user's own pipelines **Possible future assets** (ideas for contributors, not commitments): - `etl-pipeline`: Medallion-layered Declarative Pipeline with Bronze/Silver layers and DLT expectations @@ -85,7 +85,40 @@ These are larger features that require more design work and community input befo ### Pre-commit Hooks - +**Status**: Proposed +**Target**: v2.0 + +Optional git pre-commit hook that runs `databricks bundle validate` locally before a commit, catching DAB syntax and configuration errors before they reach CI/CD. Off by default: it requires a local Databricks CLI profile with workspace auth already configured, and like any pre-commit hook it can be bypassed with `--no-verify`, so it complements the CI/CD pipeline's validation step rather than replacing it. + +**Scope:** +- New `include_pre_commit_hook` prompt (`yes` / `no`, default `no`) +- Generates `.pre-commit-config.yaml` wiring a hook that runs `databricks bundle validate -t ` and fails the commit on validation errors +- Documentation note on installing the `pre-commit` framework and running `pre-commit install` + +**Open questions:** +- Which target does the local validate run against by default: `user`, or whichever the developer has authenticated to? +- Does the hook need a dependency on `pre-commit` (plus a task runner like Poe) in the generated project, or can it ship as a lighter shell-script hook with no extra Python framework? +- How should the hook behave when a developer has no local CLI auth configured yet: skip gracefully, or hard fail? + +Inspired by: [Pre-commit hook for Declarative Automation Bundles](https://www.waitingforcode.com/databricks/pre-commit-hook-declarative-automation-bundles-example/read) + +### Shared Bundles Support + +**Status**: Proposed +**Target**: v2.0 + +Scaffold a generated project to consume an existing external "shared" bundle path, a separate repo or directory holding common libraries, compute configs, or variables reused across multiple bundles, using Databricks' native `sync.paths` / `include` mechanism rather than anything template-specific. Useful for organizations running several bundles that want centralized cluster configs, shared code, or shared variable definitions instead of duplicating them per bundle. + +**Scope:** +- New `shared_bundle_path` prompt (optional; blank by default, no change to current single-bundle behavior) +- When set, wires the path into `sync.paths` and `include` in `databricks.yml.tmpl` (for example `../shared/*.yml`, `../shared` as a sync path) +- Documentation on the prerequisites (DBR 14+ or serverless for certain shared Python path operations; permissions must be configured at both bundle and target level for all consumers) and on running `databricks bundle validate` to confirm inheritance before deploying + +**Open questions:** +- Should the template scaffold the shared bundle itself as a second generated project, or only wire an existing one in? +- How does this interact with the existing service-principal and permissions model per environment target? + +Reference: [Share code and configuration between bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/sharing) ### Advanced Permissions Profiles diff --git a/assets/sdp-expectation-notifications/README.md b/assets/sdp-expectation-notifications/README.md index 604c680..939a515 100644 --- a/assets/sdp-expectation-notifications/README.md +++ b/assets/sdp-expectation-notifications/README.md @@ -1,6 +1,6 @@ # sdp-expectation-notifications -Per-expectation data-quality notification for Lakeflow Spark Declarative Pipelines (SDP), as a pair: a scheduled DABs-managed **Alert v2** sweeps the published pipeline event log (the guaranteed path, throttled by the platform to one email per state change), and a native **event hook** posts the moment a WARN expectation result is logged (the fast path, throttled by this asset to one notification per expectation per window). The pair exists because the platform explicitly does not guarantee hook delivery; each half covers the other's gap. +Per-expectation data-quality notification for Lakeflow Spark Declarative Pipelines (SDP), as a pair: a scheduled DABs-managed **Databricks SQL alert** sweeps the published pipeline event log (the guaranteed path, throttled by the platform to one email per state change), and a native **event hook** posts the moment an expectation result with failed records is logged, whether the expectation's action is `warn` or `drop` (the fast path, throttled by this asset to one notification per expectation per window). The pair exists because the platform explicitly does not guarantee hook delivery; each half covers the other's gap. ## Install @@ -36,7 +36,7 @@ samples.nyctaxi.trips │ └─► published event log (UC table: dq_notifications_event_log) ▲ - └── backstop Alert v2 scheduled sweep over a trailing + └── backstop SQL alert scheduled sweep over a trailing email / destinations 1-day window; guaranteed 1 email per state change evaluation; notify_on_ok closes the loop on recovery @@ -61,9 +61,11 @@ An update that processes no new rows emits no expectation events at all (measure The two lanes get channels in two different ways: -- **Backstop (email and beyond, zero code):** the alert subscribes an email by default. For Slack, MS Teams, PagerDuty, or a generic webhook, create a workspace notification destination (admin settings) and reference it via the commented `destination_id` line in the alert YAML. Platform feature, nothing to implement. +- **Backstop (email and beyond, zero code):** the alert subscribes an email by default. Notification destinations can only be configured for Databricks SQL and jobs, so a destination is how Slack, MS Teams, PagerDuty, or a generic webhook attach to the backstop: create a workspace notification destination (admin settings) and reference it via the commented `destination_id` line in the alert YAML. A destination is a wrapper around a webhook, not a substitute for one: a Slack destination's config is a webhook URL plus an OAuth token and channel id, requiring app-install rights in the Slack workspace; a Teams destination needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID, requiring permissions in Microsoft Copilot Studio and Entra ID on top of Databricks workspace admin. EMAIL is the only destination type taking plain addresses with no webhook. The honest benefit is centralizing the credential once at workspace level instead of pasting it into every alert, not "Slack without a webhook." Also worth knowing (measured live 2026-07-26): a destination and a `user_email` pointing at the same address produce two independent, non-deduplicated sends per trigger, not a merged one. - **Hook (webhook formats):** `dq_notify.channel_format` selects the payload: `slack` (`{"text": ...}`, validated end to end: secret scope to channel message, HTTP 200), `teams` (Workflows Adaptive Card envelope, documented format, not live-tested by this asset), or `generic` (plain JSON with the violation fields). Route hook posts to chat-style channels that tolerate an occasional repeat; email belongs to the backstop, and a hook cannot send email anyway (no email surface exists inside a hook). +**No webhook available? Deploy the backstop alone.** Without a delivery surface, print-only hook output has no advantage over the published event log the backstop already sweeps; the hook's only edge is latency, and that's worthless with nowhere to deliver it. If you have no Slack, Teams, or webhook target, skip the hook and ship just the backstop alert. + **Webhook URLs are credentials** (a Slack incoming webhook grants posting rights). Real setups put the URL in a secret scope and set `dq_notify.webhook_secret_scope`/`dq_notify.webhook_secret_key`; the pipeline resolves it via `dbutils.secrets.get` at startup, which works at module level in serverless SDP Python (validated live), and degrades to print-only if the scope is missing (also validated). Two things that do NOT work, so you do not have to rediscover them: `{{secrets/scope/key}}` in pipeline configuration is not interpolated (the literal string arrives), and the plain `dq_notify.webhook_url` config is a demo-only convenience. ## The rules of the hook (learned the honest way) @@ -73,11 +75,12 @@ Established on live serverless SDP runs (2026-07-11 and 2026-07-17/18, CLI v0.29 - **Do not write Delta from inside a hook.** `spark.sql()` in SDP pipeline Python is restricted to a read allowlist: `INSERT INTO` fails with `UNSUPPORTED_SPARK_SQL_COMMAND` (the error lists the allowlist: SELECT, DESCRIBE, SHOW variants, USE), `CREATE TABLE` fails the same way, and the SDP-patched `spark.sql()` rejects the parameterized-query `args=` kwarg. All three write paths are closed; this is definitive. - **Durable state goes through UC Volume file I/O instead.** Plain Python `open()` against `/Volumes/...` works from inside a hook: create, overwrite, read, `os.path.exists`, `os.listdir`, `os.stat`, `os.makedirs` all validated live. The one gap: append to an existing file fails (`OSError [Errno 29] Illegal seek`), which is why the markers are overwrite-only. - **Reads are fine.** `spark.sql("SELECT ...").collect()` works reliably from a hook (validated across many invocations), so a hook can consult a small gate table if you need a manual mute switch. -- **Keep the hook fast; the grace budget is seconds.** Hooks run serialized, and after an update completes the platform terminates compute within roughly 20 seconds. A hook needing about 1s per event drained 5 queued events with zero loss; a hook needing 20s per event lost **6 of 6** notifications under normal, non-forced teardown. No retries, tight webhook timeout (5s), marker I/O only (~100ms). +- **Keep the hook fast; the grace budget is seconds.** Hooks run serialized, and compute is torn down shortly after an update completes. A hook needing about 1s per event drained 5 queued events with zero loss, all within about 8 seconds of completion. A hook needing 20s per event had its first event start 3 seconds after completion, never finish, and lost **6 of 6** notifications under normal, non-forced teardown. Keep total hook work in single-digit seconds. No retries, tight webhook timeout (5s), marker I/O only (~100ms). The exact platform timer is not documented and we have not established it. - **`hook_progress` records ENABLED, FAILED, and DISABLED transitions** (correcting v1.11.0, which claimed enable-state only). Hook health is event-log-observable and alertable. It still does not record per-invocation output; the printed notifications exist only in the pipeline compute's driver log, which has no CLI or SQL surface. - **`max_allowable_consecutive_failures` is a tradeoff.** A finite value disables a flaky hook until the next restart (visible as `DISABLED` in `hook_progress`, but notifications silently stop); `None` (this asset's choice) lets a broken hook fail forever, which is why every failure path in the shipped hook prints instead of raising. - **`mode: development` masks the teardown behavior.** A DABs development-mode target keeps pipeline compute warm across updates, hiding both the delivery gap and the fresh-process-per-update behavior (in non-dev mode each update gets a new Python process; measured via per-process ids). - **Inside the hook, `event["details"]` arrives as a native Python dict**, and `event["origin"]` carries `pipeline_name` and `pipeline_id` (the basis for zero-config multi-pipeline state scoping). +- **The hook covers `warn` and `drop` identically, not just `warn`.** It filters purely on `failed_records > 0` inside the expectations array and never inspects the expectation's action type, so `@dp.expect` and `@dp.expect_or_drop` rules notify the same way. The demo ships one `warn` rule because `warn` keeps every row flowing, which makes the demo repeatable; it is not a limit of the hook itself. `details.flow_progress.data_quality.dropped_records` is DROP-only (0 on a WARN row) and is how you tell from the event log alone that rows were discarded rather than merely flagged. Databricks' own docs exclude `fail` from this metrics recording entirely, because a `fail` action stops the update instead of letting it complete. ## The backstop's notification behavior (measured) @@ -85,6 +88,7 @@ Established on live serverless SDP runs (2026-07-11 and 2026-07-17/18, CLI v0.29 - **`retrigger_seconds` is deliberate re-nagging:** re-sends at the first evaluation past each window (measured with 120s: gaps of 120s, 180s, 179s, aligned to evaluation ticks). Off by default; the commented block suggests 86400 as a daily "still broken" escalation for critical pipelines. - **ERROR does not throttle.** If the sweep query itself fails, the alert flips to ERROR and emails on **every evaluation** until fixed (measured: 5 emails in 4 minutes). The backstop cannot die silently; treat an `(ERROR)` email as a page. - The sweep query aggregates over a past-time window (trailing 1 day), not just the latest row; keep the window at least as long as the cadence if you change the cron. +- **Cron cadence should match the pipeline's mode.** The shipped daily 06:00 UTC cron, paired with a 1-day window, fits a triggered pipeline on a daily-or-slower cadence. A continuous pipeline is a different shape: Databricks positions continuous mode for freshness requirements between 10 seconds and a few minutes, so a daily sweep adds a full day of latency on top of that. Tighten both the cron and the `INTERVAL` together if the pipeline runs continuously, but tightening costs more warehouse wake-ups; the `monitoring-sql-warehouse` asset in this repo (2X-Small serverless, `auto_stop_mins: 1`) is sized for exactly that tradeoff. ## Validation results @@ -94,7 +98,8 @@ v1.11.0's pattern was validated end to end on 2026-07-11 (hook fires with exact - Two-layer throttle: 12 raw events across two consecutive full refreshes reduced to 2 notifications; cross-update suppression through marker files after a process restart; multi-pipeline isolation with two pipelines sharing one state root. - Continuous mode with Auto Loader: notify, suppress inside the window, re-notify after it, in one long-lived process; continuous pipelines auto-start on `bundle deploy`. - Slack end to end: secret scope, `dbutils.secrets.get` at module level, throttled hook, two channel messages with HTTP 200 and counts matching the hook records exactly; missing secret scope degraded to print-only with the pipeline unaffected. -- Alerts v2: one email per state transition (9 TRIGGERED evaluations, 1 email), `retrigger_seconds` re-send alignment, exactly one `notify_on_ok` recovery email, ERROR emailing every evaluation. +- Databricks SQL alerts: one email per state transition (9 TRIGGERED evaluations, 1 email), `retrigger_seconds` re-send alignment, exactly one `notify_on_ok` recovery email, ERROR emailing every evaluation. +- Drop-expectation coverage, live 2026-07-26: a `drop` expectation on the same source and tripwire condition produced the same passed/failed shape as the WARN row (18929 passed / 3003 failed on 21,932 total rows), the hook fired with an exact-matching driver-log line, the update reported `COMPLETED` while discarding 3,003 of 21,932 rows (13.7%), `dropped_records` read 3003 on the DROP row versus 0 on the WARN row, and the backstop's sweep query summed both expectations with no special-casing (6,006 = 3,003 + 3,003). ## Honest limits and open questions @@ -102,7 +107,7 @@ v1.11.0's pattern was validated end to end on 2026-07-11 (hook fires with exact - The two lanes de-duplicate independently and share no state: one violation can produce one hook message and one backstop email. Different audiences, different guarantees; by design. - The hook cannot notice "the pipeline has not run at all": no update, no events, no hook. Pair with a freshness check (for example Unity Catalog data quality monitoring's anomaly detection) if that failure mode matters. - The backstop inherits scheduled-scan limits: latency bounded by its cron cadence, and a SQL warehouse in the loop. The `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) exists for exactly this workload shape. -- Still open: the Teams payload ships doc-confirmed, not live-tested; concurrent marker writes from two simultaneously running pipelines are untested (the per-pipeline folders give them no shared file to race on, but Free Edition cannot run two updates at once to prove it); the exact numeric grace period (bounded to roughly 8 to 20-plus seconds of post-update hook budget by measurement). +- Still open: the Teams payload ships doc-confirmed, not live-tested; concurrent marker writes from two simultaneously running pipelines are untested (the per-pipeline folders give them no shared file to race on, but Free Edition cannot run two updates at once to prove it); the exact numeric grace period remains unestablished, two runs bound it loosely (one showed compute still alive at ~8s, one showed it dead before ~23s), but neither run was designed to measure the timer itself. ## Inspecting the results @@ -124,5 +129,5 @@ It demonstrates one notification pattern with its tradeoffs documented, not a ge 2. [Lakeflow SDP expectations](https://docs.databricks.com/aws/en/ldp/expectations) 3. [Monitor pipelines with the event log](https://docs.databricks.com/aws/en/ldp/monitor-event-logs) 4. [`event_log` table-valued function](https://docs.databricks.com/aws/en/sql/language-manual/functions/event_log) -5. [DABs alert resource (Alerts v2)](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) +5. [DABs alert resource](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) 6. [Notification destinations](https://docs.databricks.com/aws/en/admin/workspace-settings/notification-destinations) diff --git a/assets/sdp-expectation-notifications/databricks_template_schema.json b/assets/sdp-expectation-notifications/databricks_template_schema.json index 496d09a..af9356c 100644 --- a/assets/sdp-expectation-notifications/databricks_template_schema.json +++ b/assets/sdp-expectation-notifications/databricks_template_schema.json @@ -1,5 +1,5 @@ { - "welcome_message": "\nSDP Expectation Notifications: Asset Installer\n\nInstalls two things that ship together:\n 1. A reference Lakeflow Spark Declarative Pipeline demonstrating\n per-expectation data-quality notification: a native event hook fires the\n moment a WARN expectation result is logged, on the public\n samples.nyctaxi.trips dataset (the worked example).\n 2. A companion agent skill that instructs a coding agent to wire the same\n hook-plus-backstop pattern into YOUR own SDP pipeline.\n\nThe pattern is a pair, not a single mechanism: event hook delivery is\nexplicitly best-effort (the platform does not guarantee every hook runs\nbefore pipeline compute terminates), so the asset also installs exactly one\nDABs-managed Alert v2 that sweeps the published pipeline event log on a\nschedule as the guaranteed path. Hook = fast, alert = guaranteed.\n\nThe hook is throttled by design: without it, one failing expectation\nnotifies once per microbatch (measured live), which floods on multi-batch\nand continuous pipelines. A time-aware two-layer throttle (in-memory always,\ndurable volume markers optionally) caps it at one notification per\nexpectation per window.\n\nThe demo WARN expectation is a deliberate tripwire chosen to fail on the\nhealthy public sample, so the hook fires and the backstop alert triggers on\nthe first run with no extra setup.\n\nLet's pick the install settings...\n", + "welcome_message": "\nSDP Expectation Notifications: Asset Installer\n\nInstalls two things that ship together:\n 1. A reference Lakeflow Spark Declarative Pipeline demonstrating\n per-expectation data-quality notification: a native event hook fires the\n moment a WARN expectation result is logged, on the public\n samples.nyctaxi.trips dataset (the worked example).\n 2. A companion agent skill that instructs a coding agent to wire the same\n hook-plus-backstop pattern into YOUR own SDP pipeline.\n\nThe pattern is a pair, not a single mechanism: event hook delivery is\nexplicitly best-effort (the platform does not guarantee every hook runs\nbefore pipeline compute terminates), so the asset also installs exactly one\nDABs-managed Databricks SQL alert that sweeps the published pipeline event\nlog on a schedule as the guaranteed path. Hook = fast, alert = guaranteed.\n\nThe hook is throttled by design: without it, one failing expectation\nnotifies once per microbatch (measured live), which floods on multi-batch\nand continuous pipelines. A time-aware two-layer throttle (in-memory always,\ndurable volume markers optionally) caps it at one notification per\nexpectation per window.\n\nThe demo WARN expectation is a deliberate tripwire chosen to fail on the\nhealthy public sample, so the hook fires and the backstop alert triggers on\nthe first run with no extra setup.\n\nLet's pick the install settings...\n", "properties": { "target_dir": { diff --git a/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md b/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md index 4e31c47..ac77151 100644 --- a/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md +++ b/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md @@ -1,6 +1,6 @@ # SDP Expectation Notifications (in-bundle usage) -This asset installs per-expectation data-quality notification for a Lakeflow Spark Declarative Pipeline as a pair: a native event hook fires the moment a WARN expectation result is logged (fast, best-effort, throttled to at most one notification per expectation per window), and one DABs-managed Alert v2 sweeps the published pipeline event log on a schedule (guaranteed, one email per state change). The demo pipeline runs on the public `samples.nyctaxi.trips` dataset with a deliberately tight WARN expectation, so both paths fire on the first run. +This asset installs per-expectation data-quality notification for a Lakeflow Spark Declarative Pipeline as a pair: a native event hook fires the moment an expectation result with failed records is logged, whether the expectation's action is `warn` or `drop` (fast, best-effort, throttled to at most one notification per expectation per window), and one DABs-managed Databricks SQL alert sweeps the published pipeline event log on a schedule (guaranteed, one email per state change). The demo pipeline runs on the public `samples.nyctaxi.trips` dataset with a deliberately tight WARN expectation, so both paths fire on the first run. The full design rationale, the validated facts, and the honest caveats live in the asset README in the [databricks-bundle-template repo](https://github.com/vmariiechko/databricks-bundle-template/tree/main/assets/sdp-expectation-notifications). This file covers what you do after install. @@ -11,7 +11,7 @@ The full design rationale, the validated facts, and the honest caveats live in t | `/expectation_notifications_pipeline.py` | The SDP pipeline source: demo table, WARN expectation, and the notifier event hook. | | `/event_log_queries.sql` | Queries for expectation counts, `hook_progress` states, and the backstop's sweep. | | `resources/.pipeline.yml` | The DABs pipeline resource (publishes the event log to a UC table). | -| `resources/_backstop.alert.yml` | The backstop Alert v2 resource (scheduled sweep + email subscription). | +| `resources/_backstop.alert.yml` | The backstop alert resource (a Databricks SQL alert; scheduled sweep + email subscription). | | `/skills/sdp-expectation-notifications/SKILL.md` | Companion agent skill: wire the pattern into your own pipeline. | | `docs/sdp-expectation-notifications/README.md` | This file. | @@ -49,6 +49,8 @@ The first update ingests the full sample table, so the demo WARN expectation (`d The hook posts each due notification to a webhook when one is configured, in the format selected by `dq_notify.channel_format` (`slack`, `teams`, or `generic`). Route hook posts to chat-style channels; email belongs to the backstop alert. +No Slack, Teams, or generic webhook target available? Skip this section and the hook's webhook config entirely: print-only output has no advantage over the event log the backstop already sweeps, so the backstop alone covers you (see the asset README's "Notification channels" section for the full reasoning). + A webhook URL is a credential (it grants posting rights), so the real setup goes through a secret scope: ```bash @@ -84,7 +86,7 @@ then apply them and guide me through (or do) the deploy and verification. ## Adjusting the backstop -- **Cadence and window are coupled.** The alert query aggregates the trailing `INTERVAL 1 DAY`; the schedule is daily. If you change the cron, change the interval to stay at least as long as the cadence, or violations can fall between sweeps. +- **Cadence and window are coupled, and cadence should match your pipeline's mode.** The alert query aggregates the trailing `INTERVAL 1 DAY`; the schedule is daily. If you change the cron, change the interval to stay at least as long as the cadence, or violations can fall between sweeps. The shipped daily cadence fits a triggered pipeline; a continuous pipeline needs a tighter cron and window (Databricks positions continuous mode for freshness between 10 seconds and a few minutes, so a daily sweep would add a full day of latency on top of that), but a tighter cron means more warehouse wake-ups: the `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) is sized for that tradeoff. - **Notification behavior (measured live):** one email per state transition, however many evaluations stay TRIGGERED; `notify_on_ok: true` (the shipped default) adds exactly one recovery email; the commented `retrigger_seconds` re-sends while TRIGGERED at most once per window, a deliberate escalation knob (for example 86400 for a daily reminder on a critical pipeline). While the alert stays TRIGGERED, new failures inside the window send nothing; the recovery email is what re-arms attention. - **If the sweep query itself breaks** (dropped table, deleted warehouse), the alert flips to ERROR and emails on every evaluation until fixed. It cannot die silently; treat an `(ERROR)` email as a page. - **Recipients** live in `evaluation.notification.subscriptions` (add more `user_email` entries, or a `destination_id` for a workspace notification destination, which is how Slack, MS Teams, PagerDuty, and generic webhooks attach to the backstop with zero code). @@ -95,5 +97,5 @@ then apply them and guide me through (or do) the deploy and verification. 1. [Lakeflow SDP event hooks](https://docs.databricks.com/aws/en/ldp/event-hooks) 2. [Lakeflow SDP expectations](https://docs.databricks.com/aws/en/ldp/expectations) 3. [Monitor pipelines with the event log](https://docs.databricks.com/aws/en/ldp/monitor-event-logs) -4. [DABs alert resource (Alerts v2)](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) +4. [DABs alert resource](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) 5. [Databricks sample datasets (`samples.nyctaxi.trips`)](https://docs.databricks.com/aws/en/discover/databricks-datasets) diff --git a/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl b/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl index e37d8c0..7f6b33a 100644 --- a/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl +++ b/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl @@ -1,16 +1,19 @@ # Backstop alert: the guaranteed sweep behind the best-effort event hook. # -# One DABs-managed Alert v2 that aggregates expectation failures from the -# published pipeline event log over a trailing time window. It exists because -# event hook delivery is explicitly best-effort: there is no guarantee every -# hook runs on every event before pipeline compute terminates. A scheduled -# SQL alert either runs or visibly fails, so it closes that gap. +# One DABs-managed Databricks SQL alert that aggregates expectation failures +# from the published pipeline event log over a trailing time window. It +# exists because event hook delivery is explicitly best-effort: there is no +# guarantee every hook runs on every event before pipeline compute +# terminates. A scheduled SQL alert either runs or visibly fails, so it +# closes that gap. # # The query aggregates over a past-time window (last 1 day), NOT just the # latest row: a latest-row read silently misses violations whenever the alert # cadence differs from the pipeline cadence. Keep the window at least as long # as the evaluation cadence; if you change the cron below, change the -# INTERVAL to match. +# INTERVAL to match. The shipped daily cadence fits a triggered pipeline; a +# continuous pipeline needs both tightened to match its faster freshness +# (see the asset README's backstop notification-behavior section). # # Placeholders: if warehouse_id or the subscription email below reads # *_PLACEHOLDER, replace it before deploying. The monitoring-sql-warehouse @@ -58,7 +61,17 @@ resources: - user_email: {{.notification_email}} # Slack / MS Teams / PagerDuty / webhook delivery: create a # workspace notification destination (admin settings) and - # reference it here instead of, or alongside, the email: + # reference it here instead of, or alongside, the email. A + # destination is a wrapper around a webhook, not a substitute for + # one: Slack needs a webhook URL, an OAuth token, and a channel + # id, plus app-install rights in Slack; Teams needs a webhook + # URL, App ID, Auth Secret, Channel URL, and Tenant ID, plus + # permissions in Microsoft Copilot Studio and Entra ID, on top of + # Databricks workspace admin. EMAIL is the only type taking plain + # addresses with no webhook. The benefit is centralizing the + # credential once at workspace level instead of per-alert. Note + # (measured live): a destination alongside user_email on the same + # address sends two independent, non-deduplicated notifications. # - destination_id: # Operational note (validated live): if this alert's query itself # fails (dropped table, deleted warehouse), the alert flips to ERROR diff --git a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/SKILL.md.tmpl b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/SKILL.md.tmpl index e5af4c8..05c3d56 100644 --- a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/SKILL.md.tmpl +++ b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/SKILL.md.tmpl @@ -1,13 +1,13 @@ --- name: sdp-expectation-notifications -description: Use this skill to add per-expectation data-quality notifications to the user's own Lakeflow Spark Declarative Pipeline (SDP). The pattern is a pair, an in-pipeline event hook that notifies the moment an expectation result with failures is logged (fast, best-effort, throttled to at most one notification per expectation per window) plus exactly one DABs-managed Alert v2 that sweeps the published pipeline event log on a schedule (guaranteed, one email per state change). Use it whenever the user wants to be notified when expectations fail, wants alerting on WARN expectations that would otherwise only sit in the event log, or asks for event-hook-based monitoring, especially when they worry about notification noise or floods. The skill adapts the shipped samples.nyctaxi.trips reference to the user's pipelines (the same hook code works unchanged across any number of pipelines), keeps the hook on the supported notification surface (print and HTTP, never spark.sql writes; durable throttle state via UC Volume marker files), wires the backstop alert to the user's event log and recipients, and verifies the result with read-only queries. It proposes changes and confirms with the user; it is not an auto-rewriter. +description: Use this skill to add per-expectation data-quality notifications to the user's own Lakeflow Spark Declarative Pipeline (SDP). The pattern is a pair, an in-pipeline event hook that notifies the moment an expectation result with failures is logged (fast, best-effort, throttled to at most one notification per expectation per window) plus exactly one DABs-managed Databricks SQL alert that sweeps the published pipeline event log on a schedule (guaranteed, one email per state change). Use it whenever the user wants to be notified when expectations fail, wants alerting on WARN expectations that would otherwise only sit in the event log, or asks for event-hook-based monitoring, especially when they worry about notification noise or floods. The skill adapts the shipped samples.nyctaxi.trips reference to the user's pipelines (the same hook code works unchanged across any number of pipelines), keeps the hook on the supported notification surface (print and HTTP, never spark.sql writes; durable throttle state via UC Volume marker files), wires the backstop alert to the user's event log and recipients, and verifies the result with read-only queries. It proposes changes and confirms with the user; it is not an auto-rewriter. --- # sdp-expectation-notifications Add per-expectation notification to the user's own Lakeflow Spark Declarative Pipeline (SDP), then prove both paths work. -The pattern: an event hook in the pipeline source filters `flow_progress` events for expectation results with `failed_records > 0`, throttles (expectation counts arrive once per microbatch, so an unthrottled hook floods on multi-batch and continuous pipelines; measured live), and notifies (driver-log print, optionally a webhook in Slack, Teams, or generic format). Because hook delivery is explicitly best-effort (the platform does not guarantee every hook runs on every event before pipeline compute terminates), one scheduled Alert v2 sweeps the published event log over a trailing time window as the guaranteed path. Hook = fast and quiet, alert = guaranteed. Neither replaces the other. +The pattern: an event hook in the pipeline source filters `flow_progress` events for expectation results with `failed_records > 0`, throttles (expectation counts arrive once per microbatch, so an unthrottled hook floods on multi-batch and continuous pipelines; measured live), and notifies (driver-log print, optionally a webhook in Slack, Teams, or generic format). Because hook delivery is explicitly best-effort (the platform does not guarantee every hook runs on every event before pipeline compute terminates), one scheduled Databricks SQL alert sweeps the published event log over a trailing time window as the guaranteed path. Hook = fast and quiet, alert = guaranteed. Neither replaces the other. This asset shipped a validated reference implementation alongside this skill. Read it first; it is the worked example you adapt away from: @@ -89,4 +89,4 @@ Official Databricks docs: 2. [Lakeflow SDP expectations](https://docs.databricks.com/aws/en/ldp/expectations) 3. [Monitor pipelines with the event log](https://docs.databricks.com/aws/en/ldp/monitor-event-logs) 4. [`event_log` table-valued function](https://docs.databricks.com/aws/en/sql/language-manual/functions/event_log) -5. [DABs alert resource (Alerts v2)](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) +5. [DABs alert resource](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) diff --git a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl index 248aebf..5aa2c98 100644 --- a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl +++ b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl @@ -8,14 +8,14 @@ Three pieces, and only the first lives inside pipeline execution: - **The hook** (`notify_on_expectation_violation` in `{{.target_dir}}/expectation_notifications_pipeline.py`): a `@dp.on_event_hook(max_allowable_consecutive_failures=None)` function that ignores everything except `flow_progress` events, walks `details.flow_progress.data_quality.expectations`, throttles via `_should_notify` (two time-aware layers: in-memory always, volume marker files when `dq_notify.state_dir` is set), and notifies for each entry with `failed_records > 0` that is due. Inside the hook, `event["details"]` arrives as a native dict (the `_event_details` helper keeps a defensive JSON-string branch), and `event["origin"]` carries `pipeline_name`/`pipeline_id`, which is what scopes throttle state per pipeline with no configuration. - **The event-log publication** (the `event_log` block in the pipeline resource): publishes the pipeline event log to a UC table. This is the durable record of every expectation result, whether or not the hook got to run, and it is the backstop's scan target. -- **The backstop alert** (`resources/_backstop.alert.yml`): one Alert v2 whose query sums `failed_records` from `flow_progress` events over a trailing window, with `comparison_operator: GREATER_THAN` and threshold `0`, its own quartz cron schedule, and email subscriptions. +- **The backstop alert** (`resources/_backstop.alert.yml`): one Databricks SQL alert whose query sums `failed_records` from `flow_progress` events over a trailing window, with `comparison_operator: GREATER_THAN` and threshold `0`, its own quartz cron schedule, and email subscriptions. ## What to change for the user's pipeline 1. **The hook function.** Copy it into the user's pipeline source close to verbatim (helpers included: `_should_notify`, `_sanitize`, `_payload`, `_event_details`); it needs no per-expectation configuration because it reacts to every expectation the pipeline declares, current and future, and no per-pipeline configuration because state scopes from the event origin. Decisions to confirm with the user: - Throttle window: `dq_notify.throttle_seconds` (default 3600) is the "at most one notification per expectation per" window. `0` disables throttling (one notification per failing expectation per microbatch; only sensible for pipelines that process exactly one batch per run and never full-refresh). - Durable state: `dq_notify.state_dir` empty keeps the throttle in-memory (kills repeats within an update and within a continuous run; forgets across updates). A path under an EXISTING UC Volume adds cross-update throttling via marker files (`/dq_notify_state//__.json`, overwritten in place, human-readable). Never create a volume for this; ask which existing volume to use. Multiple pipelines share one state root safely. - - Notification surface: print only, or also a webhook, and which payload format (`dq_notify.channel_format`: `slack` validated live; `teams` documented Adaptive Card envelope, not live-tested; `generic` plain JSON). Keep the timeout short (the reference uses 5s) and no retries; hooks run one at a time and the post-update grace budget is seconds. + - Notification surface: print only, or also a webhook, and which payload format (`dq_notify.channel_format`: `slack` validated live; `teams` documented Adaptive Card envelope, not live-tested; `generic` plain JSON). Keep the timeout short (the reference uses 5s) and no retries; hooks run one at a time and the post-update grace budget is seconds. If the user has no Slack, Teams, or webhook target, propose skipping the hook entirely rather than a print-only deployment: print-only output has no advantage over the published event log the backstop already sweeps, so the hook's only edge (latency) goes unused. Only wire the hook when there is somewhere for it to deliver. - Secrets: webhook URLs are credentials. Put the URL in a secret scope and set `dq_notify.webhook_secret_scope`/`dq_notify.webhook_secret_key`; the reference resolves it via `dbutils.secrets.get` at module level (works in serverless SDP Python, validated live) and degrades to print-only on a missing scope. Two traps, both observed live: `{{"{{secrets/scope/key}}"}}` in pipeline configuration is NOT interpolated (the literal string arrives), and the plain `dq_notify.webhook_url` config is a demo-only convenience. Setup commands: `databricks secrets create-scope ` then `databricks secrets put-secret --string-value ""` (run by the user so the credential stays out of your transcript). - Filtering: if the user only wants certain expectations or datasets to notify, filter on `exp["name"]` / `exp["dataset"]` inside the loop. Keep the default broad; per-rule routing is the alert layer's job if it grows complicated. - `max_allowable_consecutive_failures`: keep `None` unless the user prefers auto-disable semantics; explain the tradeoff (finite = a flaky endpoint silently disables the hook until the next pipeline restart, visible as FAILED then DISABLED rows in `hook_progress`; `None` = a broken hook fails forever but keeps trying). The reference catches every exception and prints, so the hook itself never counts as failed. @@ -23,9 +23,9 @@ Three pieces, and only the first lives inside pipeline execution: 3. **Event-log publication.** If the user's pipeline resource has no `event_log` block, add one (catalog, schema, table name; pick a name coupled to the pipeline to avoid collisions in a shared schema). If the pipeline already publishes an event log, reuse the existing table and do not rename it: downstream consumers may already query it. 4. **The backstop alert.** Adapt the reference resource: - Point the query's `FROM` at the user's published event log FQN. - - **Window and cadence are coupled.** The query aggregates a trailing `INTERVAL`; the schedule is a quartz cron. Keep the window at least as long as the cadence, otherwise violations can fall between sweeps. The reference pairs a 1-day window with a daily 06:00 UTC cron. + - **Window and cadence are coupled, and cadence should match the pipeline's mode.** The query aggregates a trailing `INTERVAL`; the schedule is a quartz cron. Keep the window at least as long as the cadence, otherwise violations can fall between sweeps. The reference pairs a 1-day window with a daily 06:00 UTC cron, which fits a triggered pipeline; a continuous pipeline needs both tightened (Databricks positions continuous mode for freshness between 10 seconds and a few minutes, so a daily sweep would add a full day of latency on top of that). A tighter cron means more warehouse wake-ups; point the user at the `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) if that tradeoff matters to them. - Warehouse: reuse a warehouse the user already runs alerts on, or point at the `monitoring-sql-warehouse` asset's resource (`${resources.sql_warehouses.monitoring_sql_warehouse.id}`) if installed. The sweep is sub-second; a small serverless warehouse with a short auto-stop is the cost-correct shape. - - Subscriptions: `user_email` entries must belong to workspace users; a `destination_id` references a workspace notification destination. + - Subscriptions: `user_email` entries must belong to workspace users; a `destination_id` references a workspace notification destination. A destination is a wrapper around a webhook (Slack needs a webhook URL, OAuth token, and channel id; Teams needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID), not a substitute for one; EMAIL is the only type taking plain addresses with no webhook. If you create a destination on the user's behalf via `databricks notification-destinations create`, do not treat its own output as a failure signal: the command prints `Warning: unknown field: destination_type` and echoes back a config that looks empty (for example `"config": {"email": {}}`) even when the destination was created correctly; confirm with a follow-up `databricks notification-destinations get `, which shows the real config. - Scope: one backstop per event log table is the default shape. Multiple pipelines publishing to separate event logs need separate alerts (or a UNION ALL query, which couples their lifecycles; prefer separate). 5. **Bundle integration.** Confirm an `include:` glob picks up the resource files, and reuse the user's existing bundle variables where they exist (for example `catalog: ${var.catalog}` instead of a hardcoded catalog). Do not assume the default layout. diff --git a/pyproject.toml b/pyproject.toml index 20d68d1..ad2d442 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "databricks-bundle-template" -version = "1.12.0" +version = "1.12.1" description = "A custom Declarative Automation Bundles template for production-ready, multi-environment data pipeline projects" readme = "README.md" license = {text = "MIT"} diff --git a/template/{{.project_name}}/bundle_init_config.json.tmpl b/template/{{.project_name}}/bundle_init_config.json.tmpl index 0f09cdf..fc37070 100644 --- a/template/{{.project_name}}/bundle_init_config.json.tmpl +++ b/template/{{.project_name}}/bundle_init_config.json.tmpl @@ -1,5 +1,5 @@ { - "_template_version": "1.12.0", + "_template_version": "1.12.1", "_comment": "Configuration values used to generate this project with 'databricks bundle init'", "project_name": "{{.project_name}}", "environment_setup": "{{.environment_setup}}",