diff --git a/CHANGELOG.md b/CHANGELOG.md index 0317392..dc8f18f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,13 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [1.12.2] - 2026-08-23 + +### Changed +- **Asset `sdp-expectation-notifications` (docs only)**: nine corrections and additions grounded in a live verification session (2026-08-22, serverless SDP, CLI v0.297.2), none changing behavior. + - **Destination config corrected; this supersedes the 1.12.1 bullet below.** For a Databricks SQL alert, a Slack destination takes the incoming webhook URL and nothing else, and so does MS Teams. 1.12.1 said Slack also needs an OAuth token and a channel id, and Teams an App ID, Auth Secret, Channel URL, and Tenant ID; those fields are marked `(optional)` on the Add-destination form and scoped by its own section heading to dashboard subscriptions, a different feature. Confirmed on a Slack destination that delivered five messages while its stored config read `url_set: true, oauth_token_set: false, channel_id_set: false`. Fixed in the asset README, the backstop alert resource comment, and the companion skill. + - **The hook email claim re-scoped.** There is no Databricks-native email surface inside a hook, but the hook reaches arbitrary HTTPS: the shipped, unmodified code POSTed to two unrelated third-party collectors, HTTP 200. Two limits travel with that claim wherever it appears: the shipped `requests.post` passes no `headers`, so an authenticated API needs the hook edited first, and the result was measured with no egress controls configured, so in a governed workspace the network policy decides. The no-webhook guidance was broadened to match: the test is any endpoint the hook can post to, not a chat integration specifically. + - **`generic` channel format is live-tested**: all six fields arrived verbatim, the two counts as JSON numbers. `teams` remains documented and not live-tested. + - **`Empty result state: Error` documented where the sweep query invites edits.** An alert deployed from this asset's resource flips to ERROR if its query returns no rows, and ERROR emails on every evaluation; the shipped `COALESCE(SUM(...), 0)` guarantees exactly one row, so keep that property when editing the query. + - **Measured numbers replace asserted ones**: 116 ms per webhook POST on success and 3 ms to fail on an unresolvable name (instrumented), a real `WEBHOOK_DELIVERY_FAILED` instance instead of a described shape, and the alert History panel's `Notified destinations` column as the product's own evidence for one notification per state transition. + - **New: one violation, three message shapes.** The hook payload carries the dataset, expectation, and exact counts; the backstop email carries the evaluated metric against its threshold plus a five-run history; a Slack destination carries a state-change title and two links and no data. The destination type therefore also decides how much a responder can triage without clicking. + - **Workspace admin is required to manage notification destinations**, stated as documented rather than as stress-tested. + - **The companion skill gains the CLI delete behavior**: a delete takes effect immediately but `list` is eventually consistent, so verify with `get`, and a redundant delete errors rather than being idempotent. The create quirk it already documents still reproduces on CLI v0.297.2. + - **Stale comment fixed** in `event_log_queries.sql`: its `hook_progress` note still claimed enable/disable state only, which 1.12.0 had corrected everywhere else. + ## [1.12.1] - 2026-07-26 ### Changed - **Asset `sdp-expectation-notifications` (docs only)**: seven corrections grounded in two live verification sessions (2026-07-26 drop/destination testing and the 2026-07-17/18 hygiene build), none changing behavior. - **Drop coverage documented.** The hook filters purely on `failed_records > 0` and never inspects the expectation's action type, so `warn` and `drop` rules notify identically; the README previously framed the pattern as WARN-only. Live 2026-07-26: a `drop` expectation on the same source and tripwire produced the same passed/failed shape as the shipped WARN row (18929/3003 on 21,932 rows), the hook fired with a matching driver-log line, the update reported `COMPLETED` while discarding 13.7% of rows, `dropped_records` (DROP-only) read 3003 vs 0 on WARN, and the backstop's sweep summed both expectations additively (6,006). - **Naming corrected.** "Alert v2" and "Alerts v2" replaced with "Databricks SQL alert(s)" throughout the README, in-bundle doc, skill files, and install prompt text, matching current Databricks docs (legacy alerts is the prior version); `alerts-v2` is kept only where it names an actual CLI command. - - **Notification destinations clarified.** Destinations serve only Databricks SQL and jobs, not the event hook, and a destination is a wrapper around a webhook, not a substitute for one (Slack needs a webhook URL, OAuth token, and channel id plus Slack app-install rights; Teams needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID plus Microsoft Copilot Studio and Entra ID permissions). Measured live: a destination alongside `user_email` on the same address sends two independent, non-deduplicated notifications. + - **Notification destinations clarified.** Destinations serve only Databricks SQL and jobs, not the event hook, and a destination is a wrapper around a webhook, not a substitute for one (Slack needs a webhook URL, OAuth token, and channel id plus Slack app-install rights; Teams needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID plus Microsoft Copilot Studio and Entra ID permissions). Measured live: a destination alongside `user_email` on the same address sends two independent, non-deduplicated notifications. *[The destination-config detail in this bullet was corrected in 1.12.2: for an alert, Slack and Teams destinations take the webhook URL alone, and the extra fields are optional and belong to dashboard subscriptions. The rest of the bullet stands.]* - **No-webhook guidance added.** Readers and the companion skill are now steered to deploy the backstop alone when there is no Slack, Teams, or webhook target, since print-only hook output has no advantage over the event log the backstop already sweeps. - **Backstop cron tied to pipeline mode.** The shipped daily cron fits a triggered pipeline; guidance now flags that a continuous pipeline (10-seconds-to-a-few-minutes freshness) needs a tighter cron and window, at the cost of more warehouse wake-ups, pointing at the `monitoring-sql-warehouse` asset. - **Grace-period claim softened.** "Terminates compute within roughly 20 seconds" overclaimed precision two runs never measured; replaced with the honest bound (compute alive at ~8s in one run, dead before ~23s in another, neither designed to measure the timer) and the actionable rule: keep hook work in single-digit seconds. diff --git a/assets/sdp-expectation-notifications/README.md b/assets/sdp-expectation-notifications/README.md index 939a515..fecedf8 100644 --- a/assets/sdp-expectation-notifications/README.md +++ b/assets/sdp-expectation-notifications/README.md @@ -61,21 +61,34 @@ An update that processes no new rows emits no expectation events at all (measure The two lanes get channels in two different ways: -- **Backstop (email and beyond, zero code):** the alert subscribes an email by default. Notification destinations can only be configured for Databricks SQL and jobs, so a destination is how Slack, MS Teams, PagerDuty, or a generic webhook attach to the backstop: create a workspace notification destination (admin settings) and reference it via the commented `destination_id` line in the alert YAML. A destination is a wrapper around a webhook, not a substitute for one: a Slack destination's config is a webhook URL plus an OAuth token and channel id, requiring app-install rights in the Slack workspace; a Teams destination needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID, requiring permissions in Microsoft Copilot Studio and Entra ID on top of Databricks workspace admin. EMAIL is the only destination type taking plain addresses with no webhook. The honest benefit is centralizing the credential once at workspace level instead of pasting it into every alert, not "Slack without a webhook." Also worth knowing (measured live 2026-07-26): a destination and a `user_email` pointing at the same address produce two independent, non-deduplicated sends per trigger, not a merged one. -- **Hook (webhook formats):** `dq_notify.channel_format` selects the payload: `slack` (`{"text": ...}`, validated end to end: secret scope to channel message, HTTP 200), `teams` (Workflows Adaptive Card envelope, documented format, not live-tested by this asset), or `generic` (plain JSON with the violation fields). Route hook posts to chat-style channels that tolerate an occasional repeat; email belongs to the backstop, and a hook cannot send email anyway (no email surface exists inside a hook). +- **Backstop (email and beyond, zero code):** the alert subscribes an email by default. Notification destinations can only be configured for Databricks SQL and jobs, so a destination is how Slack, MS Teams, PagerDuty, or a generic webhook attach to the backstop: create a workspace notification destination (Settings > Notifications; the docs state you must be a workspace admin to manage them) and reference it via the commented `destination_id` line in the alert YAML. A destination is a wrapper around a webhook, not a substitute for one, but for an alert it is a thin one: a Slack destination takes the incoming webhook URL and nothing else, and so does Teams. The OAuth token, channel id, App ID, Auth Secret, Channel URL, and Tenant ID fields exist on the form, are marked `(optional)`, and the form's own section heading scopes them to dashboard subscriptions, a different feature; verified live 2026-08-22 on a Slack destination that delivered five messages while its stored config read `url_set: true, oauth_token_set: false, channel_id_set: false`. EMAIL is the only destination type taking plain addresses with no webhook. Minting the Slack webhook is still a Slack app install (per Slack's own docs: create an app, enable incoming webhooks, authorize it into the workspace), so the credential still has to exist; what a destination buys is centralizing it once at workspace level instead of pasting it into every alert. Also worth knowing (measured live 2026-07-26): a destination and a `user_email` pointing at the same address produce two independent, non-deduplicated sends per trigger, not a merged one. +- **Hook (webhook formats):** `dq_notify.channel_format` selects the payload: `slack` (`{"text": ...}`, validated end to end: secret scope to channel message, HTTP 200), `teams` (Workflows Adaptive Card envelope, documented format, not live-tested by this asset), or `generic` (plain JSON with the violation fields; live-tested 2026-08-22, all six fields arriving verbatim and correctly typed at a third-party collector). Route hook posts to chat-style channels that tolerate an occasional repeat; email belongs to the backstop. There is no Databricks-native email surface inside a hook, which is the accurate form of that limit: the hook does reach arbitrary HTTPS (lived 2026-08-22, shipped code, two unrelated third-party hosts), so an email API is reachable in principle, but the shipped `requests.post` passes no `headers` argument, so an authenticated email API needs the hook modified first. And the egress result was measured on Free Edition serverless SDP with no egress controls configured: in a workspace with an egress firewall, a network policy, or private-link networking, the network policy decides, not the hook. -**No webhook available? Deploy the backstop alone.** Without a delivery surface, print-only hook output has no advantage over the published event log the backstop already sweeps; the hook's only edge is latency, and that's worthless with nowhere to deliver it. If you have no Slack, Teams, or webhook target, skip the hook and ship just the backstop alert. +**No endpoint to deliver to? Deploy the backstop alone.** Without a delivery surface, print-only hook output has no advantage over the published event log the backstop already sweeps; the hook's only edge is latency, and that's worthless with nowhere to deliver it. The test is broader than a chat integration: anything the hook can POST to counts, whether that is a Slack or Teams webhook, a receiver you control, or another reachable HTTPS service, subject to the two limits above (no `headers` on the shipped call, and your network policy). If nothing qualifies, skip the hook and ship just the backstop alert. **Webhook URLs are credentials** (a Slack incoming webhook grants posting rights). Real setups put the URL in a secret scope and set `dq_notify.webhook_secret_scope`/`dq_notify.webhook_secret_key`; the pipeline resolves it via `dbutils.secrets.get` at startup, which works at module level in serverless SDP Python (validated live), and degrades to print-only if the scope is missing (also validated). Two things that do NOT work, so you do not have to rediscover them: `{{secrets/scope/key}}` in pipeline configuration is not interpolated (the literal string arrives), and the plain `dq_notify.webhook_url` config is a demo-only convenience. +### One violation, three message shapes + +Same violation, same moment, three different payloads (measured live 2026-08-22). Choosing a delivery lane, and a destination type inside it, is also choosing how much a responder can triage before clicking through. + +| Lane | What the recipient sees without clicking | +|---|---| +| Hook webhook (fast) | dataset, expectation name, exact `failed`/`passed` counts, at violation time | +| Backstop email (guaranteed) | the evaluated metric against its threshold (for example `3003.0 > 0.0`), plus an embedded five-run mini history | +| Backstop Slack destination (guaranteed) | a state-change title and two deep links, no data at all | + +The hook's message is the payload; the email is a summary with numbers; the destination message is a pointer that says go look. + ## The rules of the hook (learned the honest way) -Established on live serverless SDP runs (2026-07-11 and 2026-07-17/18, CLI v0.297.2): +Established on live serverless SDP runs (2026-07-11, 2026-07-17/18, and 2026-08-22, CLI v0.297.2): - **Do not write Delta from inside a hook.** `spark.sql()` in SDP pipeline Python is restricted to a read allowlist: `INSERT INTO` fails with `UNSUPPORTED_SPARK_SQL_COMMAND` (the error lists the allowlist: SELECT, DESCRIBE, SHOW variants, USE), `CREATE TABLE` fails the same way, and the SDP-patched `spark.sql()` rejects the parameterized-query `args=` kwarg. All three write paths are closed; this is definitive. - **Durable state goes through UC Volume file I/O instead.** Plain Python `open()` against `/Volumes/...` works from inside a hook: create, overwrite, read, `os.path.exists`, `os.listdir`, `os.stat`, `os.makedirs` all validated live. The one gap: append to an existing file fails (`OSError [Errno 29] Illegal seek`), which is why the markers are overwrite-only. - **Reads are fine.** `spark.sql("SELECT ...").collect()` works reliably from a hook (validated across many invocations), so a hook can consult a small gate table if you need a manual mute switch. -- **Keep the hook fast; the grace budget is seconds.** Hooks run serialized, and compute is torn down shortly after an update completes. A hook needing about 1s per event drained 5 queued events with zero loss, all within about 8 seconds of completion. A hook needing 20s per event had its first event start 3 seconds after completion, never finish, and lost **6 of 6** notifications under normal, non-forced teardown. Keep total hook work in single-digit seconds. No retries, tight webhook timeout (5s), marker I/O only (~100ms). The exact platform timer is not documented and we have not established it. +- **The hook reaches arbitrary HTTPS, and your network policy is what limits it.** Lived 2026-08-22: the shipped, unmodified hook POSTed to two unrelated third-party collectors (different operators), HTTP 200, the request arriving bare at one of them with no proxy header, no policy header, and no interception signature. That was Free Edition serverless SDP with no egress controls configured; a workspace with an egress firewall, a network policy, or private-link networking may behave differently, and that could not be tested here. Read the two halves together: the hook itself imposes no restriction on outbound HTTPS, and in a governed workspace the network policy is what decides. +- **Keep the hook fast; the grace budget is seconds.** Hooks run serialized, and compute is torn down shortly after an update completes. A hook needing about 1s per event drained 5 queued events with zero loss, all within about 8 seconds of completion. A hook needing 20s per event had its first event start 3 seconds after completion, never finish, and lost **6 of 6** notifications under normal, non-forced teardown. Keep total hook work in single-digit seconds. No retries, tight webhook timeout (5s), marker I/O only (~100ms); the POST itself measured 116 ms on success and 3 ms to fail on an unresolvable name (instrumented, 2026-08-22), so one call per event is comfortably affordable and unbounded fan-out is what is not. The exact platform timer is not documented and we have not established it. - **`hook_progress` records ENABLED, FAILED, and DISABLED transitions** (correcting v1.11.0, which claimed enable-state only). Hook health is event-log-observable and alertable. It still does not record per-invocation output; the printed notifications exist only in the pipeline compute's driver log, which has no CLI or SQL surface. - **`max_allowable_consecutive_failures` is a tradeoff.** A finite value disables a flaky hook until the next restart (visible as `DISABLED` in `hook_progress`, but notifications silently stop); `None` (this asset's choice) lets a broken hook fail forever, which is why every failure path in the shipped hook prints instead of raising. - **`mode: development` masks the teardown behavior.** A DABs development-mode target keeps pipeline compute warm across updates, hiding both the delivery gap and the fresh-process-per-update behavior (in non-dev mode each update gets a new Python process; measured via per-process ids). @@ -84,10 +97,11 @@ Established on live serverless SDP runs (2026-07-11 and 2026-07-17/18, CLI v0.29 ## The backstop's notification behavior (measured) -- **Default: exactly one email per state transition.** Nine consecutive TRIGGERED evaluations on an every-minute cron produced exactly one email; the state change is what notifies, not the evaluation. The masking consequence: while the alert stays TRIGGERED, new failures inside the sweep window send nothing, which is why `notify_on_ok: true` ships as the default: the single recovery email (also measured: exactly one) closes the loop and re-arms attention. +- **Default: exactly one email per state transition.** Nine consecutive TRIGGERED evaluations on an every-minute cron produced exactly one email; the state change is what notifies, not the evaluation. The masking consequence: while the alert stays TRIGGERED, new failures inside the sweep window send nothing, which is why `notify_on_ok: true` ships as the default: the single recovery email (also measured: exactly one) closes the loop and re-arms attention. Since 2026-08-22 this is displayed rather than inferred: the alert's History panel carries a `Notified destinations` column, and it is empty on evaluations that did not change state. One reading trap in the email itself: its embedded `Last 5 runs` block lists evaluations, not notifications, so rows that notified nobody appear in it. - **`retrigger_seconds` is deliberate re-nagging:** re-sends at the first evaluation past each window (measured with 120s: gaps of 120s, 180s, 179s, aligned to evaluation ticks). Off by default; the commented block suggests 86400 as a daily "still broken" escalation for critical pipelines. - **ERROR does not throttle.** If the sweep query itself fails, the alert flips to ERROR and emails on **every evaluation** until fixed (measured: 5 emails in 4 minutes). The backstop cannot die silently; treat an `(ERROR)` email as a page. - The sweep query aggregates over a past-time window (trailing 1 day), not just the latest row; keep the window at least as long as the cadence if you change the cron. +- **Do not edit the sweep query into a shape that can return zero rows.** The alert's Advanced panel reads `Empty result state: Error` on an alert deployed from this asset's resource, which sets no such field: no rows means ERROR, and ERROR emails on every evaluation. The shipped query wraps its aggregate in `COALESCE(SUM(...), 0)`, so it always returns exactly one row and cannot hit this. A `GROUP BY`, a `HAVING`, or a filter that can match nothing silently converts the backstop into a per-evaluation ERROR emailer. - **Cron cadence should match the pipeline's mode.** The shipped daily 06:00 UTC cron, paired with a 1-day window, fits a triggered pipeline on a daily-or-slower cadence. A continuous pipeline is a different shape: Databricks positions continuous mode for freshness requirements between 10 seconds and a few minutes, so a daily sweep adds a full day of latency on top of that. Tighten both the cron and the `INTERVAL` together if the pipeline runs continuously, but tightening costs more warehouse wake-ups; the `monitoring-sql-warehouse` asset in this repo (2X-Small serverless, `auto_stop_mins: 1`) is sized for exactly that tradeoff. ## Validation results @@ -101,13 +115,24 @@ v1.11.0's pattern was validated end to end on 2026-07-11 (hook fires with exact - Databricks SQL alerts: one email per state transition (9 TRIGGERED evaluations, 1 email), `retrigger_seconds` re-send alignment, exactly one `notify_on_ok` recovery email, ERROR emailing every evaluation. - Drop-expectation coverage, live 2026-07-26: a `drop` expectation on the same source and tripwire condition produced the same passed/failed shape as the WARN row (18929 passed / 3003 failed on 21,932 total rows), the hook fired with an exact-matching driver-log line, the update reported `COMPLETED` while discarding 3,003 of 21,932 rows (13.7%), `dropped_records` read 3003 on the DROP row versus 0 on the WARN row, and the backstop's sweep query summed both expectations with no special-casing (6,006 = 3,003 + 3,003). +The 2026-08-22 session (same workspace and CLI, non-development-mode target) added: + +- Arbitrary HTTPS egress from the hook: the shipped, unmodified code delivered to two unrelated third-party collectors, HTTP 200, one capture bare (no proxy or policy header). Limit: that workspace had no egress controls configured, so in a governed workspace the network policy is what decides. +- The `generic` payload live-tested: all six fields (`text`, `pipeline`, `dataset`, `expectation`, `failed_records`, `passed_records`) arrived verbatim, the two counts as JSON numbers. +- Per-event webhook cost, instrumented: 116 ms on success, 3 ms to fail on an unresolvable name. +- The delivery-failure line with a real instance rather than a described shape: `WEBHOOK_DELIVERY_FAILED: ConnectionError: HTTPSConnectionPool(...) ... NameResolutionError ... [Errno -2] Name or service not known`. The class is `requests.exceptions.ConnectionError` (`type(e).__name__` renders it bare, which collides with the Python builtin's name). The violation line printed first, the update still reported `COMPLETED`, and the hook was not disabled. +- A Slack notification destination created from the incoming webhook URL alone, delivering five messages, one per state change; its stored config reads `url_set: true, oauth_token_set: false, channel_id_set: false`. +- One notification per state transition shown by the product rather than inferred from counting emails: the History panel's `Notified destinations` column is empty on evaluations that did not change state. + ## Honest limits and open questions - Hook delivery is best-effort by design; the throttle makes the fast lane quiet, not reliable. The backstop remains the guaranteed path. - The two lanes de-duplicate independently and share no state: one violation can produce one hook message and one backstop email. Different audiences, different guarantees; by design. - The hook cannot notice "the pipeline has not run at all": no update, no events, no hook. Pair with a freshness check (for example Unity Catalog data quality monitoring's anomaly detection) if that failure mode matters. - The backstop inherits scheduled-scan limits: latency bounded by its cron cadence, and a SQL warehouse in the loop. The `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) exists for exactly this workload shape. -- Still open: the Teams payload ships doc-confirmed, not live-tested; concurrent marker writes from two simultaneously running pipelines are untested (the per-pipeline folders give them no shared file to race on, but Free Edition cannot run two updates at once to prove it); the exact numeric grace period remains unestablished, two runs bound it loosely (one showed compute still alive at ~8s, one showed it dead before ~23s), but neither run was designed to measure the timer itself. +- **Still open, and the one that matters most: whether hook egress is restricted in a governed workspace.** The arbitrary-HTTPS result was measured on Free Edition with no egress controls configured; an egress firewall, a network policy, or private-link networking could not be tested from there, and in a workspace that has them the network policy is what decides. +- Also still open on the hook: the Teams payload ships doc-confirmed, not live-tested; concurrent marker writes from two simultaneously running pipelines are untested (the per-pipeline folders give them no shared file to race on, but Free Edition cannot run two updates at once to prove it); and the exact numeric grace period remains unestablished, two runs bounding it loosely (one showed compute still alive at ~8s, one showed it dead before ~23s) without either being designed to measure the timer itself. +- Still open on destinations: whether managing them genuinely requires workspace admin (documented, but the verification account owns that workspace, so no blocked case was exercised), and MS Teams and PagerDuty delivery (their form field lists are lived, their delivery is not). ## Inspecting the results diff --git a/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md b/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md index ac77151..0d1c580 100644 --- a/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md +++ b/assets/sdp-expectation-notifications/template/docs/sdp-expectation-notifications/README.md @@ -43,13 +43,13 @@ The first update ingests the full sample table, so the demo WARN expectation (`d - **The durable record** is the published event log table `..dq_notifications_event_log`. Run the first query in `/event_log_queries.sql` to see per-expectation passed/failed counts for the latest update; they match the numbers in the hook's printed line. - **Hook registration state** is the second query (`hook_progress` events). It records ENABLED, FAILED, and DISABLED transitions (so a failing hook is visible here), but not per-invocation execution. - **Throttle state** (when `dq_notify.state_dir` is set) is browsable: `/dq_notify_state//`, one human-readable JSON marker per expectation with the last-notified time and counts, overwritten in place. Expect one notification per expectation per `dq_notify.throttle_seconds` window, not one per microbatch; a rerun inside the window notifies nothing, by design. -- **The backstop alert** evaluates daily at 06:00 UTC by default. To see it immediately, open the alert in the workspace UI (SQL > Alerts) and run it manually, or temporarily tighten `quartz_cron_schedule`. On trigger it emails the configured subscription from `noreply@databricks.com`. The third query in `event_log_queries.sql` is the alert's exact sweep query, so you can preview the evaluated value. +- **The backstop alert** evaluates daily at 06:00 UTC by default. To see it immediately, open the alert in the workspace UI (SQL > Alerts) and run it manually, or temporarily tighten `quartz_cron_schedule`. On trigger it emails the configured subscription from `noreply@databricks.com`, subject `Databricks SQL Alert: ()`, so the state is filterable from the subject line alone; the body carries the evaluated metric against its threshold plus a five-run mini history (which lists evaluations, not notifications, so rows that notified nobody appear in it). The third query in `event_log_queries.sql` is the alert's exact sweep query, so you can preview the evaluated value. ## Optional webhook (Slack, Teams, or generic) The hook posts each due notification to a webhook when one is configured, in the format selected by `dq_notify.channel_format` (`slack`, `teams`, or `generic`). Route hook posts to chat-style channels; email belongs to the backstop alert. -No Slack, Teams, or generic webhook target available? Skip this section and the hook's webhook config entirely: print-only output has no advantage over the event log the backstop already sweeps, so the backstop alone covers you (see the asset README's "Notification channels" section for the full reasoning). +No endpoint for the hook to post to (a Slack or Teams webhook, a receiver you control, any reachable HTTPS service)? Skip this section and the hook's webhook config entirely: print-only output has no advantage over the event log the backstop already sweeps, so the backstop alone covers you (see the asset README's "Notification channels" section for the full reasoning). A webhook URL is a credential (it grants posting rights), so the real setup goes through a secret scope: @@ -89,8 +89,8 @@ then apply them and guide me through (or do) the deploy and verification. - **Cadence and window are coupled, and cadence should match your pipeline's mode.** The alert query aggregates the trailing `INTERVAL 1 DAY`; the schedule is daily. If you change the cron, change the interval to stay at least as long as the cadence, or violations can fall between sweeps. The shipped daily cadence fits a triggered pipeline; a continuous pipeline needs a tighter cron and window (Databricks positions continuous mode for freshness between 10 seconds and a few minutes, so a daily sweep would add a full day of latency on top of that), but a tighter cron means more warehouse wake-ups: the `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) is sized for that tradeoff. - **Notification behavior (measured live):** one email per state transition, however many evaluations stay TRIGGERED; `notify_on_ok: true` (the shipped default) adds exactly one recovery email; the commented `retrigger_seconds` re-sends while TRIGGERED at most once per window, a deliberate escalation knob (for example 86400 for a daily reminder on a critical pipeline). While the alert stays TRIGGERED, new failures inside the window send nothing; the recovery email is what re-arms attention. - **If the sweep query itself breaks** (dropped table, deleted warehouse), the alert flips to ERROR and emails on every evaluation until fixed. It cannot die silently; treat an `(ERROR)` email as a page. -- **Recipients** live in `evaluation.notification.subscriptions` (add more `user_email` entries, or a `destination_id` for a workspace notification destination, which is how Slack, MS Teams, PagerDuty, and generic webhooks attach to the backstop with zero code). -- **Threshold** is `failed_records > 0`. Raise the threshold or filter the query to specific expectation names if the demo tripwire pattern is too chatty for your rules. +- **Recipients** live in `evaluation.notification.subscriptions` (add more `user_email` entries, or a `destination_id` for a workspace notification destination, which is how Slack, MS Teams, PagerDuty, and generic webhooks attach to the backstop with zero code; managing destinations requires workspace admin, per the docs). Worth knowing before you pick one: a Slack destination's message carries a state-change title and two links and no data, where the email carries the evaluated metric against its threshold, so the destination type also decides how much a responder can triage without clicking. +- **Threshold** is `failed_records > 0`. Raise the threshold or filter the query to specific expectation names if the demo tripwire pattern is too chatty for your rules. If you edit the query, keep it returning exactly one row: the shipped `COALESCE(SUM(...), 0)` guarantees that, and an alert whose query returns no rows flips to ERROR (`Empty result state: Error`), which emails on every evaluation until fixed. ## References diff --git a/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl b/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl index 7f6b33a..dc612ae 100644 --- a/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl +++ b/assets/sdp-expectation-notifications/template/resources/{{.pipeline_resource_key}}_backstop.alert.yml.tmpl @@ -15,6 +15,14 @@ # continuous pipeline needs both tightened to match its faster freshness # (see the asset README's backstop notification-behavior section). # +# The aggregate is wrapped in COALESCE(SUM(...), 0) on purpose: an alert +# deployed from this file reads `Empty result state: Error` in its Advanced +# panel (nothing here sets it), so a query returning no rows flips the +# alert to ERROR, and ERROR emails on EVERY evaluation until fixed. +# COALESCE guarantees exactly one row, always. If you edit the query, keep +# that property: a GROUP BY, a HAVING, or a filter that can match nothing +# turns this backstop into a per-evaluation ERROR emailer. +# # Placeholders: if warehouse_id or the subscription email below reads # *_PLACEHOLDER, replace it before deploying. The monitoring-sql-warehouse # asset from this same repo is a natural warehouse for this workload @@ -60,18 +68,25 @@ resources: subscriptions: - user_email: {{.notification_email}} # Slack / MS Teams / PagerDuty / webhook delivery: create a - # workspace notification destination (admin settings) and + # workspace notification destination (Settings > Notifications; + # the docs state managing them requires workspace admin) and # reference it here instead of, or alongside, the email. A # destination is a wrapper around a webhook, not a substitute for - # one: Slack needs a webhook URL, an OAuth token, and a channel - # id, plus app-install rights in Slack; Teams needs a webhook - # URL, App ID, Auth Secret, Channel URL, and Tenant ID, plus - # permissions in Microsoft Copilot Studio and Entra ID, on top of - # Databricks workspace admin. EMAIL is the only type taking plain - # addresses with no webhook. The benefit is centralizing the - # credential once at workspace level instead of per-alert. Note - # (measured live): a destination alongside user_email on the same - # address sends two independent, non-deduplicated notifications. + # one, but a thin one for an alert: Slack takes the incoming + # webhook URL and nothing else, and so does Teams. The OAuth + # token, channel id, App ID, Auth Secret, Channel URL, and Tenant + # ID fields are marked optional on the form and scoped by its own + # heading to dashboard subscriptions, a different feature + # (verified live on a delivering Slack destination: url_set true, + # oauth_token_set false, channel_id_set false). EMAIL is the only + # type taking plain addresses with no webhook. The benefit is + # centralizing the credential once at workspace level instead of + # per-alert. Two things measured live: a destination alongside + # user_email on the same address sends two independent, + # non-deduplicated notifications, and a Slack destination's + # message carries only a state-change title and two links, no + # counts, where the email carries the evaluated metric against + # its threshold. # - destination_id: # Operational note (validated live): if this alert's query itself # fails (dropped table, deleted warehouse), the alert flips to ERROR diff --git a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl index 5aa2c98..562c8ef 100644 --- a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl +++ b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/adapt-the-pattern.md.tmpl @@ -15,17 +15,17 @@ Three pieces, and only the first lives inside pipeline execution: 1. **The hook function.** Copy it into the user's pipeline source close to verbatim (helpers included: `_should_notify`, `_sanitize`, `_payload`, `_event_details`); it needs no per-expectation configuration because it reacts to every expectation the pipeline declares, current and future, and no per-pipeline configuration because state scopes from the event origin. Decisions to confirm with the user: - Throttle window: `dq_notify.throttle_seconds` (default 3600) is the "at most one notification per expectation per" window. `0` disables throttling (one notification per failing expectation per microbatch; only sensible for pipelines that process exactly one batch per run and never full-refresh). - Durable state: `dq_notify.state_dir` empty keeps the throttle in-memory (kills repeats within an update and within a continuous run; forgets across updates). A path under an EXISTING UC Volume adds cross-update throttling via marker files (`/dq_notify_state//__.json`, overwritten in place, human-readable). Never create a volume for this; ask which existing volume to use. Multiple pipelines share one state root safely. - - Notification surface: print only, or also a webhook, and which payload format (`dq_notify.channel_format`: `slack` validated live; `teams` documented Adaptive Card envelope, not live-tested; `generic` plain JSON). Keep the timeout short (the reference uses 5s) and no retries; hooks run one at a time and the post-update grace budget is seconds. If the user has no Slack, Teams, or webhook target, propose skipping the hook entirely rather than a print-only deployment: print-only output has no advantage over the published event log the backstop already sweeps, so the hook's only edge (latency) goes unused. Only wire the hook when there is somewhere for it to deliver. + - Notification surface: print only, or also a webhook, and which payload format (`dq_notify.channel_format`: `slack` validated live; `teams` documented Adaptive Card envelope, not live-tested; `generic` plain JSON, live-tested). Keep the timeout short (the reference uses 5s) and no retries; hooks run one at a time and the post-update grace budget is seconds (the reference's POST measured 116 ms on success, instrumented). The reference's `requests.post` passes no `headers`, so an endpoint requiring an Authorization header needs that call edited. If the user has no endpoint for the hook to post to, propose skipping the hook entirely rather than a print-only deployment: print-only output has no advantage over the published event log the backstop already sweeps, so the hook's only edge (latency) goes unused. Ask the broad question, not the narrow one: any reachable HTTPS receiver counts (a chat webhook, a collector they control, an internal service), subject to the two limits above, and a workspace whose network policy blocks the hook's egress counts as no endpoint. Only wire the hook when there is somewhere for it to deliver. - Secrets: webhook URLs are credentials. Put the URL in a secret scope and set `dq_notify.webhook_secret_scope`/`dq_notify.webhook_secret_key`; the reference resolves it via `dbutils.secrets.get` at module level (works in serverless SDP Python, validated live) and degrades to print-only on a missing scope. Two traps, both observed live: `{{"{{secrets/scope/key}}"}}` in pipeline configuration is NOT interpolated (the literal string arrives), and the plain `dq_notify.webhook_url` config is a demo-only convenience. Setup commands: `databricks secrets create-scope ` then `databricks secrets put-secret --string-value ""` (run by the user so the credential stays out of your transcript). - Filtering: if the user only wants certain expectations or datasets to notify, filter on `exp["name"]` / `exp["dataset"]` inside the loop. Keep the default broad; per-rule routing is the alert layer's job if it grows complicated. - `max_allowable_consecutive_failures`: keep `None` unless the user prefers auto-disable semantics; explain the tradeoff (finite = a flaky endpoint silently disables the hook until the next pipeline restart, visible as FAILED then DISABLED rows in `hook_progress`; `None` = a broken hook fails forever but keeps trying). The reference catches every exception and prints, so the hook itself never counts as failed. 2. **Python vs SQL pipelines.** Hooks are Python-only. If the user's pipeline is SQL, do not rewrite it: add one small Python file containing just the imports and the hook, and add it to the pipeline's `libraries`. The hook still sees all events of the update. 3. **Event-log publication.** If the user's pipeline resource has no `event_log` block, add one (catalog, schema, table name; pick a name coupled to the pipeline to avoid collisions in a shared schema). If the pipeline already publishes an event log, reuse the existing table and do not rename it: downstream consumers may already query it. 4. **The backstop alert.** Adapt the reference resource: - - Point the query's `FROM` at the user's published event log FQN. + - Point the query's `FROM` at the user's published event log FQN. Keep the aggregate wrapped in `COALESCE(SUM(...), 0)` so the query always returns exactly one row: an alert whose query returns no rows flips to ERROR (`Empty result state: Error`, observed on a deployed alert) and ERROR emails on every evaluation until fixed, so a `GROUP BY`, a `HAVING`, or a filter that can match nothing quietly turns the backstop into a per-evaluation ERROR emailer. - **Window and cadence are coupled, and cadence should match the pipeline's mode.** The query aggregates a trailing `INTERVAL`; the schedule is a quartz cron. Keep the window at least as long as the cadence, otherwise violations can fall between sweeps. The reference pairs a 1-day window with a daily 06:00 UTC cron, which fits a triggered pipeline; a continuous pipeline needs both tightened (Databricks positions continuous mode for freshness between 10 seconds and a few minutes, so a daily sweep would add a full day of latency on top of that). A tighter cron means more warehouse wake-ups; point the user at the `monitoring-sql-warehouse` asset (2X-Small serverless, `auto_stop_mins: 1`) if that tradeoff matters to them. - Warehouse: reuse a warehouse the user already runs alerts on, or point at the `monitoring-sql-warehouse` asset's resource (`${resources.sql_warehouses.monitoring_sql_warehouse.id}`) if installed. The sweep is sub-second; a small serverless warehouse with a short auto-stop is the cost-correct shape. - - Subscriptions: `user_email` entries must belong to workspace users; a `destination_id` references a workspace notification destination. A destination is a wrapper around a webhook (Slack needs a webhook URL, OAuth token, and channel id; Teams needs a webhook URL, App ID, Auth Secret, Channel URL, and Tenant ID), not a substitute for one; EMAIL is the only type taking plain addresses with no webhook. If you create a destination on the user's behalf via `databricks notification-destinations create`, do not treat its own output as a failure signal: the command prints `Warning: unknown field: destination_type` and echoes back a config that looks empty (for example `"config": {"email": {}}`) even when the destination was created correctly; confirm with a follow-up `databricks notification-destinations get `, which shows the real config. + - Subscriptions: `user_email` entries must belong to workspace users; a `destination_id` references a workspace notification destination (managing destinations requires workspace admin, per the docs). A destination is a wrapper around a webhook, not a substitute for one, but a thin one for an alert: Slack takes the incoming webhook URL and nothing else, and so does Teams; the OAuth token, channel id, App ID, Auth Secret, Channel URL, and Tenant ID fields on the form are marked optional and scoped to dashboard subscriptions, a different feature (verified live on a delivering Slack destination whose stored config read `url_set: true, oauth_token_set: false, channel_id_set: false`). EMAIL is the only type taking plain addresses with no webhook. If you create a destination on the user's behalf via `databricks notification-destinations create`, do not treat its own output as a failure signal: the command prints `Warning: unknown field: destination_type` and echoes back a config that looks empty (for example `"config": {"email": {}}`) even when the destination was created correctly; confirm with a follow-up `databricks notification-destinations get `, which shows the real config. Deleting has its own quirk: a delete takes effect immediately but `list` is eventually consistent (a deleted destination can still appear there for about a minute), so check with `get`, not `list`, and expect a redundant delete to fail with `Error: Webhook id does not exist` rather than being idempotent. - Scope: one backstop per event log table is the default shape. Multiple pipelines publishing to separate event logs need separate alerts (or a UNION ALL query, which couples their lifecycles; prefer separate). 5. **Bundle integration.** Confirm an `include:` glob picks up the resource files, and reuse the user's existing bundle variables where they exist (for example `catalog: ${var.catalog}` instead of a hardcoded catalog). Do not assume the default layout. diff --git a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/self-verify.md b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/self-verify.md index d0017f1..b309553 100644 --- a/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/self-verify.md +++ b/assets/sdp-expectation-notifications/template/{{.skill_dir}}/skills/sdp-expectation-notifications/references/self-verify.md @@ -48,7 +48,7 @@ PASS when every expectation the pipeline declares shows up with counts for the u The hook's output (`EXPECTATION VIOLATION | ...` lines, and any `WEBHOOK_DELIVERY_FAILED` lines) exists only in the pipeline compute's driver log. Ask the user to: open the pipeline in the workspace UI, open the update's compute, open Logs, and search for `EXPECTATION VIOLATION`. -PASS when the printed counts match check 2's counts for the same `update_id` (expect ONE `EXPECTATION VIOLATION` line per expectation per throttle window, not one per microbatch: suppressed repeats print nothing). If a webhook is configured, also confirm the message arrived at the endpoint (the user's channel or the endpoint's own log). +PASS when the printed counts match check 2's counts for the same `update_id` (expect ONE `EXPECTATION VIOLATION` line per expectation per throttle window, not one per microbatch: suppressed repeats print nothing). If a webhook is configured, also confirm the message arrived at the endpoint (the user's channel or the endpoint's own log). A delivery failure prints `WEBHOOK_DELIVERY_FAILED: : ` right after the violation line; a DNS failure looks like `ConnectionError: HTTPSConnectionPool(...) NameResolutionError ... [Errno -2] Name or service not known` (that class is `requests.exceptions.ConnectionError`). Read it as a delivery failure, not a hook failure: the violation line still printed and the update still reports `COMPLETED`. ## 3b. Throttle state markers (only when `dq_notify.state_dir` is set) @@ -72,7 +72,7 @@ If the log shows nothing despite check 2 showing failures: first suspect timing ``` PASS when `state` is `TRIGGERED` (given `failed_records > 0` in step 1) and `last_evaluated_at` is fresh. With no violations in the window, `OK` is the correct passing state; do not manufacture violations in real data to force a trigger. A persistent `ERROR` state means the query itself fails and the alert emails on EVERY evaluation until fixed (observed live); treat it as urgent, not as noise to ignore. -3. Delivery: only the recipient can confirm the email arrived (sender `noreply@databricks.com`, subject containing the alert display name and state). Expected volume, observed live: one email per state transition, plus one recovery email if `notify_on_ok` is true, plus windowed re-sends only if `retrigger_seconds` is set. Ask the user; record their answer as the delivery confirmation. +3. Delivery: only the recipient can confirm the email arrived (sender `noreply@databricks.com`, subject `Databricks SQL Alert: ()`, with the state as a literal suffix). Expected volume, observed live: one email per state transition, plus one recovery email if `notify_on_ok` is true, plus windowed re-sends only if `retrigger_seconds` is set. The alert's History panel in the UI carries a `Notified destinations` column that is empty on evaluations which did not change state, so which evaluations notified is observable without counting inboxes; note the email body's own `Last 5 runs` block lists evaluations, not notifications. Ask the user; record their answer as the delivery confirmation. ## Interpreting results honestly diff --git a/assets/sdp-expectation-notifications/template/{{.target_dir}}/event_log_queries.sql.tmpl b/assets/sdp-expectation-notifications/template/{{.target_dir}}/event_log_queries.sql.tmpl index baf64d8..db6453c 100644 --- a/assets/sdp-expectation-notifications/template/{{.target_dir}}/event_log_queries.sql.tmpl +++ b/assets/sdp-expectation-notifications/template/{{.target_dir}}/event_log_queries.sql.tmpl @@ -42,10 +42,11 @@ ORDER BY failing_records DESC, e.dataset, e.name; -- --------------------------------------------------------------------------- -- Hook registration state per update ("monitoring the monitor"). --- hook_progress events record ONLY enable/disable state, one row per hook per --- update, logged at registration time. They do NOT record which events a hook --- processed, counts, timings, or failures; per-invocation output (the printed --- notifications) exists only in the pipeline compute's driver log. +-- hook_progress events record the hook's ENABLED, FAILED, and DISABLED +-- transitions (observed live), so a failing hook is visible here. They do NOT +-- record which events a hook processed, counts, or timings; per-invocation +-- output (the printed notifications) exists only in the pipeline compute's +-- driver log. -- --------------------------------------------------------------------------- SELECT timestamp, diff --git a/assets/sdp-expectation-notifications/template/{{.target_dir}}/expectation_notifications_pipeline.py b/assets/sdp-expectation-notifications/template/{{.target_dir}}/expectation_notifications_pipeline.py index 6316994..be0bba0 100644 --- a/assets/sdp-expectation-notifications/template/{{.target_dir}}/expectation_notifications_pipeline.py +++ b/assets/sdp-expectation-notifications/template/{{.target_dir}}/expectation_notifications_pipeline.py @@ -42,6 +42,9 @@ interpolation in pipeline configuration does NOT resolve and arrives as the literal string). The plain `dq_notify.webhook_url` config is the demo-only fallback. A missing scope is caught and the hook degrades to print-only. +The POST sends JSON with no `headers` argument, so the shipped hook targets +incoming-webhook URLs, where the URL itself is the credential; an endpoint +needing an Authorization header requires editing that call. """ import json @@ -65,7 +68,8 @@ # Webhook payload format: "slack" (validated live), "teams" (documented # Workflows Adaptive Card envelope, not live-tested), or "generic" (plain JSON -# with the violation fields, for webhook receivers you control). +# with the violation fields, for webhook receivers you control; live-tested +# 2026-08-22: all six fields arrived verbatim, the two counts as JSON numbers). CHANNEL_FORMAT = spark.conf.get("dq_notify.channel_format", "slack") # Webhook resolution: secret scope first (real setups), plain config second diff --git a/pyproject.toml b/pyproject.toml index ad2d442..17466bb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "databricks-bundle-template" -version = "1.12.1" +version = "1.12.2" description = "A custom Declarative Automation Bundles template for production-ready, multi-environment data pipeline projects" readme = "README.md" license = {text = "MIT"} diff --git a/template/{{.project_name}}/bundle_init_config.json.tmpl b/template/{{.project_name}}/bundle_init_config.json.tmpl index fc37070..26a8964 100644 --- a/template/{{.project_name}}/bundle_init_config.json.tmpl +++ b/template/{{.project_name}}/bundle_init_config.json.tmpl @@ -1,5 +1,5 @@ { - "_template_version": "1.12.1", + "_template_version": "1.12.2", "_comment": "Configuration values used to generate this project with 'databricks bundle init'", "project_name": "{{.project_name}}", "environment_setup": "{{.environment_setup}}",