Throttle sdp-expectation-notifications hook and surface backstop notification controls

ASSETS.md

@@ -22,7 +22,7 @@ Run the command from the root of your bundle project. The CLI will prompt for an
 | `monitoring-sql-warehouse` | Small, dedicated serverless SQL warehouse (2X-Small, `auto_stop_mins: 1`) for scheduled Databricks Alerts and monitoring queries. Keeps cost proportional to actual query time instead of idle warm-up. | Stable | [README](assets/monitoring-sql-warehouse/README.md) |
 | `sdp-quarantine-pattern` | Lakeflow SDP pipeline demonstrating the inverse-expectations quarantine pattern on `samples.nyctaxi.trips`: critical (drop) expectations route bad rows into a separate quarantine table (silver schema), valid rows flow to silver, warn expectations log to a queryable event log. NULL-safe predicates keep the silver/quarantine split a clean partition. Ships a companion agent skill (`SKILL.md`) that adapts the pattern to your own dataset and self-verifies it. | Stable | [README](assets/sdp-quarantine-pattern/README.md) |
 | `pyspark-test-runner` | Single-file Python wrapper around `pytest` for local PySpark suites that prints a bounded, agent-friendly digest (counts, runnable failing node ids, failures deduplicated by signature) and keeps full output in a log file, so a suite that floods with repetitive failures does not burn a coding agent's context window. Ships a `SKILL.md` for agent integration. | Stable | [README](assets/pyspark-test-runner/README.md) |
-| `sdp-expectation-notifications` | Per-expectation data-quality notification for Lakeflow SDP as a validated pair: a native event hook notifies the moment a WARN expectation result is logged (fast, best-effort by platform design), and one DABs-managed Alert v2 sweeps the published pipeline event log over a past-time window on a schedule (guaranteed). Demo pipeline on `samples.nyctaxi.trips` fires both paths on the first run. Ships a companion agent skill (`SKILL.md`) that wires the pattern into your own SDP pipeline. | Stable | [README](assets/sdp-expectation-notifications/README.md) |
+| `sdp-expectation-notifications` | Per-expectation data-quality notification for Lakeflow SDP as a validated pair: a native event hook notifies the moment a WARN expectation result is logged (fast, best-effort by platform design, throttled to at most one notification per expectation per window via a two-layer time-aware de-dup with optional UC Volume marker state), and one DABs-managed Alert v2 sweeps the published pipeline event log over a past-time window on a schedule (guaranteed, one email per state change). Webhook payloads for Slack, Teams, or generic receivers, with secret-scope URL resolution. Demo pipeline on `samples.nyctaxi.trips` fires both paths on the first run. Ships a companion agent skill (`SKILL.md`) that wires the pattern into your own SDP pipelines. | Stable | [README](assets/sdp-expectation-notifications/README.md) |
 
 ## What an asset is not
 

CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Changed
+- **Asset `sdp-expectation-notifications`**: notification hygiene revision, grounded in a two-day live investigation (2026-07-17/18, serverless SDP on a non-development-mode target). The v1.11.0 hook notified on every qualifying `flow_progress` event, and the platform emits expectation counts once per microbatch, so one failing expectation notified once per file on a multi-file backlog (measured: 4 notifications for one expectation in one update, counts summing exactly to the single-batch total), re-notified everything on full refresh, and fired within ~70s of every arriving file on a continuous pipeline. Wired to a real channel, that floods.
+  - **The hook now throttles: at most one notification per (pipeline, dataset, expectation) per `dq_notify.throttle_seconds` (default 3600).** Two time-aware layers, both fail-open (any state error notifies rather than stays silent): an in-memory last-notified map (always on; timestamp-based, not a seen-set, because continuous pipelines keep one Python process alive and a notify-once set would go silent forever; validated live NOTIFY -> SUPPRESS in window -> NOTIFY after window on a continuous Auto Loader pipeline) and optional durable marker files under `<dq_notify.state_dir>/dq_notify_state/<pipeline>/<dataset>__<expectation>.json` in an existing UC Volume (one human-readable JSON per expectation, overwritten on notify, never growing; validated across full refreshes and process restarts: 12 raw events across two consecutive full refreshes reduced to 2 notifications). State scopes per pipeline automatically from the event's own `origin` (`pipeline_name`/`pipeline_id`, observed present in hook events), so the same code serves any number of pipelines and cannot cross-suppress (validated with two pipelines sharing one state root). The asset deliberately ships no volume resource: state that describes operations should outlive any one bundle, so a new `state_volume_path` prompt (placeholder default = in-memory only) points under a volume the user already owns. `dq_notify.throttle_seconds: "0"` restores the old per-microbatch behavior.
+  - **Webhook channels and secrets.** `dq_notify.channel_format` selects the hook's payload: `slack` (validated end to end: secret scope -> `dbutils.secrets.get` -> throttled hook -> channel message, HTTP 200), `teams` (documented Workflows Adaptive Card envelope, marked doc-confirmed, not live-tested), or `generic` (plain JSON with the violation fields). Webhook URLs are credentials, so the shipped path resolves them from a secret scope via `dbutils.secrets.get` at module level, which works in serverless SDP pipeline Python (validated live), with two traps documented from observation: `{{secrets/scope/key}}` in pipeline configuration is NOT interpolated (the literal string arrives), and a missing scope raises `IllegalArgumentException`, which the code catches to degrade to print-only without touching the pipeline (validated). Plain `dq_notify.webhook_url` remains as the labeled demo-only fallback.
+  - **Backstop alert defaults and documentation.** `notify_on_ok: true` is the new shipped default: Alerts v2 notifies exactly once per state transition (measured: 9 consecutive TRIGGERED evaluations, 1 email), which means persistent failures mask new ones until recovery, and the single OK email (measured: exactly one) closes that loop. `retrigger_seconds` ships commented with measured semantics (re-send at the first evaluation past each window; a deliberate escalation knob, suggested 86400 for critical pipelines) and a commented `destination_id` line documents Slack/Teams/PagerDuty delivery via workspace notification destinations. New measured operational note: a broken sweep query flips the alert to ERROR and emails on EVERY evaluation until fixed (5 emails in 4 minutes observed), so the backstop cannot die silently but its failure mode is itself a flood.
+  - **README corrections and additions from measurement:** `hook_progress` records ENABLED, FAILED, and DISABLED transitions (v1.11.0 claimed enable-state only); the post-update teardown grace budget is seconds (a ~1s/event hook drained 5 queued events with zero loss; a 20s/event hook lost 6 of 6 under natural teardown), closing two formerly open questions; the in-hook `spark.sql()` write door is definitively shut (`INSERT INTO` fails with `UNSUPPORTED_SPARK_SQL_COMMAND`; allowlist: SELECT, DESCRIBE, SHOW variants, USE) while UC Volume file I/O from a hook works (create/overwrite/read/exists/listdir/stat; append fails with `OSError Errno 29`), which is exactly the surface the durable throttle uses; an update that processes no new rows emits no expectation events (re-running on static data never re-notified); continuous pipelines auto-start on `bundle deploy`. The skill grows a third non-negotiable rule (the throttle stays time-aware and fail-open), the adaptation reference covers throttle/window/state/channel/secret decisions and multi-pipeline rollout, and self-verify gains a marker-file check (suppression proof: a rerun inside the window must not advance `last_notified_at`).
+  - Tests extended accordingly: the throttle decision logic is exercised offline by exec-ing the pure helpers from the installed source against a temp state dir (suppress within window, durable across simulated process restarts, per-pipeline scoping, marker overwrite, `0` disables, empty state dir writes nothing), payload formats and guarded secret resolution are asserted, and the no-`spark.sql()`-calls AST guard still holds over the reworked source.
+
 ## [1.11.0] - 2026-07-11
 
 ### Added