Add sdp-expectation-notifications asset (v1.11.0) - #16
Merged
Conversation
Dual-door asset packaging the validated hook-plus-backstop pattern for per-expectation data-quality notification on Lakeflow SDP: a native event hook notifies the moment a WARN expectation result is logged (fast, best-effort by platform design), and one DABs-managed Alert v2 sweeps the published event log over a trailing 1-day window on a schedule (guaranteed). Ships a reference pipeline on samples.nyctaxi.trips, the backstop alert resource, event-log queries, an in-bundle usage doc, and a companion agent skill that wires the pattern into the user's own pipeline. Validated end to end on a live serverless SDP workspace (2026-07-11, CLI v0.297.2): hook registered and fired with counts matching the event log exactly (3003 failed / 18929 passed), backstop alert evaluated on cron to TRIGGERED and delivered by email. Release metadata finalized per RELEASING.md: changelog heading renamed to [1.11.0] - 2026-07-11 and both version markers bumped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
None (asset ships directly per the asset-library authoring workflow).
Summary
Adds the
sdp-expectation-notificationsasset: per-expectation data-quality notification for Lakeflow SDP, packaged as the pair the pattern requires. A native event hook (@dp.on_event_hook) notifies the moment a WARN expectation result is logged (fast path, best-effort by platform design: no guarantee every hook runs before pipeline compute terminates), and exactly one DABs-managed Alert v2 sweeps the published pipeline event log over a trailing 1-day window on a schedule (guaranteed path). This PR also cuts release v1.11.0.Changes
assets/sdp-expectation-notifications/: reference pipeline onsamples.nyctaxi.trips(one streaming table, a deliberate WARN tripwiretrip_distance < 5, the notifier hook withprint+ optionalrequestswebhook), pipeline resource withevent_logpublication, backstop Alert v2 resource (GREATER_THAN 0over summedfailed_records, daily 06:00 UTC cron, email subscription), event-log queries, in-bundle usage doc, and a companion agent skill (SKILL.md+ 2 references) that wires the pattern into the user's own pipeline.warehouse_idandnotification_emaildefault to placeholders; the alert resource filename/key derive frompipeline_resource_key(<key>_backstop).tests/assets/test_sdp_expectation_notifications.py, 18 tests) including an AST-based guard that the shipped pipeline source makes no.sql()call (the live-proven hook anti-pattern) and that the alert query aggregates a past-time window rather than latest-row.[1.11.0] - 2026-07-11, both version markers bumped).Change Area
assets/<name>/)Configuration Axes Affected
Testing
pytest tests/ -V): 2440 passed, 163 skippeddatabricks bundle init ./assets/sdp-expectation-notifications --output-dir <tmp> --config-file tests/configs/assets/sdp_expectation_notifications.json)Asset Changes
databricks bundle init . --template-dir assets/<name> --output-dir <dir>library/helpers.tmplor other assets)tests/configs/assets/sdp_expectation_notifications.jsonaddedLive validation (playground, 2026-07-11, CLI v0.297.2)
Installed from this branch into a fresh bundle and validated end to end on a Free Edition workspace:
bundle validateclean on first try; pipeline updatea8b9fec1completed on serverless with the hook registered (hook_progress: ENABLED); driver log carried the exact notification line (failed=3003 | passed=18929, matching the event log per update); backstop alert evaluated on its cron toTRIGGERED(evaluated value 3003) and the email was delivered and confirmed.Release (this PR cuts v1.11.0)
CHANGELOG.mdfinalized:[Unreleased]renamed to[1.11.0] - 2026-07-11, fresh empty[Unreleased]added above1.11.0(pyproject.tomlandtemplate/{{.project_name}}/bundle_init_config.json.tmpl)tests/test_release_metadata.py, 4 passed)Checklist
{{ }}blocks).tmplfiles appear in generated output