Skip to content

Add sdp-expectation-notifications asset (v1.11.0) - #16

Merged
vmariiechko merged 1 commit into
mainfrom
feature/sdp-expectation-notifications-asset
Jul 11, 2026
Merged

Add sdp-expectation-notifications asset (v1.11.0)#16
vmariiechko merged 1 commit into
mainfrom
feature/sdp-expectation-notifications-asset

Conversation

@vmariiechko

Copy link
Copy Markdown
Owner

Related Issue

None (asset ships directly per the asset-library authoring workflow).

Summary

Adds the sdp-expectation-notifications asset: per-expectation data-quality notification for Lakeflow SDP, packaged as the pair the pattern requires. A native event hook (@dp.on_event_hook) notifies the moment a WARN expectation result is logged (fast path, best-effort by platform design: no guarantee every hook runs before pipeline compute terminates), and exactly one DABs-managed Alert v2 sweeps the published pipeline event log over a trailing 1-day window on a schedule (guaranteed path). This PR also cuts release v1.11.0.

Changes

  • New dual-door asset assets/sdp-expectation-notifications/: reference pipeline on samples.nyctaxi.trips (one streaming table, a deliberate WARN tripwire trip_distance < 5, the notifier hook with print + optional requests webhook), pipeline resource with event_log publication, backstop Alert v2 resource (GREATER_THAN 0 over summed failed_records, daily 06:00 UTC cron, email subscription), event-log queries, in-bundle usage doc, and a companion agent skill (SKILL.md + 2 references) that wires the pattern into the user's own pipeline.
  • 8 prompts with safe defaults; warehouse_id and notification_email default to placeholders; the alert resource filename/key derive from pipeline_resource_key (<key>_backstop).
  • Asset-specific tests (tests/assets/test_sdp_expectation_notifications.py, 18 tests) including an AST-based guard that the shipped pipeline source makes no .sql() call (the live-proven hook anti-pattern) and that the alert query aggregates a past-time window rather than latest-row.
  • Catalog/roadmap/changelog rows; release metadata finalized ([1.11.0] - 2026-07-11, both version markers bumped).

Change Area

  • Asset Library (assets/<name>/)

Configuration Axes Affected

  • Asset Library (new asset, asset schema, or framework changes)

Testing

  • All tests pass (pytest tests/ -V): 2440 passed, 163 skipped
  • Manual template generation tested (asset install via databricks bundle init ./assets/sdp-expectation-notifications --output-dir <tmp> --config-file tests/configs/assets/sdp_expectation_notifications.json)
  • New tests added for new functionality

Asset Changes

  • Asset installs standalone via databricks bundle init . --template-dir assets/<name> --output-dir <dir>
  • Asset is self-contained (no references to library/helpers.tmpl or other assets)
  • tests/configs/assets/sdp_expectation_notifications.json added
  • Asset appears in ASSETS.md catalog

Live validation (playground, 2026-07-11, CLI v0.297.2)

Installed from this branch into a fresh bundle and validated end to end on a Free Edition workspace: bundle validate clean on first try; pipeline update a8b9fec1 completed on serverless with the hook registered (hook_progress: ENABLED); driver log carried the exact notification line (failed=3003 | passed=18929, matching the event log per update); backstop alert evaluated on its cron to TRIGGERED (evaluated value 3003) and the email was delivered and confirmed.

Release (this PR cuts v1.11.0)

  • CHANGELOG.md finalized: [Unreleased] renamed to [1.11.0] - 2026-07-11, fresh empty [Unreleased] added above
  • Both version markers bumped to 1.11.0 (pyproject.toml and template/{{.project_name}}/bundle_init_config.json.tmpl)
  • Version guard test passes (tests/test_release_metadata.py, 4 passed)

Checklist

  • Go template syntax is valid (no unclosed {{ }} blocks)
  • No .tmpl files appear in generated output
  • Generated YAML files are valid
  • Documentation updated

Dual-door asset packaging the validated hook-plus-backstop pattern for
per-expectation data-quality notification on Lakeflow SDP: a native event
hook notifies the moment a WARN expectation result is logged (fast,
best-effort by platform design), and one DABs-managed Alert v2 sweeps the
published event log over a trailing 1-day window on a schedule (guaranteed).
Ships a reference pipeline on samples.nyctaxi.trips, the backstop alert
resource, event-log queries, an in-bundle usage doc, and a companion agent
skill that wires the pattern into the user's own pipeline.

Validated end to end on a live serverless SDP workspace (2026-07-11,
CLI v0.297.2): hook registered and fired with counts matching the event log
exactly (3003 failed / 18929 passed), backstop alert evaluated on cron to
TRIGGERED and delivered by email.

Release metadata finalized per RELEASING.md: changelog heading renamed to
[1.11.0] - 2026-07-11 and both version markers bumped.
@vmariiechko
vmariiechko merged commit ee601ee into main Jul 11, 2026
1 check passed
@vmariiechko
vmariiechko deleted the feature/sdp-expectation-notifications-asset branch July 11, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant