Skip to content

feat: quarantine failed records during ingestion - #33

Merged
br413 merged 3 commits into
mainfrom
feat/quarantine-failed-records-31
Aug 11, 2026
Merged

br413 merged 3 commits into
mainfrom
feat/quarantine-failed-records-31

Conversation

@br413

@br413 br413 commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Summary

Implements ADR 0004 phases 2-4:

  • \�ronze.quarantine_events\ schema + Postgres/JSONL quarantine stores
  • Per-record validation routing; valid rows land while failures quarantine

  • ecords_quarantined\ in \IngestionSummary; --enable-quarantine\ / --alert-on-quarantine\ CLI flags
  • Airflow DAG enables quarantine + webhook alert
  • dbt \stg_events\ excludes quarantined event IDs

Closes #31

Test plan

  • \pytest tests/ -q\ (13 passed, 2 skipped integration without postgres)
  • \ ests/test_quarantine.py\ covers file + postgres paths

br413 added 3 commits August 10, 2026 23:30
Document dead-letter path for validation failures, ops runbook stub, and roadmap item. Part of dead-letter design spike.
 into docs/adr-0004-failed-record-quarantine-v2
Add bronze.quarantine_events, per-record validation routing, records_quarantined metric, webhook alerts, and dbt exclusion. Closes #31.
@br413
br413 merged commit d780a6a into main Aug 11, 2026
1 check failed
@br413
br413 deleted the feat/quarantine-failed-records-31 branch August 11, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: failed-record quarantine (dead-letter path)

1 participant