Skip to content

Productionize v1: outbox-backed atomic LISTEN/NOTIFY - #1

Merged
ypopivniak merged 12 commits into
mainfrom
feat/v1-scaffold
Apr 24, 2026
Merged

ypopivniak merged 12 commits into
mainfrom
feat/v1-scaffold

Conversation

@ypopivniak

Copy link
Copy Markdown
Collaborator

Summary

  • Rewrites the prototype as a productionized v1: ephemeral-outbox payload model, typed subscriptions, hybrid install, reconnect/health, overflow policy, self-managed cleanup, pluggable logger/metrics.
  • Adds unit + integration tests (testcontainers-go), CI workflow (lint + tests), runnable example, MIT LICENSE, full README. Version stamped at 1.0.0.
  • Module path moved to github.com/fastforgeinc/pgxevents; Go toolchain bumped to 1.25.

Design highlights

  • Atomic delivery via ephemeral outbox. PG NOTIFY is capped at 8000 bytes; the trigger writes a row snapshot into pgxevents_outbox (UNLOGGED) and pg_notify carries only the outbox UUID. Atomic with the mutation transaction; payload reflects state at commit, not at delivery.
  • Hybrid install. Default MigrationsRequired mode expects the canonical SQL (exported as InstallSQL() and TableTriggerSQL(table)) to be installed via the consumer's own migrations; library validates against pg_catalog and a version-stamped COMMENT ON FUNCTION, fails fast on mismatch with copy-paste SQL in the error. WithRuntimeInstall(true) installs at startup for dev/test.
  • Self-managed cleanup. Background ticker in the listener acquires a Postgres advisory lock and trims outbox rows older than WithCleanupTTL. No external cron, no pg_cron dependency, no leader election — all listener pods participate and the advisory lock elects one per tick.
  • Health-driven reconnection. Health() <-chan error lets consumers drain their downstream streams on listener failure and reconcile via fresh snapshots on reconnect. Reconnect uses exponential backoff with jitter.
  • Typed API via generics. ListenTyped[T](listener, table) returns events with *T populated from the snapshot — no fetcher callback. Decode errors are logged + counted as decode_error and skipped.

Test plan

  • go vet ./... and go vet -tags=integration ./... clean
  • go test -race ./... — unit tests pass (action, options, backoff, jitter, overflow policy, decode)
  • go test -tags=integration -race -count=1 -timeout=10m ./... — 8 integration tests against real Postgres pass (~13s)
  • gofmt -l . — no diffs
  • CI workflow exercises the same suite on PRs

Integration coverage:

  • Trigger fires on INSERT/UPDATE/DELETE with correct Action and Row state (DELETE carries the pre-delete row).
  • Large row (~20 KB JSON) delivered correctly via outbox — proves the 8000-byte NOTIFY limit is not a constraint.
  • Validation fails fast on missing function and missing per-table trigger, with actionable errors that include the SQL to install.
  • WithRuntimeInstall(true) bootstraps DDL end-to-end on a fresh database.
  • Health channel emits on startup.
  • Cleanup ticker deletes expired outbox rows under advisory-lock contention.
  • OverflowDisconnect closes the subscription when its buffer fills.

@ypopivniak
ypopivniak merged commit f77b249 into main Apr 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant