feat(openfga): async reconciliation outbox (Options B+C follow-up to #567) by JohnRDOrazio · Pull Request #631 · Liturgical-Calendar/LiturgicalCalendarAPI

JohnRDOrazio · 2026-06-03T04:49:13Z

Closes the Options B and C work from issue #567. Lands the spec, the implementation plan, and the 37 implementation commits as one PR per the design's "B and C share a single coherent design" decision.

What this delivers

A Postgres outbox + Redis Streams + systemd consumer + cron backstop that turns OpenFGA tuple writes/deletes from "best-effort during the request" into "eventually consistent with admin-visible DLQ".

Architecture

HTTP request → handler (PG tx wraps DB write + outbox.insertBatch)
            → sync fast-path: OutboxProcessor::processSync per row
            → XADD to Redis Stream for any still-pending row
            
systemd ↔ ConsumerLoop ↔ RedisStreamConsumer ↔ OutboxProcessor (XREADGROUP, sub-second reconciliation)
cron    → bin/reconcile-outbox backstop → BackstopRunner → OutboxProcessor (every 5 min, picks up cracks)
admin   → GET /admin/outbox?status=… | ?summary=1 | POST /admin/outbox/{id}/retry
ops     → GET /health → openfga_outbox block (counts, oldest_pending_age, consumer.redis_reachable, oldest_pel_idle)

Invariants

Atomicity: handler's PG transaction wraps repo.write + outbox.insertBatch. Either both commit or neither.
Idempotency: deterministic handler-built keys + UNIQUE index on metadata->>'idempotency_key' + ON CONFLICT DO NOTHING. Reissuing the same approve/revoke returns the same outbox row IDs.
Terminal-status stickiness: all four UPDATE mutations include WHERE … AND status IN (…) guards. markRetryable cannot downgrade a failed_terminal row.
At-least-once: SKIP LOCKED + consumer PEL + backstop together guarantee every row is eventually attempted.
Graceful degradation: Redis unreachable → handler skips XADD (logged WARNING); backstop drains within 5 min. PG durability unaffected.

Retry policy

Exponential backoff 1s → 2s → 4s → … → 512s, capped at 2^9. Max 10 attempts (~17 min budget) before a row goes failed_terminal. Validation 4xx is terminal on first attempt. Benign duplicates (TupleAlreadyExistsException / TupleNotFoundException) count as success.

Semantic shift from #628

success: true now means DB committed rather than all sync FGA work succeeded. New fields outbox_pending, outbox_failed, outbox_ids carry the side-effect state; the message field carries the warning when reconciliation is deferred. Pre-existing tests that asserted success: false on transient FGA failures were updated to reflect the new contract.

Files

New (~30): migration, OutboxOperation/OutboxStatus/OutboxDisposition enums, OutboxBackoff, OutboxClassifier, OutboxRow, OutboxRepository, OutboxProcessor (+ OutboxProcessorInterface), OutboxNotifier, RedisStreamConsumer (+ StreamConsumerInterface), ConsumerLoop, BackstopRunner, bin/reconcile-outbox, OutboxAdminHandler, HealthHandler, Handlers/Ops/, deploy artifacts (systemd unit + cron + runbook), tests for every layer.

Modified: AccessRequestAdminHandler (approve + revoke refactored), PermissionAdminHandler (grant + revoke refactored), RoleCascadeService (silent catch → outbox enqueue), Router (/admin/outbox and /health routes), Health.php (outbox stats), .env.example, composer.json.

Test plan

composer test — 1235 tests / 5 pre-existing skips / no new skips
composer analyse — PHPStan L10 clean (267 files)
composer lint — phpcs PSR-12 clean
composer lint:md — markdown lint clean
composer reconciler:backstop — end-to-end CLI works
Migration up/down roundtrip — verified
Reviewer: deploy the systemd unit + cron on staging and verify the consumer drains a real approve into OpenFGA

Process trail

Spec: docs/superpowers/specs/2026-06-02-openfga-async-reconciliation-design.md
Plan: docs/superpowers/plans/2026-06-02-openfga-async-reconciliation.md
Final whole-branch review (opus): Approved with 5 Important items, all addressed in commits 270117d / 625076b / b268982 / 4742d15 / 77f03f6 / 3cc05e8.
Tracking issue opened during brainstorming: Add API usage analytics + metrics emission framework #630 (future metrics framework — out of scope for this PR).

Out of scope (per spec §10)

Zitadel role-sync moving into the outbox (its zitadel_sync_status column stays).
Automated pruning of succeeded rows (runbook documents the SQL; ship automation only when table size warrants).
Bulk /retry-all endpoint (single-row retry covers v1).
lease_until column for multi-consumer scale (YAGNI; trivial to add later).
Metrics framework (tracked in Add API usage analytics + metrics emission framework #630).
Retry-After header honoring for 429s (premature until observed).

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Health endpoint with DB + outbox status; admin endpoint to list/summary/retry outbox rows
- Async OpenFGA reconciliation: consumer loop, Redis-stream notifications, cron backstop, CLI entrypoint and systemd service
Documentation
- Operator runbook and design doc with install, diagnostics, and troubleshooting
Tests
- Extensive test coverage for outbox pipeline, consumer, processor, notifier, classifier, backoff, handlers, and health
Chores
- Example env vars for reconciliation, composer scripts and extension suggestions, Redis stub for CI/dev

#567 PR #628 closed #567 Option A (typed exceptions + fail-fast in approve/revoke) and #571 (ListObjects in filterByAdminAccess). The original issue framing was "Pick one (or layer them)" across three options; B (compensating actions) and C (outbox + async reconciliation) remain. This commit adds the design document for landing B+C layered together as one PR. Architecture: - DB outbox table (canonical) atomically committed with the business write - Redis Streams + systemd long-lived consumer for sub-second reconciliation on the happy path - Cron-driven backstop every 5 min as durability anchor for the narrow window where PG committed but the API process died before XADD - Hybrid response model: success = (DB committed); outbox_pending / outbox_failed / outbox_ids carry the side-effect state separately - Exponential backoff (1s..512s, 10 attempts, ~17 min budget) then failed_terminal with admin-visible DLQ at /admin/outbox No code in this commit — spec only. Implementation plan will be generated next via the writing-plans skill, then executed in subsequent commits on this branch. Related: opens #630 (tracking issue for future API metrics / analytics framework — out of scope for this work). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

31 tasks across 11 phases, each one TDD-cycle granular: Phase 1 — schema (migration) Phase 2 — pure types and logic (operation/status/disposition enums, backoff, classifier) Phase 3 — repository (insertBatch with idempotency, getById, mark{Succeeded,Retryable,FailedTerminal}, pickupPending with FOR UPDATE SKIP LOCKED, resetForRetry, countByStatus) Phase 4 — OutboxProcessor (single point of OpenFGA contact) Phase 5 — Redis layer (Notifier best-effort XADD, Streams consumer with XCLAIM stale reclaim) Phase 6 — runners + CLI (BackstopRunner, ConsumerLoop, bin/reconcile-outbox) Phase 7 — handler integration (Access* and Permission* handlers, plus RoleCascadeService silent-catch fix) Phase 8 — /admin/outbox endpoint + router Phase 9 — /health outbox block Phase 10 — deployment artifacts (systemd unit, cron, runbook, env) Phase 11 — full quality gates + smoke test Each task pins exact file paths, complete test+impl code, commands with expected output, and an exact commit message. No "fill in" placeholders. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

One row per OpenFGA tuple operation. Atomically committed with the business write; drained by the consumer (Redis Streams) and the cron backstop. Partial indexes on (status, next_attempt_at) keep the pickup hot path tight; UNIQUE constraint on metadata->>'idempotency_key' makes re-issued handler calls a no-op insert. Part of issue #567 Options B+C. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>