fix(backend): route poison events to DLQ and back off batch growth on zero progress (#302)

[solaawojobi00-bit]

Fix Poison Event Stuck Batch Head Loop & Adaptive Sizing Hot Loop (#302)

Problem

In the event-sourcing dispatcher (EventStoreService), per-event projection dispatch failures caught during processBatch were recorded by incrementing failedCount and leaving processed = false. Because getUnprocessed selects with WHERE processed = false ORDER BY occurred_at ASC, version ASC, a poison event permanently remained at the head of every subsequent batch.
Meanwhile, recordBatchOutcome interpreted a saturated batch (total >= limit) as a growing backlog and exponentially doubled batchSize towards EVENT_STORE_MAX_BATCH_SIZE (2000), while nextDelayMs returned the 10ms catch-up interval (EVENT_STORE_CATCHUP_INTERVAL_MS) regardless of whether any events succeeded. This transformed a stuck backlog into a maximum-batch, near-zero-delay hot loop against Postgres while completely blocking downstream events.

Scenarios

Scenario	Previous Behavior	Desired Behavior
Poison event at batch head	Retried indefinitely on every tick; blocks all subsequent events in stream.	Attempt counter increments; after N failures (default: 3), routed to event_dead_letter and marked processed=true, dead_lettered=true. Downstream events proceed.
Non-failing events in same batch	Failing event caused batch failure, but valid events in the batch could not free the stream head.	Valid events in the same batch are successfully processed and marked processed; poison event retries or lands in DLQ.
Consecutive batches with zero progress	Saturated batch treated as backlog behind -> batchSize doubled up to 2000.	When processed === 0 && total > 0, batch sizing backs off towards baseline (EVENT_STORE_BATCH_SIZE) rather than doubling.
Scheduler tick delay on stuck backlog	Returned 10ms EVENT_STORE_CATCHUP_INTERVAL_MS, spinning the event loop against DB.	Returns aligned poll interval (EVENT_STORE_POLL_INTERVAL_MS, 500ms) on zero progress, eliminating hot loops.
Operator visibility	Failures logged only to stdout; impossible to query failed events or root causes from DB.	event_dead_letter stores payload, error message, and stack trace; queryable via getDeadLetterEvents() and getDeadLetterCount().

---

Solution

1. Per-Event Attempt Tracking & DLQ Destination:
   • Added attempts, last_error, dead_lettered, and dead_lettered_at columns to event_stream.
   • Created event_dead_letter table storing failed events with aggregate info, payload, attempts, error message, and stack trace.
   • Configured EVENT_STORE_MAX_ATTEMPTS (default: 3) in environment configuration and schema.
   • When an event exhausts retries, recordDeadLetter atomically writes to event_dead_letter and updates event_stream so it is no longer retrieved by getUnprocessed.
2. Adaptive Batch Backoff & Cadence Correction:
   • Updated recordBatchOutcome to check madeProgress (processed > 0). If total > 0 && !madeProgress, batchSize backs off towards EVENT_STORE_BATCH_SIZE.
   • Updated nextDelayMs so that EVENT_STORE_CATCHUP_INTERVAL_MS (10ms) is only returned when saturated && madeProgress. When zero progress is made, it returns the standard poll interval.
3. Operator Inspection:
   • Added getDeadLetterEvents(limit) and getDeadLetterCount() query helpers on EventStoreService.
   • Exposed deadLettered and maxAttempts in getSchedulerStats().

---

Changes

• backend/src/db/schema.sql: Added attempts, last_error, dead_lettered, dead_lettered_at columns to event_stream and defined the event_dead_letter table with indices.
• backend/src/config/env.js & backend/.env.example: Added EVENT_STORE_MAX_ATTEMPTS schema validation (default: 3) and documented optional setting.
• backend/src/eventSourcing/eventStore.js: Added retry tracking, dead-letter routing, backoff handling on zero progress, and operator query methods.
• backend/src/eventSourcing/eventStore.test.js: Added comprehensive unit test coverage for poison events, DLQ routing, backoff, and operator queries.

---

Regression Tests

Test Case	Acceptance Criteria Mapped	Result
retries failing events and increments attempt count below maxAttempts	Per-event attempt count with a dead-letter destination after N failures.	PASSED
routes to dead-letter queue after N failed attempts without blocking stream	Failed events are visible to operators with error message & payload; DLQ routing.	PASSED
one permanently failing event neither blocks subsequent events nor loops after DLQ	Test: one permanently-failing event neither blocks later events nor spins scheduler.	PASSED
backs off batch size instead of growing when saturated batch processes 0 events	Batch growth backs off when consecutive batches process nothing.	PASSED
nextDelayMs returns poll interval instead of catch-up interval when processed === 0	Scheduler does not spin on stuck backlog.	PASSED

---

Testing Output

PASS src/eventSourcing/eventStore.test.js
  EventStoreService.append inserted flag
    √ first insert of a stream_id+version is reported as inserted (12 ms)
    √ re-inserting the same stream_id+version is reported as NOT inserted (2 ms)
    √ distinct stream_id+version pairs are each reported as inserted (2 ms)
    √ the singleton eventStore exposes the same accurate semantics (2 ms)
  EventStoreService.appendBatch inserted flag
    √ duplicates inside a batch are skipped (inserted false) without throwing (6 ms)
  Poison event retry tracking & Dead-Letter Queue (DLQ)
    √ retries failing events and increments attempt count below maxAttempts (8 ms)
    √ routes to dead-letter queue after N failed attempts without blocking stream (76 ms)
    √ one permanently failing event neither blocks subsequent events nor loops after DLQ (32 ms)
  Adaptive batch backoff when zero progress is made
    √ backs off batch size instead of growing when saturated batch processes 0 events (3 ms)
    √ nextDelayMs returns poll interval instead of catch-up interval when processed === 0 (2 ms)

Test Suites: 41 passed, 41 of 45 total (4 skipped)
Tests:       362 passed, 370 total (8 skipped)
Snapshots:   0 total
Time:        79.692 s

---

Notes for Reviewers

• The attempt tracking and DLQ insertion handle single-statement fallback for mock/custom test pools as well as full transactional client queries for production postgres connections.
• Existing load harness and high-throughput donation spike scenarios remain fully intact and performant.

Closes #302

[AbuJulaybeeb]

KIndly update branch

[solaawojobi00-bit]

KIndly update branch

CI has been fixed and branch updated. Kindly Merge

Thanks

[Emmy123222]

Nice Work