Problem
Events that exhaust the publish retry budget are lost with no durable trace. In StreamSession.pump() (xstreamroll-processing/src/session.ts), when handlers.publish fails maxPublishRetries + 1 times the event is "dead-lettered":
this.logger.error(`[${this.workerId}] session ${this.id} publish FAILED after ${attempts} attempt(s) — dead-lettering event: ${error.message}`)
this.emit("dead-letter", next, error)
break // skip this event, carry on with the queueThe only consumer of the dead-letter event would be code that subscribed to the session's EventEmitter — a repo-wide search shows nothing does. SessionRegistry.spawn() (xstreamroll-processing/src/session-registry.ts) registers a passive session.on("error", ...) listener and a state listener, but no dead-letter listener. So a dead-lettered event produces one log line and then ceases to exist: it is not persisted, not retried later, and not visible in any API surface.
The event is also not recoverable from the source: the stream_data row that produced it stays in the pending table (nothing drains it today), so the same event will be re-fetched and re-dead-lettered on the next poll, permanently. With the missing POST /streams/processed endpoint this is masked (everything 404s), but once the drain endpoint lands, a genuinely bad event (e.g. one that violates a future constraint) would be re-processed and re-dead-lettered in a tight loop with no operator-visible artifact beyond log lines.
Consequence: the platform's only "permanently failed event" signal is a log line. Operators cannot inspect what failed, why, or how often; a poison event can spin the worker forever; and any at-least-once consumer expecting a DLQ-style record has nothing to read.
Root cause
// xstreamroll-processing/src/session.ts — pump(), retry exhausted
this.emit("dead-letter", next, error) // ← emitted, but no subscriber anywhereWhy this is architecturally hard
- The worker is deliberately stateless about failures ("The session keeps running after a dead-letter" — session JSDoc), so a durable dead-letter store is a new persistence concern for the processing service. The codebase precedent is the API's
webhook_deliveries table (database/schema.sql): a statused row with last_error, attempt_count, and next_attempt_at, written by the API which owns Postgres. The worker has no database dependency in its default memory lock config — it only talks to the API over HTTP — so the natural design is to POST dead-lettered events back to the API (mirroring the POST /streams/processed publish path) rather than giving the worker its own DB connection. That decision — worker-owned store vs API-owned store — is the core design question.
StreamEvent has no stable id today (streamId, data, timestamp), so a dead-letter record cannot be deduplicated. Whatever store is chosen needs the same idempotency key the processed-event endpoint defines, or a poison event floods the store on every poll.- The retry budget and backoff live in
StreamSession (constructor params maxPublishRetries); the dead-letter signal must escape the session abstraction (through the registry's handlers or an explicit listener wiring in worker.ts) without re-entangling the deliberately dependency-free StreamSession class.
- Operators need a way to see the queue. If the store is API-side, that means an endpoint or the admin surface; if worker-side, a metrics/file surface. Pick one so the acceptance criteria can be verified.
Acceptance criteria
Behaviour
Observability
Tests
Documentation
Out of scope
Automatic replay of dead-lettered events, and the POST /streams/processed drain endpoint itself (this issue depends on the idempotency-key decision there and should be sequenced after it).
Getting started
Real files in scope: xstreamroll-processing/src/session.ts, xstreamroll-processing/src/session-registry.ts, xstreamroll-processing/src/worker.ts, api/src/webhooks/webhook-delivery.entity.ts + webhook-deliveries-db.repository.ts (store pattern), database/schema.sql (webhook_deliveries as the schema pattern).
Verify with:
cd xstreamroll-processing && npm run typecheck && npm test
cd ../api && npm run typecheck && npm test
Good first files to read: xstreamroll-processing/src/session.ts (the dead-letter branch in pump), api/src/webhooks/webhook-deliveries-db.repository.ts (statused-delivery store pattern).
Problem
Events that exhaust the publish retry budget are lost with no durable trace. In
StreamSession.pump()(xstreamroll-processing/src/session.ts), whenhandlers.publishfailsmaxPublishRetries + 1times the event is "dead-lettered":The only consumer of the
dead-letterevent would be code that subscribed to the session's EventEmitter — a repo-wide search shows nothing does.SessionRegistry.spawn()(xstreamroll-processing/src/session-registry.ts) registers a passivesession.on("error", ...)listener and astatelistener, but nodead-letterlistener. So a dead-lettered event produces one log line and then ceases to exist: it is not persisted, not retried later, and not visible in any API surface.The event is also not recoverable from the source: the
stream_datarow that produced it stays in the pending table (nothing drains it today), so the same event will be re-fetched and re-dead-lettered on the next poll, permanently. With the missingPOST /streams/processedendpoint this is masked (everything 404s), but once the drain endpoint lands, a genuinely bad event (e.g. one that violates a future constraint) would be re-processed and re-dead-lettered in a tight loop with no operator-visible artifact beyond log lines.Consequence: the platform's only "permanently failed event" signal is a log line. Operators cannot inspect what failed, why, or how often; a poison event can spin the worker forever; and any at-least-once consumer expecting a DLQ-style record has nothing to read.
Root cause
Why this is architecturally hard
webhook_deliveriestable (database/schema.sql): a statused row withlast_error,attempt_count, andnext_attempt_at, written by the API which owns Postgres. The worker has no database dependency in its defaultmemorylock config — it only talks to the API over HTTP — so the natural design is to POST dead-lettered events back to the API (mirroring thePOST /streams/processedpublish path) rather than giving the worker its own DB connection. That decision — worker-owned store vs API-owned store — is the core design question.StreamEventhas no stable id today (streamId,data,timestamp), so a dead-letter record cannot be deduplicated. Whatever store is chosen needs the same idempotency key the processed-event endpoint defines, or a poison event floods the store on every poll.StreamSession(constructor paramsmaxPublishRetries); the dead-letter signal must escape the session abstraction (through the registry'shandlersor an explicit listener wiring inworker.ts) without re-entangling the deliberately dependency-freeStreamSessionclass.Acceptance criteria
Behaviour
Observability
attempt_countandlast_errorfor each record.Tests
xstreamroll-processing/__tests__/session.test.ts(orworker.test.ts): a publish that always fails produces exactly one dead-letter record aftermaxPublishRetries + 1attempts, and processing continues for subsequent events.__tests__/integration/pipeline.integration.test.ts) pass unchanged.Documentation
StreamSessionJSDoc's dead-letter description matches the durable behaviour.Out of scope
Automatic replay of dead-lettered events, and the
POST /streams/processeddrain endpoint itself (this issue depends on the idempotency-key decision there and should be sequenced after it).Getting started
Real files in scope:
xstreamroll-processing/src/session.ts,xstreamroll-processing/src/session-registry.ts,xstreamroll-processing/src/worker.ts,api/src/webhooks/webhook-delivery.entity.ts+webhook-deliveries-db.repository.ts(store pattern),database/schema.sql(webhook_deliveriesas the schema pattern).Verify with:
Good first files to read:
xstreamroll-processing/src/session.ts(the dead-letter branch inpump),api/src/webhooks/webhook-deliveries-db.repository.ts(statused-delivery store pattern).