Make the realtime invoice stream work across replicas with heartbeats and reconnect replay

[Cedarich]

• Complexity: Medium
• Labels: Backend, realtime, reliability, Medium
• Overview: The realtime invoice stream is backed by a plain in-process RxJS Subject. InvoiceEventsService holds new Subject<InvoiceStatusEvent>(), publishStatusChange pushes into it, and streamForMerchant filters it by merchant ID. Events therefore reach only the clients connected to the same Node process that produced them. There is no cross-instance transport, no keepalive, and no replay, so the stream is silently unreliable in exactly the deployment shape the rest of the backend assumes.
• Details:
  • With more than one replica, the merchant's SSE connection and the watcher that reconciles their payment are routinely on different processes. The Horizon watcher runs on whichever replica happens to poll; the SSE client is routed independently by the load balancer. When they differ, publishStatusChange fires into a Subject that the merchant's connection is not subscribed to, and the payment simply never appears live. This is the same single-instance assumption already filed for the queue and crons in Add cross-instance claim locking for the webhook queue and scheduled backend jobs #442.
  • The failure is invisible. The connection stays open and healthy, no error is raised on either side, and the merchant just sees an invoice that never updates. The web and mobile clients both fall back to polling in places, which masks the gap intermittently rather than exposing it.
  • There is no heartbeat. @Sse("invoices") returns the filtered observable directly, so a stream with no invoice activity sends no bytes. Proxies and load balancers commonly close idle connections after thirty to sixty seconds, so a quiet merchant's stream is dropped by infrastructure and the client cannot distinguish that from a stream that is simply quiet.
  • There is no event ID and no Last-Event-ID handling, so nothing is replayed on reconnect. A status change that occurs during a reconnect window is lost permanently — there is no cursor, and the Subject has no buffer.
  • There is no connection limit. Any authenticated merchant can hold arbitrarily many concurrent SSE connections, each an open subscription and an open socket, with no cap and no accounting.
  • streamForMerchant(user.merchantId) is called with a value the User type permits to be absent. When it is, the filter compares against undefined and matches nothing, producing a connection that is open and permanently silent rather than an explicit failure.
  • This underpins work that has already shipped — the web live status in Replace web invoice status polling with the backend realtime stream where it matters most #288 and the mobile invoice detail stream in Add live invoice status updates to mobile invoice detail screens #289 — so the reliability gap is inherited by both clients.
• Scope:
  • Move event distribution onto a transport shared across replicas so a status change published on one instance reaches subscribers on any instance; Redis is already a dependency for the throttler and is the natural candidate.
  • Emit a periodic heartbeat so idle connections are not closed by intermediaries and clients can distinguish a live quiet stream from a dead one.
  • Assign event IDs and support Last-Event-ID on reconnect, replaying missed events within a bounded window so a reconnect does not silently drop a status change.
  • Cap concurrent connections per merchant and account for open streams, so connection count is bounded and observable.
  • Reject a stream request from a principal without a resolvable merchant ID instead of returning a connection that can never emit.
  • Document the delivery guarantee the stream actually offers, and confirm the client fallbacks in web and mobile are consistent with it rather than compensating for undefined behaviour.
  • Add tests covering cross-instance delivery, heartbeat emission, reconnect replay, and the missing-merchant case.
• Technical scope:
  • backend/src/realtime/invoice-events.service.ts
  • backend/src/realtime/realtime.controller.ts
  • backend/src/invoices/invoices.service.ts
  • backend/src/throttler/throttler.module.ts
  • web/lib/invoice-realtime-client.ts
  • mobile/lib/invoice-realtime-client.ts
• Acceptance criteria:
  • A status change published on one replica reaches a subscriber connected to a different replica.
  • Idle streams emit a heartbeat and survive intermediary idle timeouts.
  • A client reconnecting with Last-Event-ID receives the events it missed within the documented replay window.
  • Concurrent connections per merchant are capped, and open connection counts are observable.
  • A request without a resolvable merchant ID is rejected rather than yielding a permanently silent stream.
  • The stream's delivery guarantee is documented and the web and mobile clients align with it.
  • Tests cover cross-instance delivery, heartbeat, reconnect replay, and the missing-merchant path.

[Devadakene]

I am experienced enough to implement this issue having worked on similar issues in the past. Please assign

[abdulomeiza]

I am a Full Stack Developer specializing in Next.js,Python, TypeScript, Node.js,Rust, PostgreSQL,Oracle, JavaScript and DevOps Engineering. With over 50 contributions across projects in web3 and specifically stellar ecosystem, I’ve developed strong proficiency in delivering high-quality solutions and resolving complex issues within tight deadlines. My experience spans frontend, backend, smart contracts, and the optimization and maintenance of scalable codebases.
I can handle this task. Kindly assign.

[grantfox-oss]

🦊 GrantFox — @abdulomeiza has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #524)
2. Your PR will be reviewed by the ZyntariHQ maintainers

Good luck! Track your progress on GrantFox.