feat: decouple indexer and web server via Redis pub/sub

[success-OG]

Summary

This PR implements the decoupled microservices architecture for Open-Audit: indexing and web serving run as separate processes connected by Redis pub/sub. The indexer worker streams Stellar events, translates them, and publishes to Redis; the web server subscribes and fans events out over WebSocket to dashboard clients.

This addresses the known limitations called out in server.ts's deprecation notice: indexing starving the HTTP/WebSocket server under load, no fault isolation when the indexer crashes, and inability to scale components independently.

The legacy monolithic path (server.ts + npm run dev:ws) is preserved as a documented fallback for simple deployments.

---

Problem

Today, server.ts runs Next.js, WebSocket, and the Stellar indexer in a single Node.js process. Under heavy load:

• Indexing competes with HTTP/WebSocket for CPU
• WebSocket connections can drop during high transaction velocity
• An indexer crash takes down the entire server
• Web and indexer cannot be scaled or deployed independently

The worker side (src/worker/indexer.ts) already published to Redis, but there was no corresponding web subscriber, no shared server utilities, and no deployment tooling to run them as separate services.

---

Solution

Architecture

Stellar Horizon SSE
       │
       ▼
Indexer Worker (src/worker/indexer.ts)
  • stream events
  • translateEvent()
  • Redis PUBLISH → stellar:events
  • heartbeat → open-audit:worker:heartbeat
       │
       ▼
Redis Pub/Sub
       │
       ▼
Web Server (server-decoupled.ts)
  • Redis SUBSCRIBE
  • broadcast TranslatedEvent → WebSocket /ws/events
  • serve Next.js HTTP API (no indexing)
       │
       ▼
Dashboard (useLiveFeed hook)

New files

File	Purpose
server-decoupled.ts	Web-only entry point: Next.js + WebSocket + Redis subscriber
lib/server/ws-server.ts	Shared WebSocket server with per-IP connection limits and broadcast
lib/server/csp.ts	Shared Content-Security-Policy header
lib/redis/config.ts	Centralized REDIS_URL, REDIS_CHANNEL, heartbeat key
lib/redis/publisher.ts	Worker-side Redis client with reconnect, publish queue, heartbeat
lib/redis/subscriber.ts	Web-side Redis subscriber → WebSocket fan-out
lib/events/message-envelope.ts	Pub/sub message contract (serialize / parse / toWebSocketPayload)
docker-compose.microservices.yml	Redis + worker + web with health checks
Dockerfile.web	Production image for the decoupled web server
Dockerfile.worker	Production image for the indexer worker
ecosystem.config.js	PM2 config with restart-on-crash for both processes

Refactored files

File	Change
server.ts	Uses shared ws-server and csp modules; legacy monolith preserved
src/worker/indexer.ts	Slimmed down to use shared RedisPublisher and message envelope
ARCHITECTURE.md	Documents the real implemented topology with ASCII + Mermaid diagrams
package.json	Adds dev:decoupled, worker:indexer, build:server, PM2/Docker scripts
lib/metrics.ts	Adds startTelemetry() and recordTranslationDuration() hooks used by both servers

Redis message format

The worker publishes an envelope; the web server unwraps it and broadcasts only the TranslatedEvent (what useLiveFeed expects):

{
  "type": "event",
  "timestamp": 1234567890,
  "workerId": "worker-1",
  "raw": { /* RawEvent */ },
  "translated": { /* TranslatedEvent */ }
}

---

How to run

Local development

# 1. Start Redis (if not already running)
docker run -p 6379:6379 redis:7-alpine

# 2. Copy env and configure REDIS_URL / REDIS_CHANNEL if needed
cp .env.example .env.local

# 3. Start both processes together
npm run dev:decoupled

# Or run individually:
npm run dev:decoupled:web   # web server only
npm run worker:indexer      # indexer worker only

# Legacy monolith (unchanged fallback)
npm run dev:ws

Docker Compose

docker compose -f docker-compose.microservices.yml up --build

Brings up three services: open-audit-redis, open-audit-worker, open-audit-web (port 3000).

PM2 (non-Docker production)

npm run build && npm run build:server
pm2 start ecosystem.config.js

Both open-audit-web and open-audit-worker restart automatically on crash.

---

Fault isolation

This is the core value of the split and is covered in two ways:

1. Automated test — lib/server/__tests__/fault-isolation.test.ts verifies WebSocket connections stay alive when the Redis subscriber reports errors (web server logs the error and continues).

2. Manual verification:

   • Start npm run dev:decoupled
   • Connect a WebSocket client to ws://localhost:3000/ws/events
   • Kill the worker process (kill <pid> or docker stop open-audit-worker)
   • Confirm the web server and existing WebSocket stay connected; only new events stop until the worker restarts

---

Test plan

• npm run validate:docs passes (no dead script or doc links)
• Start Redis locally
• npm run dev:decoupled — both web and worker start; events flow indexer → Redis → web → WebSocket → dashboard
• npm run worker:indexer — indexer runs standalone and publishes to Redis
• Kill worker while web is running — existing WebSocket connections remain open
• docker compose -f docker-compose.microservices.yml up --build — all three services reach healthy state
• npm run build && npm run build:server && pm2 start ecosystem.config.js — both processes restart after simulated crash
• npm test -- lib/events/__tests__/message-envelope.test.ts lib/server/__tests__/fault-isolation.test.ts

closes #403