feat: Implement Graceful Shutdown Drain Protocol (#1187)

[DevNetlife]

Description

Closes #1187

This protocol ensures safe process teardown during maintenance or node replacement by pausing queue ingestion, gracefully draining active WebSockets, tracking in-flight HTTP requests, and returning HTTP 503 with a Retry-After header for incoming state-mutating requests.

Key Changes

• Data Model & Persistence: Added Knex migration 20260829000000_shutdown_drain_protocol.ts defining shutdown_drain_sessions and shutdown_drain_logs for session lifecycle tracking and auditing.
• Service Behavior (DrainProtocolService):
  • Manages node lifecycle states: ACTIVE, DRAINING, DRAINED, CANCELLED, FAILED.
  • Pauses background queues (BullMQ, supply verification) and webhook workers.
  • Gracefully closes WebSocket connections with notification frames.
  • Tracks in-flight HTTP request counter and supports graceful timeouts.
  • Exports Prometheus metrics: bridge_watch_drain_status, bridge_watch_drain_in_flight_requests, and bridge_watch_drain_events_total.
• HTTP Middleware (drainProtectionMiddleware):
  • Rejects incoming state-mutating requests (POST, PUT, DELETE, PATCH) with HTTP 503 Service Unavailable and Retry-After: 30 header while server is draining.
  • Exempts health checks (/health, /healthz, /readyz, /api/v1/health) and administrative drain management endpoints.
• API Surface: Added admin endpoints under /api/v1/admin/shutdown/drain:
  • POST /start — Initiate drain session
  • GET /status — Fetch current node drain status and in-flight counters
  • POST /cancel — Abort drain and resume normal operations
  • POST /force — Force immediate process shutdown
  • GET /history — View past drain audit logs
• Process Signal Integration: Integrated DrainProtocolService.startDrain() with SIGTERM and SIGINT handlers in index.ts.
• Documentation & Tests:
  • Created operator guide in docs/SHUTDOWN_DRAIN_PROTOCOL.md.
  • Added unit and route integration tests in drainProtocol.service.test.ts and drainProtocol.routes.test.ts.

Acceptance Criteria Verification

• Defined and documented protocol behavior, inputs, outputs, and compatibility.
• Enforced RBAC authentication/authorization (admin:write) and safe failure handling.
• Added Knex database persistence, migration, audit logging, and Prometheus metrics.
• Covered feature with unit and integration tests (10 passing tests).
• Documented rollout, rollback, and operator workflows in docs/SHUTDOWN_DRAIN_PROTOCOL.md.

How to Test

npm run test --workspace=backend -- drainProtocol

[Mosas2000]

The graceful shutdown protocol ensures zero-downtime deployments by letting in-flight tasks finish while safely routing new traffic away. Just verify that polling intervals in startDrain are cleared upon cancellation to prevent memory leaks; perfectly oksy!