feat: autonomous systems upgrades — health monitoring, SSE streaming, drift detection, GRU, webhook security

[Copilot]

Broad platform upgrade targeting autonomous operation: real-time observability, model performance self-monitoring, secure integrations, and modernised AI client.

Description

Adds foundational autonomous-systems capabilities across backend, Python ML service, and ChatGPT client — covering live training observability, drift-triggered retraining, webhook security hardening, and modern LLM API usage.

Type of Change

• ✨ New feature (non-breaking change which adds functionality)
• 🔧 Refactor (no functional changes)
• 🧪 Test updates

Changes Made

Backend (Node.js)

• Graceful shutdown — SIGTERM/SIGINT handlers with configurable SHUTDOWN_TIMEOUT_MS; fixes Jest open-handle leak
• GET /api/status — aggregate health endpoint polling backend uptime + Python service; returns 200/503 for orchestration probes
• GET /api/models/:id/stream — SSE endpoint that polls DB every 5 s and pushes live training status/metrics; auto-closes on completion or client disconnect
• Webhook HMAC-SHA256 — verifies X-Webhook-Signature against the raw request body (captured via express.json({ verify })); uses equal-length timingSafeEqual to prevent timing attacks

ChatGPT Client (src/chatgpt.js)

• Default model updated gpt-4 → gpt-4o; available list includes gpt-4o-mini, gpt-4-turbo
• streamChat(message, onChunk) — token-by-token streaming with per-chunk callback
• analyzeTimeSeries(data) — structured GPT prompt returning parsed JSON: trend, seasonality, anomalies, insights, recommendations

const analysis = await chatgpt.analyzeTimeSeries(dataPoints);
// { trend: 'increasing', anomalies: [], recommendations: ['...'] }

Python ML Service

• Real GRU architecture — LSTMModel(use_gru=True) swaps all recurrent layers to keras.layers.GRU; previously just aliased LSTM
• detect_drift(recent_data, baseline_metrics, drift_threshold) — computes MAE on recent observations vs. baseline test_loss; returns drift_score and drift_detected flag
• POST /api/monitor — drift detection endpoint; validates modelId as alphanumeric (prevents path traversal), optionally POSTs to backend /retrain when drift exceeds threshold
• Error responses no longer leak stack traces; print() replaced with logging (ISO timestamps + levels)

Infrastructure

• Docker healthcheck added to backend and python-service containers
• .env.example updated: WEBHOOK_SECRET, SHUTDOWN_TIMEOUT_MS, DRIFT_THRESHOLD, OPENCLAW_API_KEY, PYTHON_SERVICE_PORT

Testing

• Unit tests pass
• Integration tests pass
• Manual testing completed
• No breaking changes

All 40 JS unit tests pass with clean exit. Python monitor endpoint validated for missing fields, invalid modelId, insufficient data, and missing model file.

Checklist

• My code follows the code style of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

Additional Context

The /api/monitor endpoint is designed for periodic calls from an external scheduler (cron, Kubernetes CronJob, etc.) or from the frontend. When autoRetrain: true, the service autonomously notifies the backend to re-queue training — no human intervention required.