Skip to content

feat(ops): generalize operator alerts and add the daily digest - #83

Merged
matthew-demidoff merged 1 commit into
masterfrom
feat/operator-alerts-digest
Sep 6, 2026
Merged

feat(ops): generalize operator alerts and add the daily digest#83
matthew-demidoff merged 1 commit into
masterfrom
feat/operator-alerts-digest

Conversation

@matthew-demidoff

Copy link
Copy Markdown
Member

Third of the ops tranche. Closes the gap that nothing inside the host reaches
a human unless a webhook endpoint auto-disables.

Before: two webhook-only alerts in a TS service the worker could not import,
a second copy in worker.js with no rate limit, and nothing at all for Redis
falling back, a worker loop erroring every tick, or a migration failing at
boot (a crash loop that pages nobody). No signal that the channel was alive.

After:

  • lib/server/services/operatorAlerts.ts replaces webhookAlerts.ts with one
    sendOperatorAlert(key, text, { windowSeconds }). Two stacked windows: Redis
    NX (alert:<key>) dedupes across app and worker; a per-process map in
    front of it bounds the rate when Redis itself is down, which is exactly
    when redis_degraded fires on every request. 5 min default, 30 min for
    conditions that repeat per tick. Production-only, no-op without a chat id,
    never throws, never awaited on a request path.
  • worker-alert.js is the CommonJS twin on a dedicated bounded Redis
    connection (bullmq's connection runs with maxRetriesPerRequest: null and
    would block for the whole outage). noopAlerter until startWorker() wires
    it, so tests that require worker.js need no Redis.
  • Wired: webhook enqueue failure and auto-disable (as before), the rate
    limiter's fallback (both sites), every worker loop's catch and runBatch,
    and scripts/migrate.mjs: best effort, deduped through a short-lived ioredis
    import (the standalone image bundles it; verified in the running
    container), sent anyway when Redis is unreachable.
  • worker-digest.js: daily digest at DIGEST_HOUR_UTC (compose default 8) from
    security_events, webhook_deliveries, webhook_endpoints and users, plus
    uptime and the alerts-sent-today counter. The hour gate plus a 36h NX
    window on digest: gives one send per UTC day across restarts. The
    digest arriving is the liveness proof; node worker.js --digest sends
    one on demand for the runbook. Exercised against scratch stores with a
    bogus token: builds, prints, posts fail cleanly, exit 0.
  • Logger note: metadata fields named key are redacted by the secret-name
    regex, so the failure log field is alert.
  • Dockerfile COPY, compose DIGEST_HOUR_UTC, .env.example, docs.

Explicitly deferred from the plan: backup age in the digest (arrives with the
backup sidecar in 3.5, nothing produces it yet).

Tests: 6 unit for the app alerter, 6 for the worker twin, 2 integration for
the digest; rate-limit mock gained getLastRedisError. 477 pass locally with
both stores; build has 0 edge-runtime warnings; worker image loads the new
modules; hygiene clean.

Deploy: ALERT_TELEGRAM_CHAT_ID is already set in prod.env (channel verified
with a test message). docker compose up -d --build app worker, then
docker compose exec -T worker node worker.js --digest should land the
first digest in the chat.

Operator alerts existed only for two webhook events, through a service the
worker could not share, and the worker's own copy had no rate limit. Nothing
alerted on Redis falling back, on a worker loop erroring every tick, or on a
migration failing at boot (a crash loop that pages nobody), and there was no
signal that the alert channel itself was alive.

lib/server/services/operatorAlerts.ts replaces webhookAlerts.ts with one
sendOperatorAlert(key, text, { windowSeconds }) behind two stacked windows:
Redis NX for cross-process dedupe and a per-process map in front of it, so
the redis_degraded alert that fires on every request during a Redis outage
still yields one message per half hour. worker-alert.js is its CommonJS twin
on a dedicated bounded connection, since the bullmq connection blocks for the
whole of an outage. Wired: webhook enqueue failure and auto-disable, the rate
limiter's fallback, every worker loop's catch, and scripts/migrate.mjs (best
effort, deduped through a short-lived ioredis import, sent anyway when Redis
is down).

worker-digest.js sends a daily digest at DIGEST_HOUR_UTC from tables the
service already writes; the digest arriving is the proof the worker, the
database, and the channel are alive, and `node worker.js --digest` sends one
on demand.
@matthew-demidoff
matthew-demidoff merged commit 04d0832 into master Sep 6, 2026
4 checks passed
@matthew-demidoff
matthew-demidoff deleted the feat/operator-alerts-digest branch September 6, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant