feat(ops): generalize operator alerts and add the daily digest - #83
Merged
Conversation
Operator alerts existed only for two webhook events, through a service the
worker could not share, and the worker's own copy had no rate limit. Nothing
alerted on Redis falling back, on a worker loop erroring every tick, or on a
migration failing at boot (a crash loop that pages nobody), and there was no
signal that the alert channel itself was alive.
lib/server/services/operatorAlerts.ts replaces webhookAlerts.ts with one
sendOperatorAlert(key, text, { windowSeconds }) behind two stacked windows:
Redis NX for cross-process dedupe and a per-process map in front of it, so
the redis_degraded alert that fires on every request during a Redis outage
still yields one message per half hour. worker-alert.js is its CommonJS twin
on a dedicated bounded connection, since the bullmq connection blocks for the
whole of an outage. Wired: webhook enqueue failure and auto-disable, the rate
limiter's fallback, every worker loop's catch, and scripts/migrate.mjs (best
effort, deduped through a short-lived ioredis import, sent anyway when Redis
is down).
worker-digest.js sends a daily digest at DIGEST_HOUR_UTC from tables the
service already writes; the digest arriving is the proof the worker, the
database, and the channel are alive, and `node worker.js --digest` sends one
on demand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of the ops tranche. Closes the gap that nothing inside the host reaches
a human unless a webhook endpoint auto-disables.
Before: two webhook-only alerts in a TS service the worker could not import,
a second copy in worker.js with no rate limit, and nothing at all for Redis
falling back, a worker loop erroring every tick, or a migration failing at
boot (a crash loop that pages nobody). No signal that the channel was alive.
After:
sendOperatorAlert(key, text, { windowSeconds }). Two stacked windows: Redis
NX (
alert:<key>) dedupes across app and worker; a per-process map infront of it bounds the rate when Redis itself is down, which is exactly
when redis_degraded fires on every request. 5 min default, 30 min for
conditions that repeat per tick. Production-only, no-op without a chat id,
never throws, never awaited on a request path.
connection (bullmq's connection runs with maxRetriesPerRequest: null and
would block for the whole outage). noopAlerter until startWorker() wires
it, so tests that require worker.js need no Redis.
limiter's fallback (both sites), every worker loop's catch and runBatch,
and scripts/migrate.mjs: best effort, deduped through a short-lived ioredis
import (the standalone image bundles it; verified in the running
container), sent anyway when Redis is unreachable.
security_events, webhook_deliveries, webhook_endpoints and users, plus
uptime and the alerts-sent-today counter. The hour gate plus a 36h NX
window on digest: gives one send per UTC day across restarts. The
digest arriving is the liveness proof;
node worker.js --digestsendsone on demand for the runbook. Exercised against scratch stores with a
bogus token: builds, prints, posts fail cleanly, exit 0.
keyare redacted by the secret-nameregex, so the failure log field is
alert.Explicitly deferred from the plan: backup age in the digest (arrives with the
backup sidecar in 3.5, nothing produces it yet).
Tests: 6 unit for the app alerter, 6 for the worker twin, 2 integration for
the digest; rate-limit mock gained getLastRedisError. 477 pass locally with
both stores; build has 0 edge-runtime warnings; worker image loads the new
modules; hygiene clean.
Deploy: ALERT_TELEGRAM_CHAT_ID is already set in prod.env (channel verified
with a test message).
docker compose up -d --build app worker, thendocker compose exec -T worker node worker.js --digestshould land thefirst digest in the chat.