feat(ops): heartbeat pings for the worker and bot, and a tunnel readiness check - #84
Merged
Conversation
…ness check Nothing outside the host could tell whether the worker or the bot was alive: alerts are sent from the host, so a dead process, a hung loop, or a lost tunnel produced silence indistinguishable from a quiet day. The worker now records each loop's last error-free completion and, once a minute, pings HEARTBEAT_URL only while every loop is inside its tolerance and Redis answers; otherwise it logs heartbeat_withheld and raises a worker_unhealthy alert, so the external monitor's missed-ping page and the in-band alert corroborate each other. Loop bodies report a swallowed error by resolving false, which runBatch treats the same as a throw. The bot pings the same way while its Telegram long-poll keeps succeeding, and its pinned status message now reads DEGRADED with the age of the last good poll instead of UP unconditionally. The worker also probes cloudflared's /ready every minute and alerts cloudflared_not_ready while the tunnel has no edge connection. Vendor-neutral: any ping-URL monitoring service works. Unset URLs disable the pings; nothing else changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth of the ops tranche: the dead-man's switch. Alerts (#83) are sent
from the host, so a dead worker, a hung loop, or a lost tunnel looked
identical to a quiet day. This adds the outbound half of an external
monitor; the monitor itself is a vendor choice (any healthchecks.io-style
ping-URL service; it can deliver into the same Telegram chat).
worker.js
false when they swallow an error; runBatch treats that like a throw).
every loop is inside its tolerance (a little over two periods each) and
Redis answers a PING. Otherwise it logs heartbeat_withheld with the stale
loops and raises a worker_unhealthy alert, so the vendor's missed-ping page
and the in-band alert corroborate each other.
each minute; cloudflared_not_ready alerts while the tunnel has no edge
connection. It does not gate the worker's own ping: the tunnel's dead-man
is the external HTTP probe.
never alerted; it now does like the other loops.
bot/index.js
within 2x the long-poll window; withholds (and logs) otherwise. The pinned
status message now says DEGRADED with the age of the last good poll
instead of UP unconditionally.
Compose maps HEARTBEAT_URL_WORKER / HEARTBEAT_URL_BOT / CLOUDFLARED_READY_URL
from the env file; unset means no pings. docs/deployment.md gains a
Monitoring section: which HTTP probes to configure (readiness, discovery,
optional JWKS), heartbeat periods and grace, and the Cloudflare WAF skip
rule vendor probes may need.
Verified against scratch stores with a fake vendor endpoint (2s interval):
5 pings in 12s while healthy; DB paused -> pings continue for the 30s
tolerance then stop, heartbeat_withheld names webhook_delivery, one deduped
worker_unhealthy alert; DB unpaused -> pings resume within seconds; a closed
tunnel port -> cloudflared_not_ready; bot with a bad token -> zero pings and
one withheld line after its 60s tick. 484 tests pass; 0 edge warnings; both
images build.
Deploy: pick a vendor, create two heartbeat checks (1 min period, 3 min
grace) and put their ping URLs in prod.env as HEARTBEAT_URL_WORKER and
HEARTBEAT_URL_BOT, set CLOUDFLARED_READY_URL=http://cloudflared:2000/ready,
then
docker compose up -d --build worker bot, confirm pings arrive, thenadd the HTTP probes from the Monitoring section.