Skip to content

feat(ops): heartbeat pings for the worker and bot, and a tunnel readiness check - #84

Merged
matthew-demidoff merged 1 commit into
masterfrom
feat/heartbeats
Sep 6, 2026
Merged

feat(ops): heartbeat pings for the worker and bot, and a tunnel readiness check#84
matthew-demidoff merged 1 commit into
masterfrom
feat/heartbeats

Conversation

@matthew-demidoff

Copy link
Copy Markdown
Member

Fourth of the ops tranche: the dead-man's switch. Alerts (#83) are sent
from the host, so a dead worker, a hung loop, or a lost tunnel looked
identical to a quiet day. This adds the outbound half of an external
monitor; the monitor itself is a vendor choice (any healthchecks.io-style
ping-URL service; it can deliver into the same Telegram chat).

worker.js

  • Each loop records its last error-free completion (loop bodies resolve
    false when they swallow an error; runBatch treats that like a throw).
  • Every HEARTBEAT_INTERVAL_MS (60s) the worker GETs HEARTBEAT_URL only while
    every loop is inside its tolerance (a little over two periods each) and
    Redis answers a PING. Otherwise it logs heartbeat_withheld with the stale
    loops and raises a worker_unhealthy alert, so the vendor's missed-ping page
    and the in-band alert corroborate each other.
  • CLOUDFLARED_READY_URL (http://cloudflared:2000/ready from build(compose): read the tunnel token from the environment and expose cloudflared metrics #81) is probed
    each minute; cloudflared_not_ready alerts while the tunnel has no edge
    connection. It does not gate the worker's own ping: the tunnel's dead-man
    is the external HTTP probe.
  • Also fixed in passing: the activation-expiry sweep's catch logged but
    never alerted; it now does like the other loops.

bot/index.js

  • Pings HEARTBEAT_URL once a minute while the last successful getUpdates is
    within 2x the long-poll window; withholds (and logs) otherwise. The pinned
    status message now says DEGRADED with the age of the last good poll
    instead of UP unconditionally.

Compose maps HEARTBEAT_URL_WORKER / HEARTBEAT_URL_BOT / CLOUDFLARED_READY_URL
from the env file; unset means no pings. docs/deployment.md gains a
Monitoring section: which HTTP probes to configure (readiness, discovery,
optional JWKS), heartbeat periods and grace, and the Cloudflare WAF skip
rule vendor probes may need.

Verified against scratch stores with a fake vendor endpoint (2s interval):
5 pings in 12s while healthy; DB paused -> pings continue for the 30s
tolerance then stop, heartbeat_withheld names webhook_delivery, one deduped
worker_unhealthy alert; DB unpaused -> pings resume within seconds; a closed
tunnel port -> cloudflared_not_ready; bot with a bad token -> zero pings and
one withheld line after its 60s tick. 484 tests pass; 0 edge warnings; both
images build.

Deploy: pick a vendor, create two heartbeat checks (1 min period, 3 min
grace) and put their ping URLs in prod.env as HEARTBEAT_URL_WORKER and
HEARTBEAT_URL_BOT, set CLOUDFLARED_READY_URL=http://cloudflared:2000/ready,
then docker compose up -d --build worker bot, confirm pings arrive, then
add the HTTP probes from the Monitoring section.

…ness check

Nothing outside the host could tell whether the worker or the bot was
alive: alerts are sent from the host, so a dead process, a hung loop, or a
lost tunnel produced silence indistinguishable from a quiet day.

The worker now records each loop's last error-free completion and, once a
minute, pings HEARTBEAT_URL only while every loop is inside its tolerance
and Redis answers; otherwise it logs heartbeat_withheld and raises a
worker_unhealthy alert, so the external monitor's missed-ping page and the
in-band alert corroborate each other. Loop bodies report a swallowed error
by resolving false, which runBatch treats the same as a throw. The bot pings
the same way while its Telegram long-poll keeps succeeding, and its pinned
status message now reads DEGRADED with the age of the last good poll
instead of UP unconditionally. The worker also probes cloudflared's /ready
every minute and alerts cloudflared_not_ready while the tunnel has no edge
connection.

Vendor-neutral: any ping-URL monitoring service works. Unset URLs disable
the pings; nothing else changes.
@matthew-demidoff
matthew-demidoff merged commit 5c15a45 into master Sep 6, 2026
4 checks passed
@matthew-demidoff
matthew-demidoff deleted the feat/heartbeats branch September 6, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant