feat(monitor): add the external uptime monitor as a Cloudflare Worker - #85
Merged
Conversation
The heartbeat pings and HTTP probes from the previous change needed a receiver outside the host. This is it: a Worker on the free plan with a cron trigger every minute and a D1 database. GET /ping/<worker|bot>?token=... records a heartbeat (token compared in constant time via SHA-256 digests and timingSafeEqual, never logged). The cron probes /api/health/ready and the discovery document, judges the heartbeats against a grace period, and posts to the operator chat only on a state change: DOWN after the configured consecutive failures (two for HTTP, one for an already-graced heartbeat), RECOVERED with the outage duration. Rows are written only when a check changes, so a day costs a few thousand D1 writes against a 100,000 allowance. One summary line per day at SUMMARY_HOUR_UTC is the monitor's own liveness signal. The decision logic lives in src/logic.ts with no bindings and is unit tested; index.ts does the I/O. CI type-checks against the committed wrangler-generated types and bundles with deploy --dry-run. The root tsconfig excludes monitor/, which has its own toolchain.
The head-of-line regression test (and the original delivery test) stamped next_attempt_at from the host's clock and then asked the claim, which compares against Postgres's now(), to pick the row up. Under Docker Desktop the container clock can lag the host by a few milliseconds, so the row was sometimes not due yet and the claim skipped it: the unreproduced single failure seen earlier, and a rejected pre-push here. Both tests now push the row into the past on the database's own clock.
The wrangler-generated worker-configuration.d.ts carries the text "Runtime types generated with workerd@...", which the public-hygiene gate reads as an attribution marker, and it is 13k lines of tool output that regenerates deterministically from wrangler.jsonc. typecheck now runs `wrangler types` before tsc, the file is ignored, and CI does the same. The verify job's docker-logs step now tolerates a missing container, so an early failure no longer reports a second, spurious one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
3.4b: the receiver for the heartbeats and probes from #84, on the Workers
free plan instead of a third-party vendor.
Free-tier fit (checked against current docs): Workers 100k requests/day,
5 cron triggers/account with every-minute schedules; D1 100k writes and 5M
reads/day. This Worker uses ~1.4k cron ticks + ~2.9k pings/day and writes
D1 rows only when a check changes state. KV was ruled out: its free tier
allows 1,000 writes/day, below two per-minute heartbeats.
monitor/ (self-contained: own package.json, tsconfig, vitest config)
* * * * *, D1 binding (id omitted; provisioned onfirst deploy), vars for target URL, heartbeat grace (180s), HTTP failures
before alerting (2), summary hour. Secrets PING_TOKEN, TELEGRAM_BOT_TOKEN,
ALERT_TELEGRAM_CHAT_ID via wrangler secret put; merged into the generated
Env by declaration on the global
Env(the generated interface does notextend Cloudflare.Env, which is why the augmentation targets
Env).transition function that goes down after N consecutive failures and back
up on the first success, emitting an event only on the flip; message
formatting with outage duration; a daily summary.
SHA-256 + timingSafeEqual, 404 on mismatch, never logged) upserts the
heartbeat; scheduled() probes readiness + discovery, reads heartbeats and
stored states, applies transitions, batches only the changed rows, sends
one Telegram message per event, and once per UTC day (insert-or-ignore
guard) a "External monitor: n/4 checks up" line.
threshold, no repeated events, recovery, unchanged-row detection, and
message text.
Root: tsconfig excludes monitor/; .gitignore covers .dev.vars and .wrangler;
a
monitorCI job runswrangler types --check+ tsc, the tests, and adeploy --dry-runbundle (no credentials needed; verified locally).docs/deployment.md Monitoring now leads with the Worker and its setup
order; runbooks/oncall.md rotation matrix gains the Worker as a
TELEGRAM_BOT_TOKEN consumer and a PING_TOKEN row.
Also carries one test fix, because the pre-push hook rejected this branch
with it: the webhook claim tests stamped next_attempt_at from the host clock
and compared it against Postgres's now(); Docker Desktop's container clock
can lag by milliseconds, so the row was sometimes not yet due. That was the
single unreproduced failure noted in the Phase 1 handoff. Both tests now set
the row due on the database clock.
Owner steps after merge (need a browser login): wrangler login, deploy,
d1 migrations apply, three secret puts, then the two ping URLs go into
prod.env as HEARTBEAT_URL_WORKER/BOT with CLOUDFLARED_READY_URL, and
docker compose up -d --build worker bot.