feat(monitor): add the external uptime monitor as a Cloudflare Worker

.github/workflows/security.yml

@@ -86,11 +86,32 @@ jobs:
       - run: npm run test:e2e
         env:
           PLAYWRIGHT_BASE_URL: http://127.0.0.1:3000
+      # Runs on any earlier failure, including ones before the container exists.
       - if: failure()
-        run: docker logs auth-runner
+        run: docker logs auth-runner 2>/dev/null || true
       - if: always()
         run: docker rm -f auth-runner
 
+  monitor:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: monitor
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-node@v7
+        with:
+          node-version-file: .nvmrc
+          cache: npm
+          cache-dependency-path: monitor/package-lock.json
+      - run: npm ci
+      # typecheck regenerates worker-configuration.d.ts from wrangler.jsonc (it
+      # is not committed) before tsc; `wrangler deploy --dry-run` bundles the
+      # Worker without publishing and needs no credentials.
+      - run: npm run typecheck
+      - run: npm test
+      - run: npx wrangler deploy --dry-run --outdir /tmp/bundle
+
   sdk:
     runs-on: ubuntu-latest
     strategy:

.gitignore

@@ -29,6 +29,11 @@ test-results/
 playwright-report/
 sdk/*/dist/
 
+# Cloudflare Worker (monitor/): local secrets, wrangler state, generated types
+monitor/.dev.vars
+monitor/.wrangler/
+monitor/worker-configuration.d.ts
+
 docs/prompts/
 handoffs/
 

docs/deployment.md

@@ -144,9 +144,36 @@ monitor described next.
 
 ## Monitoring
 
-Two kinds of check, both run by a monitoring service outside the host (any
-vendor that offers HTTP probes and "expect a ping every N minutes" checks,
-delivering to the same Telegram chat if you like):
+The external monitor is a Cloudflare Worker in `monitor/`, running on the
+Workers free plan: a cron trigger every minute probes the public endpoints,
+judges the heartbeats it has received, and posts to the alert chat only when
+a check changes state (`DOWN: ...`, `RECOVERED: ... after 7m`), plus one
+"External monitor: n/4 checks up" line per day at `SUMMARY_HOUR_UTC`. It runs
+on Cloudflare's network, so it keeps working when this host does not. Setup:
+
+```sh
+cd monitor && npm ci
+npx wrangler login                       # once, opens a browser
+npx wrangler deploy                      # first deploy also provisions the D1 database
+npx wrangler d1 migrations apply auth-monitor --remote
+npx wrangler secret put PING_TOKEN       # openssl rand -hex 32
+npx wrangler secret put TELEGRAM_BOT_TOKEN
+npx wrangler secret put ALERT_TELEGRAM_CHAT_ID
+```
+
+The cron starts with the first deploy, so `npx wrangler tail` shows a few
+error lines until the migration and the secrets are in place; nothing is
+sent to the chat before the secrets exist. Later code changes are just
+`npx wrangler deploy`.
+
+`wrangler deploy` prints the Worker URL; the heartbeat URLs for the env file
+are `https://<worker-url>/ping/worker?token=<PING_TOKEN>` and
+`.../ping/bot?token=<PING_TOKEN>`. The token is compared in constant time
+and never logged. `TELEGRAM_BOT_TOKEN` now has a second consumer (see the
+runbook's rotation matrix). Any other ping-URL monitoring service works in
+its place; the checks below are what it must implement.
+
+Two kinds of check:
 
 HTTP probes, from outside, through the tunnel:
 

monitor/.dev.vars.example

@@ -0,0 +1,5 @@
+# Copy to .dev.vars for `wrangler dev`; production values are set with
+# `wrangler secret put <NAME>` and never committed.
+PING_TOKEN=
+TELEGRAM_BOT_TOKEN=
+ALERT_TELEGRAM_CHAT_ID=

monitor/README.md

@@ -0,0 +1,16 @@
+# auth-monitor
+
+Cloudflare Worker that watches auth.bneck.com from outside the host: a cron
+trigger every minute probes `/api/health/ready` and the discovery document,
+judges the worker and bot heartbeats it receives on `/ping/<source>`, and
+posts to the operator chat only when a check changes state. State lives in
+D1. Free plan; see `docs/deployment.md` (Monitoring) for setup and the
+rotation notes in `runbooks/oncall.md`.
+
+```sh
+npm ci
+npm run typecheck     # wrangler types (generates worker-configuration.d.ts) + tsc
+npm test              # decision logic, no bindings needed
+npm run dev           # wrangler dev --test-scheduled; GET /__scheduled runs the cron once
+npm run deploy
+```

monitor/migrations/0001_init.sql

@@ -0,0 +1,21 @@
+-- Heartbeats received from inside the host, one row per source.
+create table heartbeats (
+  name text primary key,
+  last_seen_at integer not null
+);
+
+-- Current state of every check; rows are written only on change plus the
+-- failure counter, so the write budget stays a fraction of the free tier.
+create table checks (
+  name text primary key,
+  status text not null check (status in ('up', 'down')),
+  since integer not null,
+  failures integer not null default 0,
+  detail text
+);
+
+-- Small key/value state: which daily summaries have gone out.
+create table monitor_state (
+  key text primary key,
+  value text not null
+);