feat: website health check cron (standalone GH Actions workflow)#39
Merged
Conversation
Add a public, gateway-native GET /readyz that fans out concurrently to
each downstream service's shallow /health (auth, expense, finance,
datarights), bounded by a per-probe timeout. It returns 200
{"status":"ok"} only when all four are healthy, else 503 naming each
service's state, so a single probe proves the whole backend is reachable.
Kept separate from the shallow /health (Docker HEALTHCHECK) to avoid
coupling container liveness to downstream availability. Classified Public
via GatewayResolve alongside /health and /metrics.
Mount a public GET /healthz on the shell that mirrors the gateway's /readyz aggregate: 200 when the gateway reports all backend services healthy, else 503, relaying the gateway JSON body so CI logs name the failing service. A hung/unreachable gateway is bounded by AbortSignal.timeout so the endpoint never hangs. The handler is a DI factory (gatewayUrl, fetchFn, timeoutMs) so it is unit-testable without booting SSR, and is mounted after the /api proxy and before the SSR catch-all. Extend the shell vitest include glob with server/**/__tests__/** so the new server test is discovered and run (previously app/**-only globs would silently skip it).
Add .github/workflows/healthcheck.yml, independent of ci.yml/cd.yml. On a 12h schedule (and workflow_dispatch) it curls the public, header-gated https://usegofin.com/healthz with the X-Health-Token secret header, retries a bounded 3x to avoid false alarms from transient blips, and fails the job on non-200/unreachable. Only on failure it posts exactly one Discord alert (repo, last status, relayed body, run URL) via jq-built JSON to the DISCORD_WEBHOOK_URL secret; success is silent. A top-of-file comment records the Cloudflare WAF Skip rule expression (path + X-Health-Token, not the token value) so an operator can reconstruct the edge gate from a repo artifact.
Address review: /readyz previously used a second hardcoded service-name map (servicesFromURLs) parallel to the proxies map, a latent drift hazard (a 5th service added to proxies but omitted here would make /readyz silently skip it). Introduce serviceURLMap as the single canonical name->URL source; build both the reverse-proxy handlers and the readiness fan-out from it, so the existing prefix->proxy panic guard now also guarantees /readyz probes every proxied service by construction. Also drain probe response bodies before Close (keep-alive reuse) and make the timeout test's stub honour request-context cancellation so t.Cleanup no longer blocks on the full 2s sleep (~2s -> 0.1s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a standalone GitHub Actions workflow that health-checks the production site (https://usegofin.com) every 12 hours, verifying the public edge and the backend are up. On failure the run goes red and posts a Discord alert; success is silent.
"Up" is proven end-to-end by a single probe: Cloudflare → tunnel → shell → gateway → all 4 downstream services (auth, expense, finance, datarights).
What's included
GET /readyz(gateway) — new public readiness aggregate that fans out concurrently to each downstream service's shallow/health, bounded by a 2s per-probe timeout. Returns200 {"status":"ok"}only when all four are healthy, else503naming each service's state. Kept separate from the shallow/health(DockerHEALTHCHECK) to avoid coupling container liveness to downstream availability.GET /healthz(shell) — new public liveness endpoint that mirrors the gateway's/readyz(relaying its JSON body so CI logs name the failing service). Bounded byAbortSignal.timeoutso it never hangs. Mounted after the/apiproxy and before the SSR catch-all..github/workflows/healthcheck.yml— standalone cron (0 */12 * * *) +workflow_dispatch, independent ofci.yml/cd.yml. Curls the header-gated/healthzwith theX-Health-Tokensecret, retries 3× to absorb transient blips, fails the job on non-200/unreachable, and posts exactly one Discord message on failure only.The existing shallow
/healthendpoints (gateway + per-service) are unchanged. Docs (docs/api.md,docs/monitoring.md) are deferred/out of scope per the plan.How it gets past Cloudflare Under Attack Mode
A pre-provisioned Cloudflare WAF Skip rule bypasses Under Attack Mode for
path == /healthz AND header X-Health-Token == <token>. Only this probe (with the secret header) can reach the origin; a normal automated client still gets a403challenge. The rule expression is documented at the top of the workflow (never the token value).Testing
go test ./...(all packages pass),go vetclean,golangci-lint0 issues. New tests cover all-healthy, one-down (names the service), unreachable, hung/timeout (bounded), genuine concurrency, and theHandler200/503 + router/readyzcases.vitest13 files / 116 tests,server/healthz.tsat 100% coverage, coverage thresholds pass, eslint clean. Thevitest.config.tsincludewas extended so the newserver/**test is actually discovered (not silently skipped).actionlint(incl. shellcheck) 0 issues; happy/failure paths and the Discord JSON payload verified locally against a stub server.Open dependencies (operator actions, not code)
DISCORD_WEBHOOK_URLmust be added as a repo-level GitHub Actions secret (today it only exists as an Alertmanager env var). Until then the failure path still reddens the job, but the Discord POST would error./healthz+/readyzmust ship to prod viacd.ymlon merge tomainbefore the first live cron run can pass.workflow_dispatch's happy path is therefore not verifiable pre-merge; post-deploy smoke (dispatch → green/no Discord; forced bad token → red + one Discord) remains an operator step.Note
HEALTHCHECK_TOKENand the Cloudflare WAF Skip rule were provisioned during planning and are not touched here.