Watch mainnet separately from testnet in the uptime checker

[collinsezedike]

What

scripts/uptime-check.mjs only ever checked one API/web pair, hardcoded to the testnet deploy. render.mainnet.yaml stands up a separate quay-api-mainnet service that nothing pings — after cutover, the service that actually moves money could be down with no automated signal at all, while the existing checker keeps reporting a green testnet nobody is paying through.

Closes #162

How

• scripts/uptime-check.mjs now builds a list of environments (buildEnvironments) instead of one hardcoded pair. Testnet keeps its exact original defaults, env var names, and unprefixed target ids (api / web / synthetic) — zero migration for existing docs/uptime-state.json history or the README's badge files. Mainnet has no URL default of any kind: it's only checked once UPTIME_MAINNET_API_URL is actually configured, and its targets are prefixed (mainnet-api, etc.) so they get their own history series and can't collide with testnet's.
• docs/STATUS.md is now grouped by environment (## Testnet, ## Mainnet, each with its own ### API / ### Web dashboard / ### Create-link (synthetic) subsections) — a section only appears once that environment has actually been checked.
• Every target's label now includes its environment ("Mainnet — API"), which flows straight through to the auto-filed incident title (🔴 Uptime: Mainnet — API is down) with no workflow-side change needed.
• .github/workflows/uptime.yml passes the new UPTIME_MAINNET_API_URL / UPTIME_MAINNET_WEB_URL / UPTIME_MAINNET_SYNTHETIC_CHECK through from repo Variables (plain hostnames, not secrets).
• Mainnet's synthetic /links check defaults off even once the API URL is set — it would write a throwaway row into the real production database on every successful run, and POST /links has no scoped-credential story yet (issue 8.9 - The synthetic create-link check has 401'd since seller auth landed #163). Documented as an explicit opt-in.
• Added a "Uptime monitoring" section to docs/RUNBOOK.md documenting exactly which variables activate mainnet watching.
• The script itself now exports its pure logic (buildEnvironments, buildTargets, renderStatusMd, recordResult, uptimePct) behind an isCliInvocation() guard, matching the existing pattern in scripts/db-backup.ts — no more side effects on import, so it's actually testable.

Deliberately out of scope, per TODO.md §5's own delegation for this issue: I did not set a real UPTIME_MAINNET_API_URL, and did not re-enable the disabled cron schedule in uptime.yml. Both need real infrastructure/judgement only the repo owner has.

Test plan

• Added scripts/uptime-check.test.ts (pnpm test:scripts) — covers testnet's back-compat defaults/ids, mainnet's no-default/opt-in behavior, target id prefixing and label formatting, and renderStatusMd's per-environment grouping (including that an unconfigured/unchecked environment produces no section).
• pnpm test:scripts — my new file passes in full (18/18). Unrelated: scripts/db-backup-restore.test.ts fails locally on Windows with EBUSY: resource busy or locked on temp-file cleanup — a pre-existing environment issue on this file I didn't touch, not a regression from this change.
• Not run this pass: pnpm typecheck / pnpm test / pnpm build across the workspace — this PR touches no workspace package source (scripts/, one workflow YAML, one doc), so none of the affected packages' build graphs changed. Flagging this explicitly rather than claiming a full green I didn't re-verify this round; happy to run it if you'd like it confirmed before merge.

[drips-wave]

@collinsezedike Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[vercel]

@collinsezedike is attempting to deploy a commit to the determined's projects Team on Vercel.

A member of the Team first needs to authorize it.

[determined-001]

Reviewed and merged as-is — no changes needed.

You flagged that you had not run the full gate this pass. Ran it: pnpm typecheck 8/8, pnpm test 8/8, pnpm build 3/3, pnpm test:scripts 21/21 — including scripts/db-backup-restore.test.ts, which passes here. Your diagnosis of the Windows EBUSY as a pre-existing environment issue rather than a regression was correct.

I exercised the exported logic directly rather than trusting the description, because the back-compat and fail-safe claims are the whole risk surface:

• No env vars set → only testnet is active, target ids are api / web / synthetic. Those are exactly the three keys in the committed docs/uptime-state.json, so no history series is orphaned and no badge path moves.
• UPTIME_MAINNET_API_URL set → mainnet-api / mainnet-web appear alongside, never colliding with testnets series.
• Mainnet synthetic stays off with the API URL set, and only appears with UPTIME_MAINNET_SYNTHETIC_CHECK=1. Given it writes a real row into the production database on every success, defaulting it off is the right call, and gating it on 8.9 - The synthetic create-link check has 401'd since seller auth landed #163 rather than inventing a credential story here is the right scope.
• renderStatusMd emits ## Testnet with no Mainnet string at all, both when mainnet is unconfigured and when it is configured but not yet checked — so there is no empty section implying a green that was never measured.
• Labels come through as Mainnet — API, which lands in the incident title with no workflow change, as you describe.

One property worth stating for the record, since it is the issues actual thesis: if nobody sets UPTIME_MAINNET_API_URL after cutover, STATUS.md still shows only a green testnet. The gap moves from "impossible to monitor mainnet" to "someone must set one variable", and the RUNBOOK section is what closes it. That is the correct trade — a hardcoded mainnet URL default would be worse than no default — but it does mean the RUNBOOK addition is load-bearing, not decorative.

Leaving the cron schedule disabled and the real URL unset was right; both are owner decisions.