8.8 - Nothing monitors the mainnet service

[determined-001]

Complexity: Medium - 150 Points · type:ops

Milestone: M6 - Ops & rigor

Band lever: none

Context
scripts/uptime-check.mjs pings an API and a web origin, records history, and
files a GitHub issue when a target stays down. Its defaults are
quay-api.onrender.com and quay-web.vercel.app — the testnet/staging pair.

Problem
render.mainnet.yaml deliberately stands up a separate service,
quay-api-mainnet, at a different URL. Nothing pings it. After cutover, the
service that actually moves money can be down with no automated signal at all:
no issue filed, no badge change, nothing in docs/STATUS.md — while the
monitoring that does exist keeps reporting green for a testnet service nobody is
paying through. This is distinct from the anchor-probe and db-backup gaps
already listed in TODO.md; those are known, this one is not written down
anywhere.

What needs to be done

1. Make the checker multi-environment: take a list of targets (id, label, API
   URL, web URL) rather than one pair, from a small config file or env.
2. Give each environment its own history series and its own section in
   docs/STATUS.md, so a green testnet cannot mask a red mainnet.
3. Include the environment name in the auto-filed issue title and body.
4. Document in docs/RUNBOOK.md which secrets/variables must be set for the
   mainnet environment to be watched.

Key files

• scripts/uptime-check.mjs:17 — API_URL / WEB_URL defaults, TARGETS
• .github/workflows/uptime.yml — the (currently disabled) schedule
• render.mainnet.yaml:39 — the mainnet service name
• docs/STATUS.md

Done when

• Both environments are checked in one run and reported separately.
• A mainnet outage files an issue naming mainnet.

---

Tracked in ISSUES.md — issue 8.8.

[collinsezedike]

@collinsezedike has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Let me help you with this issue.

Here is my plan:

• Refactor scripts/uptime-check.mjs to support multiple environment targets (id, label, API, Web URLs) to monitor both testnet and mainnet services.
• Update docs/STATUS.md to track and display history series separately for each environment.
• Modify the auto-issue filing logic to include the target environment name in the title and body.
• Document the required secrets/variables for mainnet monitoring in docs/RUNBOOK.md.

Please assign this issue to me!

ℹ️ Repo Maintainers: To accept this application, review their application or assign @collinsezedike to this issue.

[GauravKarakoti]

@GauravKarakoti has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Allow me to work on this one

ℹ️ Repo Maintainers: To accept this application, review their application or assign @GauravKarakoti to this issue.

[ibochivincent-lang]

@ibochivincent-lang has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I’d like to work on Issue #162 because the current uptime checker only monitors the testnet deployment, while the separate mainnet API service can go down without generating any alert. That creates a serious monitoring gap for the environment handling real production traffic.

I will first review scripts/uptime-check.mjs, .github/workflows/uptime.yml, render.mainnet.yaml, and the current status-history logic to understand how targets, failures, and GitHub issues are currently handled.

I will then update the checker to support multiple environments instead of a single API/web pair. Each target will have its own identifier, label, API URL, and web URL, with the configuration kept simple and easy to maintain.

The monitoring results will be kept separate for testnet and mainnet so one environment cannot hide problems in the other. I will update docs/STATUS.md so each environment has its own clearly labelled health and history section.

When an outage is detected, the automatically created GitHub issue will include the affected environment in both the title and body, making it immediately clear whether the problem is on testnet or mainnet.

I will also update docs/RUNBOOK.md to document the variables and secrets required for mainnet monitoring, including how the target is configured and what operators should verify when the monitor reports an outage.

For validation, I will test a normal run where both environments are healthy, a testnet-only failure, and a mainnet-only failure. I will verify that each environment maintains its own history and that a mainnet outage produces an issue explicitly identifying mainnet.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ibochivincent-lang to this issue.

[drips-wave]

Congratulations, @collinsezedike! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on August 31, 2026.

🧑‍💻 @collinsezedike: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊