Skip to content

fix(network-config): require ESPLORA_URL + ESPLORA_WS_URL on Mainnet - #115

Merged
TaprootFreak merged 1 commit into
release/mainnet-hardeningfrom
fix/require-explicit-mainnet-urls
May 26, 2026
Merged

fix(network-config): require ESPLORA_URL + ESPLORA_WS_URL on Mainnet#115
TaprootFreak merged 1 commit into
release/mainnet-hardeningfrom
fix/require-explicit-mainnet-urls

Conversation

@TaprootFreak

Copy link
Copy Markdown
Contributor

Summary

NETWORK_CONFIG silently fell back to the Mutinynet defaults
(https://mutinynet.com/api, wss://mutinynet.com/api/v1/ws)
regardless of IS_MAINNET. On Mainnet that produced today's PRD
outage (Deploy PRD run 26441824314
after PR #18 merged): the event-driven scanner (#84) subscribed to
Mutinynet block events while HTTP pointed at the Mainnet
electrs-mainnet, every get_block_txids 404'd, and
scanner_runtime entered a 5 s HTTP-retry loop that never advances
processed_blocks. Service stayed up, /health/ready reported
green, no chain ingestion happened.

This PR makes both URLs required env vars when IS_MAINNET=true,
mirroring the existing PUBLISHER_KEY / USERNAME_DOMAIN /
DATABASE_URL panic-on-missing idiom already in lib.rs. Empty /
whitespace-only values are rejected the same way so a stray
ESPLORA_URL= line in a compose file panics with the same
diagnostic instead of leaving the field as "".

Why this is a real outage, not a smoke-test miss

Probe Sees the bug?
/health no (HTTP listener bound)
/health/ready no (only checks DB pool + Esplora REST reachability)
Smoke test /api/info returns 200 no
Scanner forward progress yes — stuck on the same Mutinynet block hash forever

The deploy-prd.yaml smoke test gated on /api/info 200 and let the
broken build through. The panic in this PR puts the failure on the
bootstrap path so a future broken Mainnet config never reaches
listening at all.

DEV / Mutinynet impact

None. IS_MAINNET=false (the unset default) keeps the existing
Mutinynet defaults for both URLs, the pre-push hook still passes
without setting any new env var, and the CI node-tests job
(.github/workflows/ci.yaml) does not set IS_MAINNET=true.

Implementation

  • Pulled the env-resolution out of the lazy_static! block into a
    pure pub fn build_network_config_from_env<F>(env: F) so the
    panic rules are unit-testable via a fake-env closure instead of
    std::env::set_var (which would poison the NETWORK_CONFIG cell
    across other tests in the same binary).
  • IS_MAINNET=true branches:
    • ESPLORA_URL required, non-empty
    • ESPLORA_WS_URL required, non-empty
  • IS_MAINNET=false branches: unchanged Mutinynet defaults.
  • Diagnostic panic messages name the env var, explain the Mainnet
    rationale, and point at concrete working values
    (http://electrs-mainnet:3000, https://mempool.space/api,
    wss://mempool.space/api/v1/ws).

Test plan

  • cargo fmt --all --check clean
  • cargo clippy -p node -p shared -- -D warnings clean
  • cargo clippy -p node --all-features -- -D warnings clean
  • cargo clippy -p zkcoins-program-plonky2 -p zkcoins-prover-plonky2 --lib -- -D warnings clean
  • cargo check --workspace --all-features clean
  • 7 new #[test]s in node/src/main_tests.rs covering:
    • Default to Mutinynet when IS_MAINNET unset
    • Default to Mutinynet on non-"true" IS_MAINNET (e.g. "1")
    • DEV with explicit URL + WS-URL + NETWORK_NAME overrides
    • Full Mainnet with both URLs set
    • Mainnet with explicit NETWORK_NAME override
    • #[should_panic] on Mainnet missing ESPLORA_URL
    • #[should_panic] on Mainnet empty ESPLORA_URL ("")
    • #[should_panic] on Mainnet missing ESPLORA_WS_URL
    • #[should_panic] on Mainnet whitespace-only ESPLORA_WS_URL
  • CI lint-and-build green on this PR
  • After ci:full label or auto-release-PR sync, the M3 Ultra
    node-tests + coverage gate green

Defense-in-depth pairing

Server-side compose for the DFX dfxprd Mainnet stack will set
ESPLORA_WS_URL=wss://mempool.space/api/v1/ws explicitly so the
deploy is self-documenting and matches this new contract:
https://github.com/DFXServer/server/pull/250

Out of scope / follow-ups

  • scanner_ws::ScannerWsConfig::from_env and publisher.rs (lines
    440, 722) still call std::env::var("ESPLORA_WS_URL")
    independently with the Mutinynet fallback. In the production
    binary main.rs dereferences NETWORK_CONFIG during bootstrap
    before any of those sites runs, so the panic in this builder
    fires first and the structural bypass is unreachable today.
    Having those sites consume NETWORK_CONFIG.ws_url (or take an
    explicit &EsploraConfig) directly is a separate refactor and
    worth a follow-up to close the bypass for future entry points.
  • Pre-existing repo-wide gap: no mod tests in the crate carries
    #[cfg_attr(coverage_nightly, coverage(off))] (CONTRIBUTING.md
    §7.10). Not introduced by this PR; tracking separately.
  • deploy-prd.yaml is missing the
    ZKCOINS_E2E_ALLOW_FEATURE_TRIMMED_SERVER: "true" env on its
    api-e2e job — tracked as issue fix(ci): set ZKCOINS_E2E_ALLOW_FEATURE_TRIMMED_SERVER on PRD api-e2e #114, separate PR.

`NETWORK_CONFIG` silently fell back to the Mutinynet defaults
(`https://mutinynet.com/api`, `wss://mutinynet.com/api/v1/ws`) when
the env var was missing, regardless of `IS_MAINNET`. On DEV that
matches the chain. On Mainnet it's a silent footgun:

- An HTTP-only mismatch panics quickly on the first publisher
  round-trip and the operator sees the breakage immediately.
- The new event-driven scanner (#84) instead subscribes to Mutinynet
  block events and tries to fetch them from the Mainnet HTTP Esplora.
  Every `get_block_txids` returns 404 and `scanner_runtime` enters a
  5 s HTTP-retry loop that never updates `processed_blocks`: the
  service stays up, `/health/ready` reports green, no chain ingestion
  happens, no on-chain mint or send commit is ever picked up. The
  binary never self-heals after restart because no env-derived state
  has changed.

`ESPLORA_URL` and `ESPLORA_WS_URL` are now both **required env vars
when `IS_MAINNET=true`** (panic with diagnostic message, mirroring
the existing `PUBLISHER_KEY` / `USERNAME_DOMAIN` / `DATABASE_URL`
idiom in the same file). Empty / whitespace-only values are treated
as unset so a `ESPLORA_URL=` line in a compose file panics with the
same message instead of leaving `EsploraConfig.url = ""`.

The `IS_MAINNET=false` (DEV / Mutinynet) path is unchanged — both
URLs keep their Mutinynet defaults, the pre-push hook and the M3
Ultra coverage gate are unaffected.

Implementation: pulled the env-resolution out of the `lazy_static!`
block into a pure `build_network_config_from_env<F>(env: F)` so the
panic rules are unit-testable without `std::env::set_var` (which
would poison the `NETWORK_CONFIG` cell across tests in the same
binary). Seven new `#[test]`s cover the headline shapes plus the
empty-string and whitespace-only rejection paths.

The guard is enforced at the `NETWORK_CONFIG` access path only.
`scanner_ws::ScannerWsConfig::from_env` and `publisher.rs` still
read `ESPLORA_WS_URL` independently with the Mutinynet fallback —
in the main binary the panic in this builder fires first (main.rs
dereferences `NETWORK_CONFIG` during bootstrap, before any scanner
or publisher env read), so the structural bypass is unreachable
today. Closing that bypass by having those sites consume
`NETWORK_CONFIG.ws_url` directly is tracked as a follow-up.
@TaprootFreak
TaprootFreak marked this pull request as ready for review May 26, 2026 09:49
@TaprootFreak
TaprootFreak changed the base branch from develop to release/mainnet-hardening May 26, 2026 16:30
@TaprootFreak
TaprootFreak merged commit 7805692 into release/mainnet-hardening May 26, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant