Skip to content

Gracefully handle missing availability service - #13

Open
lkeyson-earthscope wants to merge 28 commits into
mainfrom
handle-availability
Open

Gracefully handle missing availability service#13
lkeyson-earthscope wants to merge 28 commits into
mainfrom
handle-availability

Conversation

@lkeyson-earthscope

@lkeyson-earthscope lkeyson-earthscope commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

The FDSN fdsnws-availability service and MUSTANG's ts_* time-series metrics (e.g. ts_percent_availability_total, ts_num_gaps) are currently down for an undetermined period. Though their return will likely occur at the same time, these two services could go down and come back independently of each other. This PR lets PIQQA reports keep generating while either or both are unavailable, and switch back onto them automatically (no manual flag/config change) once they recover.

What changed
Independent health checks, one per service, run once per report:

  • doAvailability() probes fdsnws-availability directly (PIQQA.py) and only treats it as down on an explicit HTTP 410 (or another HTTP error / connection failure) - no longer relies on a getcode() == 410 check that could never actually be reached (urlopen raises HTTPError for 4xx/5xx rather than returning a response object).
  • reportUtils.checkTsMetricsExist() (new) independently probes MUSTANG for ts_percent_availability_total against a stable reference station (IU.ANMO), so we don't conflate "availability service is down" with "ts_ metrics are down" - they're detected and handled separately.

Fallback paths when a service is down:

  • reportUtils.get_extents_from_mustang() (new) computes data start/end extents from MUSTANG's percent_availability measurements when fdsnws-availability is unreachable, replacing the availability-service lookup used to determine data start/end for PDF and spectrogram plots. If it cannot retrieve percent_availability metrics, it will use the report dates.
  • reportUtils.getDataStartEnd() (new) wraps the "get start/end for a target" logic (previously duplicated 4x across doPDFs/doSpectrograms) and picks between getAvailability() and get_extents_from_mustang() based on the availability-service flag.
  • doAvailability() skips straight to the percent_availability fallback (instead of attempting ts_percent_availability_total and only falling back after a failed retry-with-sleep) when checkTsMetricsExist() has already determined ts_ metrics are down — avoids ~5-10s of wasted retries per channel group.
  • doBoxPlots() drops ts_num_gaps_total and substitutes num_gaps for ts_num_gaps in the requested metric list when ts_ metrics are down.

Updated urls:

  • service.iris.edu --> service.earthscope.org

This MR also bumps python to 3.12, off of end-of-life 3.9.

Example report that was run on the latest code:
UW_2026.zip

@lkeyson-earthscope
lkeyson-earthscope marked this pull request as draft July 28, 2026 14:21
@lkeyson-earthscope
lkeyson-earthscope marked this pull request as ready for review July 28, 2026 16:37
@lkeyson-earthscope lkeyson-earthscope self-assigned this Jul 28, 2026

@gillian-earthscope gillian-earthscope left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly just comments about updating IRIS references to EarthScope

Comment thread reportUtils.py Outdated
Comment thread reportUtils.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Comment thread PIQQA.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants