GeoNet S3 archive + live Earth2Studio source with mandatory calibration - #5
Conversation
waveforms/miniseed/{Y}/{Y}.{DDD}/{STA}.{NET}/{Y}.{DDD}.{STA}.{LOC}-{CHA}.{NET}.D
per-channel day objects in geonet-open-data (ap-southeast-2), verified
live: exact-key fetch (NZ.WEL.10.HHZ, 13.2 MB), HH? wildcard discovery
(three components), and station listing. NZ routes to geonet.
Also corrects the NCEDC region to us-west-2 (was us-east-2; worked via
S3 redirect but paid a cross-region hop on every request).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SeisfetchLiveSource is time-indexed like every other Earth2Studio source (GFS, ERA5, ...): call it with timestamps and it fetches day objects on demand from EarthScope/SCEDC/NCEDC/GeoNet with a small day cache, trims sample-precise windows, and always returns physical units — gain correction by default (channel Scale from the FDSN station text service, one stdlib-parsed HTTP request via new ChannelEpoch/parse_channel_text), or full response deconvolution via contrib.response. Raw counts are never returned. The existing adapters' async fetch is now real (asyncio.to_thread) and SeismicDataFrameSource can auto-fill station coordinates. No band/RMS feature extraction by design — derived low-rate feature series are a later, separate interface. Verified live on GeoNet: NZ.WEL.10.HHZ window in m/s with real Wellington coordinates; unit tests cover parsing, epoch selection, gain mechanics, NaN gap fill, cache, and async. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR extends seisfetch with a new GeoNet (NZ) S3 backend and adds a time-indexed Earth2Studio DataSource (SeisfetchLiveSource) that fetches waveform windows on-demand with mandatory physical-unit calibration, supported by new FDSN station-text parsing utilities.
Changes:
- Added GeoNet S3 archive support (keying, discovery, station listing) and routed
NZnetworks to GeoNet; corrected NCEDC bucket region. - Introduced
ChannelEpoch+parse_channel_text()for lean station-text parsing, enabling gain-based calibration without StationXML. - Added
SeisfetchLiveSource(time-indexed, cached day bundles, NaN gap fill) and made existing Earth2 adapters’fetch()truly async viaasyncio.to_thread, with new tests and changelog entries.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_s3.py | Adds GeoNet key-format and routing tests. |
| tests/test_earth2_live.py | New tests for station-text parsing, mandatory calibration, window assembly, and true async fetch. |
| seisfetch/s3.py | Adds GeoNet backend (key fn + discovery + station listing), updates routing, fixes NCEDC region. |
| seisfetch/fdsn.py | Adds ChannelEpoch and parse_channel_text() for station text parsing. |
| seisfetch/earth2.py | Makes adapter fetch() truly async; adds auto station coords; introduces SeisfetchLiveSource + metadata plumbing. |
| CHANGELOG.md | Documents new GeoNet support, live source, station-text parsing, and async fetch change. |
Suppressed comments (2)
seisfetch/earth2.py:509
- _day_bundle() updates _day_cache and evicts entries without any locking. With async/to_thread enabled, concurrent calls can interleave and cause cache inconsistency or exceptions. Protect cache read/write+eviction with the instance lock.
key = (nslc, day.isoformat())
if key not in self._day_cache:
net, sta, loc, cha = nslc.split(".")
raw = S3OpenClient(datacenter=self._datacenter).get_raw(
net, sta, day.isoformat(), location=loc, channel=cha
)
self._day_cache[key] = parse_mseed(raw)
while len(self._day_cache) > self._cache_days * len(self.channels):
self._day_cache.pop(next(iter(self._day_cache)))
return self._day_cache[key]
seisfetch/earth2.py:553
- _deconvolve() also ignores the datacenter override when selecting the station provider (it always uses route_network(net)). For consistency with waveform fetching, the provider should follow the forced datacenter when one is set.
if nslc not in self._resp:
net, sta, loc, cha = nslc.split(".")
xml = FDSNClient(
provider=_DC_TO_PROVIDER[route_network(net)]
).get_station_text(
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def covers(self, time_iso: str) -> bool: | ||
| t = str(time_iso) | ||
| return self.start <= t and (self.end is None or t <= self.end) |
There was a problem hiding this comment.
Fixed in ae54c9b: covers() now parses both sides to aware UTC datetimes (stdlib _iso_to_utc, handling Z, numeric offsets, and any fraction width). Regression test includes the exact failing case — a -05:00 timestamp landing on the epoch-boundary instant selects the modern epoch, where string comparison picked the old one.
| t_iso = t0.isoformat() | ||
| if self.calibrate == "gain": | ||
| return out / self._gain(nslc, t_iso) | ||
| return self._deconvolve(nslc, out, fs, t_iso) |
There was a problem hiding this comment.
Fixed in ae54c9b: _iso_utc() normalizes to UTC and strips the offset suffix before epoch selection and StationXML parsing.
| def _gain(self, nslc: str, time_iso: str) -> float: | ||
| for ep in self._epochs(nslc): | ||
| if ep.covers(time_iso): | ||
| if not ep.scale: | ||
| raise LookupError( | ||
| f"{nslc}: channel epoch has no Scale (sensitivity) — " | ||
| "cannot calibrate" | ||
| ) | ||
| return ep.scale | ||
| raise LookupError(f"{nslc}: no channel epoch covers {time_iso}") |
There was a problem hiding this comment.
Fixed in ae54c9b: scale is None (missing) and scale == 0.0 (defective metadata) are now distinct loud failures — 0.0 is never treated as absent, and never divided by.
| def _geonet_key(network, station, year, doy, location="", channel="", **_): | ||
| """GeoNet (New Zealand): one object per channel-day. | ||
|
|
||
| Layout (verified on the live bucket, 2026-08-07): | ||
| ``waveforms/miniseed/{Y}/{Y}.{DDD}/{STA}.{NET}/{Y}.{DDD}.{STA}.{LOC}-{CHA}.{NET}.D`` | ||
| e.g. ``waveforms/miniseed/2022/2022.002/WEL.NZ/2022.002.WEL.10-HHZ.NZ.D``. | ||
| GeoNet channels always carry a numeric location code (10, 20, ...). | ||
| """ | ||
| loc = location if location and location != "*" else "" | ||
| return ( | ||
| f"waveforms/miniseed/{year}/{year}.{doy:03d}/{station}.{network}/" | ||
| f"{year}.{doy:03d}.{station}.{loc}-{channel}.{network}.D" | ||
| ) |
There was a problem hiding this comment.
Fixed in ae54c9b: _geonet_key raises ValueError on blank location with guidance to use location='*' (wildcard discovery) — no more guaranteed-404 keys.
| attrs={ | ||
| "units": "m/s (gain-corrected)" | ||
| if self.calibrate == "gain" | ||
| else "m/s (response-removed)", | ||
| "calibration": self.calibrate, |
There was a problem hiding this comment.
Fixed in ae54c9b: the units attr now reports the selected epoch's ScaleUnits (accelerometers show M/S**2 (gain-corrected); heterogeneous variable sets are labeled mixed: ...). Hard-coded m/s remains only for calibrate='response', where the deconvolution output really is velocity.
| self.channels = list(channels) | ||
| self.window_s = float(window_s) | ||
| self.calibrate = calibrate | ||
| self._datacenter = datacenter | ||
| self._cache_days = cache_days | ||
| self._day_cache: dict = {} # (nslc, date) -> TraceBundle | ||
| self._meta: dict = {} # nslc -> list[ChannelEpoch] | ||
| self._resp: dict = {} # nslc -> ChannelResponse (calibrate="response") |
There was a problem hiding this comment.
Fixed in ae54c9b: one coarse lock guards _day_cache/_meta/_resp. Day fetches use double-checked population so the network I/O stays outside the lock — concurrent fetch() calls can still overlap downloads across channels without racing eviction.
| if nslc not in self._meta: | ||
| net, sta, loc, cha = nslc.split(".") | ||
| self._meta[nslc] = channel_metadata( | ||
| net, sta, loc, cha, "1970-01-01", "2100-01-01" | ||
| ) | ||
| if not self._meta[nslc]: | ||
| raise LookupError(f"no channel metadata for {nslc}") |
There was a problem hiding this comment.
Fixed in ae54c9b: _provider() honors the datacenter override for both channel-text metadata and StationXML fetches, so calibration metadata always comes from the same archive as the waveforms. Test asserts the override wins over network routing.
- ChannelEpoch.covers compares aware UTC datetimes, not ISO strings — offsets and fraction-width variants now select the right epoch (a -05:00 timestamp equal to the epoch boundary instant was the failing case); SeisfetchLiveSource passes normalized UTC ISO times downstream. - Scale=None (missing) and Scale=0 (defective) are now distinct loud failures; 0.0 is never treated as absent. - _geonet_key raises on blank location with guidance (GeoNet keys always carry a location code; wildcard discovery finds it). - DataArray units come from the channel's ScaleUnits metadata — accelerometer channels report M/S**2, mixed sets say so. - All three caches behind one lock (fetch() runs in worker threads); day fetches stay outside the lock via double-checked population. - Metadata provider honors the datacenter override instead of always routing by network, so calibration metadata and waveforms always come from the same archive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What
Two lean additions with zero new core dependencies (core stays numpy + pymseed + boto3; earth2studio/xarray remain optional extras):
GeoNet (New Zealand) S3 archive
geonet-open-data(ap-southeast-2), layout verified on the live bucket:waveforms/miniseed/{Y}/{Y}.{DDD}/{STA}.{NET}/{Y}.{DDD}.{STA}.{LOC}-{CHA}.{NET}.DNZauto-routesHH?discovery of all three components, station censusSeisfetchLiveSource— a genuine Earth2Studio DataSourceThe existing adapters wrapped already-fetched bundles; every real Earth2Studio source is time-indexed. The new source is called with timestamps and fetches on demand from any of the four archives (day-bundle cache, sample-precise window trim, NaN gap fill).
Physical units are mandatory — raw counts are never returned:
calibrate="gain"(default): divides by the channel Scale from the FDSN station text service — one stdlib-parsed HTTP request (ChannelEpoch/parse_channel_text, new inseisfetch.fdsn), no StationXML, no scipycalibrate="response": full deconvolution viacontrib.responseAlso: the adapters'
fetchis now genuinely async (asyncio.to_thread), andSeismicDataFrameSource(auto_coords=True)fills station lat/lon from the same text service.Deliberately not included (per design discussion): band/RMS feature extraction — derived low-rate feature series belong in a later, separate interface, not the raw-waveform source.
Verification
NZ.WEL.10.HHZ10-min window at 2.6e-6 m/s with real Wellington coordinates from service.geonet.org.nz; cached second window is free🤖 Generated with Claude Code