Phase 1: Foundation Layer - Centralized Config, Exceptions, and Tests by SBFRF · Pull Request #26 · SBFRF/murgtools

SBFRF · 2026-02-01T13:46:28Z

Summary

Centralized Configuration: Created murgtools/config.py with all THREDDS URLs, STAC endpoints, Argus base URL, NCEP URL, and helper functions for server selection
Custom Exceptions: Added murgtools/exceptions.py with exception hierarchy (MurgToolsError, InvalidGaugeError, DataNotFoundError, NetworkError, InvalidParameterError, InvalidTimeRangeError)
Datetime Utility: Added roundDatetimeToInterval() to sblib.py for Argus 30-minute rounding
Endpoint Tests: Created tests/test_config.py with 27 tests including actual data download verification from THREDDS, STAC, Argus, and NCEP servers
New Gauge Support: Scanned THREDDS server and added support for newly discovered gauges:
- Wave gauges: sig940-400, sig940-600, awac-jpier-11m, waverider-17m-1D, waverider-20m-1d
- Current meters: sig769-300, sig940-300, sig940-400, sig940-600, awac-jpier-11m, awac-5m
- Fixed paros naming inconsistencies (supports both paros940-200 and paros-200-940m)

Test plan

All 119 tests pass (pytest tests/ -v)
Config values tests verify URL formats and types
Server selection helper tests cover FRF/CHL network detection
Endpoint reachability tests verify servers respond
Data download tests verify actual NetCDF, STAC JSON, and image data
New gauge URL lookup tests verify all new gauges are recognized

Related Issues

Closes #21 - URL endpoints are now centralized in config.py and new gauges discovered on THREDDS have been added

🤖 Generated with Claude Code

This commit establishes the shared infrastructure for the murgtools refactoring effort: Task 1.1 - Centralized Configuration (config.py): - Move all hardcoded THREDDS URLs to single module - Add STAC, Argus, and NCEP endpoint URLs - Add get_thredds_server() helper for auto-selecting server based on network - Add network config constants (timeouts, IP prefixes) Task 1.2 - Custom Exception Hierarchy (exceptions.py): - Add MurgToolsError base exception - Add InvalidGaugeError with valid_gauges context - Add DataNotFoundError, NetworkError, InvalidParameterError - Update getDataFRF.py to use InvalidGaugeError instead of NameError Task 1.3 - Datetime Rounding Utility (sblib.py): - Add roundDatetimeToInterval() function for 30-min Argus rounding - Replace duplicated rounding code in getArgusImagery and findArgusImagery Task 1.4 - Endpoint Verification Tests (test_config.py): - Add unit tests for config values and helper functions - Add integration tests that verify endpoints are reachable (HEAD requests) - Add data download tests that actually retrieve and validate data: - THREDDS: Opens NetCDF file and verifies variables - STAC: POSTs search query and verifies JSON response structure - Argus: Downloads actual image and verifies numpy array - NCEP: Verifies directory listing is accessible All 102 tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Scanned THREDDS server and added support for newly discovered gauges: Wave gauges added: - sig940-400, sig940-600 (Nortek Signature profilers) - awac-jpier-11m (Jetty pier AWAC) - waverider-17m-1D, waverider-20m-1d (new waveriders) Current meters added: - sig769-300, sig940-300, sig940-400, sig940-600 (Signature profilers) - awac-jpier-11m, awac-5m Also fixed: - Paros naming: now supports both paros940-200 and paros-200-940m conventions - Updated waveGaugeList, directionalWaveGaugeList, currentsGaugeList - Removed skip decorator from test_can_reach_chl_thredds - Changed getCurrents to raise InvalidGaugeError instead of AssertionError Closes #21 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Two critical bugs fixed: 1. NCML indexing bug in getWaveData(): - When querying recent data from large NCML aggregation files, getnc() would return a subset of records with a relative offset - getWaveData() used these relative indices directly on the full netCDF, resulting in reading data from years earlier than requested - Fix: Capture indexRef from getnc() and compute absolute indices for netCDF file access while keeping relative indices for allEpoch access 2. Satellite imagery bounds calculation: - getSatelliteImagery() used STAC item['bbox'] which represents the full tile extent, not the actual GeoTIFF extent - This caused crop coordinates to be shifted by several kilometers - Fix: Extract actual extent from GeoTIFF tags (ModelTiepointTag and ModelPixelScaleTag), convert from UTM to lat/lon, then use for cropping Also includes: - Regression tests for NCML indexing to verify timestamps match query range - Minor fix: Remove x-axis label from top plot in test_wave_and_imagery.py Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

This pull request establishes a foundation layer for the murgtools package by centralizing configuration, introducing custom exceptions, and adding comprehensive test coverage. The changes address issue #21 by updating gauge URL endpoints and discovering new gauges available on the THREDDS server.

Changes:

Created centralized configuration module (murgtools/config.py) containing all THREDDS URLs, STAC endpoints, and other configuration constants
Implemented custom exception hierarchy with MurgToolsError as base class and specific exceptions for common error scenarios
Added roundDatetimeToInterval() utility function for standardized datetime rounding (particularly for Argus 30-minute intervals)
Integrated configuration module across existing codebase, replacing hardcoded URLs
Added support for 10 newly discovered gauges (signature profilers, AWAC jetty pier, new waveriders, paros sensors)
Created 27 comprehensive tests for configuration values, endpoint reachability, and actual data downloads

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
murgtools/config.py	New centralized configuration module with THREDDS URLs, STAC endpoints, network detection helpers
murgtools/exceptions.py	New custom exception hierarchy for better error handling
murgtools/getdata/getDataFRF.py	Integrated config module, custom exceptions, added 10 new gauge definitions
murgtools/getdata/getOutsideData.py	Replaced hardcoded URLs with config constants
murgtools/utils/sblib.py	Added roundDatetimeToInterval() function for datetime rounding
murgtools/utils/gridTools.py	Replaced hardcoded URLs with config constants
murgtools/utils/Thredds_checker.py	Replaced hardcoded URLs with config constants
tests/test_config.py	New comprehensive test suite with 27 tests for config values and endpoint verification
tests/test_getDataFRF.py	Added 550+ lines of tests for new gauge support and existing functionality
tests/test_integration.py	Removed redundant skip decorator (test already marked slow at class level)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

…satellite-bounds

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

…ll ncfile accesses Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

…b-pr-27-again

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Move inline imports to module level in getOutsideData.py

Fix missing ncfileindex computation in getWaveSpecModel causing AttributeError

…b-pr-27

Move inline imports to module level per PEP 8

Fix inconsistent indexing in getWaveSpecModel and remove unused test variables

Fix NCML indexing bug and satellite imagery bounds

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-02-02T01:52:59Z

@SBFRF I've opened a new pull request, #34, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Copilot · 2026-02-02T01:57:17Z

@SBFRF I've opened a new pull request, #35, to work on those changes. Once the pull request is ready, I'll request review from you.

Add parameter validation to roundDatetimeToInterval

[WIP] Address feedback on Phase 1 Foundation Layer PR

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-02T02:08:46Z

+        from murgtools.getdata.getDataFRF import getObs
+        obs = getObs(d1, d2)
+
+        with pytest.warns(UserWarning, match="getWaveSpec is depreciated"):


Spelling error in comment. The word should be spelled "deprecated" not "depreciated". "Depreciated" refers to financial depreciation, while "deprecated" means something is discouraged or obsolete in programming contexts.

Suggested change

with pytest.warns(UserWarning, match="getWaveSpec is depreciated"):

with pytest.warns(UserWarning, match="getWaveSpec is deprecated"):

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot · 2026-02-02T02:11:47Z

@SBFRF I've opened a new pull request, #37, to work on those changes. Once the pull request is ready, I'll request review from you.

SBFRF and others added 3 commits February 1, 2026 07:52

SBFRF requested a review from Copilot February 2, 2026 01:25

Copilot started reviewing on behalf of SBFRF February 2, 2026 01:25 View session

Merge branch 'main' into refactor/phase1-foundation

adf7215

Copilot AI reviewed Feb 2, 2026

View reviewed changes

Copilot AI and others added 22 commits February 2, 2026 01:35

Initial plan

16925ed

Initial plan

6c1db0d

Update murgtools/getdata/getDataFRF.py

d2c1cc2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update tests/test_ncml_indexing_regression.py

4672489

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Initial plan

6b991c0

Update murgtools/getdata/getDataFRF.py

24190b9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Move pyproj and re imports to top of file

6f74454

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Initial plan

9eb688e

Move imports to top of file and organize per PEP 8

337e9ce

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Merge branch 'refactor/phase1-foundation' into fix/ncml-indexing-and-…

dc1c169

…satellite-bounds

Update murgtools/getdata/getOutsideData.py

ca31039

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Fix getWaveSpecModel to capture indexRef and compute ncfileindex

88bd6f8

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Fix inconsistent indexing in getWaveSpecModel - use ncfileindex for a…

4ed8da2

…ll ncfile accesses Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Merge branch 'fix/ncml-indexing-and-satellite-bounds' into copilot/su…

c43e8e7

…b-pr-27-again

Fix inconsistent indexing and remove unused variables

00629a0

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Merge pull request #29 from SBFRF/copilot/sub-pr-27-again

1b981a9

Move inline imports to module level in getOutsideData.py

Merge pull request #30 from SBFRF/copilot/sub-pr-27-another-one

462985e

Fix missing ncfileindex computation in getWaveSpecModel causing AttributeError

Merge branch 'fix/ncml-indexing-and-satellite-bounds' into copilot/su…

da12d44

…b-pr-27

Merge pull request #28 from SBFRF/copilot/sub-pr-27

f49a642

Move inline imports to module level per PEP 8

Merge pull request #31 from SBFRF/copilot/sub-pr-27-yet-again

bae4feb

Fix inconsistent indexing in getWaveSpecModel and remove unused test variables

Merge pull request #27 from SBFRF/fix/ncml-indexing-and-satellite-bounds

d3e1919

Fix NCML indexing bug and satellite imagery bounds

Update murgtools/getdata/getDataFRF.py

18eaa58

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Initial plan

373568b

Copilot AI mentioned this pull request Feb 2, 2026

Add parameter validation to roundDatetimeToInterval #34

Merged

SBFRF and others added 7 commits February 1, 2026 20:53

Update tests/test_getDataFRF.py

8eb4765

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update murgtools/getdata/getDataFRF.py

fa19570

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update tests/test_getDataFRF.py

7055046

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update murgtools/getdata/getDataFRF.py

0168ee7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update tests/test_getDataFRF.py

03363ff

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Add parameter validation to roundDatetimeToInterval function

d9bf24a

Co-authored-by: SBFRF <8375832+SBFRF@users.noreply.github.com>

Initial plan

4195893

Copilot AI mentioned this pull request Feb 2, 2026

[WIP] Address feedback on Phase 1 Foundation Layer PR #35

Merged

4 tasks

SBFRF added 2 commits February 1, 2026 20:59

Merge pull request #34 from SBFRF/copilot/sub-pr-26

3cceb43

Add parameter validation to roundDatetimeToInterval

Merge pull request #35 from SBFRF/copilot/sub-pr-26-again

531d0f4

[WIP] Address feedback on Phase 1 Foundation Layer PR

SBFRF requested a review from Copilot February 2, 2026 02:03

Copilot started reviewing on behalf of SBFRF February 2, 2026 02:04 View session

Copilot AI reviewed Feb 2, 2026

View reviewed changes

Update murgtools/getdata/getOutsideData.py

5f30064

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI mentioned this pull request Feb 2, 2026

[WIP] Address feedback on centralized config and exceptions implementation #37

Merged

4 tasks

SBFRF merged commit 1ecc337 into main Feb 2, 2026
4 checks passed

SBFRF deleted the refactor/phase1-foundation branch February 2, 2026 02:12

Copilot AI mentioned this pull request Feb 2, 2026

Fix waverider-17m-1d URL case inconsistency #38

Merged

	with pytest.warns(UserWarning, match="getWaveSpec is depreciated"):
	with pytest.warns(UserWarning, match="getWaveSpec is deprecated"):

Conversation

SBFRF commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Feb 2, 2026

Uh oh!

Copilot AI commented Feb 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SBFRF commented Feb 1, 2026 •

edited

Loading