OPRConnection caches local files unnecessarily during catalog builds (i.e., copies the full set of data files)

[espg]

Problem

Running bulk catalog generation via scripts/build_catalog.py against local radar data fills disk with redundant copies. A partial reprocessing run accumulated 729GB in ./radar_cache/ before running out of disk space.

The root cause is that OPRConnection._open_file() unconditionally wraps every URL in fsspec's filecache:: prefix when cache_dir is set — including paths that are already on the local filesystem. This tells fsspec to copy the file into the cache directory before reading it. For local-to-local access, this is a no-op copy that doubles disk usage.

Call chain

build_catalog.py
  └─ create_items_from_flight_data(local_path)     [stac/catalog.py:193]
       └─ extract_item_metadata(local_path)         [stac/metadata.py:131-132]
            └─ OPRConnection(cache_dir="radar_cache")   ← hardcoded
            └─ opr.load_frame_url(local_path)
                 └─ _open_file(local_path)           [opr_access.py:132-137]
                      └─ fsspec.open_local("filecache::/data/.../Data_001.mat", ...)
                           └─ COPIES 20MB file to ./radar_cache/<hash>
                           └─ Returns cached path

What it should do

Local paths should be opened directly — no fsspec wrapping, no copy. Remote URLs (https://, s3://, gs://) should keep using filecache:: as before; that's the correct behavior for notebooks and remote data access.

Scale

~2000 files/campaign × ~20MB each × dozens of campaigns = 729GB of redundant copies of data that was already on local disk. The catalog output is ~5MB total.

Additional issue

extract_item_metadata() hardcodes cache_dir="radar_cache" with no config knob and no way to disable caching. Even if _open_file() is fixed for local paths, this hardcoded value should be removed or parameterized.

Affected code

• src/xopr/opr_access.py — _open_file() (lines 132-137), __init__ (lines 107-112)
• src/xopr/stac/metadata.py — extract_item_metadata() (line 131)

What is NOT affected

• All 6 docs notebooks pass remote URLs with cache_dir — correct behavior, unchanged.
• test_cache_data() validates remote caching — unchanged.
• build_layer_parquet.py uses remote URLs — unchanged.
• load_frame_url(), load_layers_file(), format dispatch (HDF5 vs MATLAB) — unchanged.

[espg]

Plan: fix local-path caching in OPRConnection

Scope

One change with two touch points. No new abstractions, no API changes, no format-dispatch refactoring. load_frame_url() keeps doing exactly what it does today — it just gets handed the real local path instead of a cache copy of the real local path.

Change 1: _open_file() detects local paths

File: src/xopr/opr_access.py, _open_file() method (lines 132-137)

Current:

def _open_file(self, url):
    if self.fsspec_url_prefix:
        return fsspec.open_local(f"{self.fsspec_url_prefix}{url}", filecache=self.fsspec_cache_kwargs)
    else:
        return fsspec.open_local(f"simplecache::{url}", simplecache=self.fsspec_cache_kwargs)

Proposed:

def _open_file(self, url):
    # Local paths don't need fsspec caching — they're already on disk.
    if not url.startswith(('http://', 'https://', 's3://', 'gs://')):
        return url

    if self.fsspec_url_prefix:
        return fsspec.open_local(f"{self.fsspec_url_prefix}{url}", filecache=self.fsspec_cache_kwargs)
    else:
        return fsspec.open_local(f"simplecache::{url}", simplecache=self.fsspec_cache_kwargs)

What changes: local filesystem paths are returned as-is. Remote URLs keep the existing filecache:: / simplecache:: behavior. The scheme check (http://, https://, s3://, gs://) covers every remote protocol OPR data is served over.

What doesn't change: load_frame_url(), load_layers_file(), _get_layers_files(), format dispatch (HDF5 vs MATLAB v5 vs v7.3), xarray loading, parameter extraction — all unchanged. They receive a path string either way.

Change 2: Remove hardcoded cache_dir in extract_item_metadata()

File: src/xopr/stac/metadata.py, line 131

Current:

opr = OPRConnection(cache_dir="radar_cache")

Proposed:

opr = OPRConnection()

With Change 1 in place, local paths bypass caching regardless of cache_dir. But the hardcoded "radar_cache" is still wrong in principle — it creates the directory unconditionally and sets up fsspec state that's never used for local paths. Removing it makes the intent clear: metadata extraction doesn't need a cache.

If a future caller passes a remote URL to extract_item_metadata(), fsspec will use simplecache:: (temporary, no persistent disk state) — which is the correct default for one-shot metadata reads.

Blast radius assessment

Component	Impact
_open_file()	Changed — adds a 2-line early return for local paths
extract_item_metadata()	Changed — removes hardcoded cache_dir
load_frame_url()	Unchanged — still receives a path string, still does format dispatch
load_frame() / load_frames()	Unchanged — pass remote URLs from STAC items
get_layers() / _get_layers_files()	Unchanged — pass remote URLs
All 6 notebooks	Unchanged — all use remote URLs with cache_dir
test_cache_data()	Unchanged — tests remote URL caching, still works
test_convert_single_frame()	Unchanged — uses remote URLs from STAC items
build_layer_parquet.py	Unchanged — uses remote URLs
build_catalog.py	Fixed — local paths no longer cached

Testing

• Existing test_cache_data() covers remote caching (no change needed).
• Add one test: OPRConnection(cache_dir=tmp_path)._open_file("/local/path.mat") returns the path as-is without creating cache files.
• Run full catalog build on one season, verify ./radar_cache/ is not created.

[espg]

closed by #73