Skip to content

GODEEEP CF registry: config-declared retrieval with no fallback paths - #806

Merged
ktehranchi merged 1 commit into
developfrom
godeeep-cf-registry
Aug 31, 2026
Merged

ktehranchi merged 1 commit into
developfrom
godeeep-cf-registry

Conversation

@ktehranchi

Copy link
Copy Markdown
Collaborator

Closes #803.

What

Replaces the hardcoded Zenodo record table and the download hidden inside build_renewable_profiles with a single declarative registry for GODEEEP compressed capacity-factor files:

  • godeeep_cf_registry.py — sources (local mirror / Zenodo) are declared in config["godeeep_cf_registry"] with exactly the (dataset, year) pairs they hold. Sources are walked in config order; the first one declaring the request wins. There are no fallback paths: no default hub height, no nearest-year substitution, no unscreened-variant swap. All configured scenario × year × technology combinations are resolved eagerly at snakemake parse time; failures aggregate into one error naming the dataset key, the requested year, and the available years per source.
  • rule retrieve_godeeep_cf — retrieval is now an explicit DAG step. Local-mirror hits are symlinked (or copied via copy_local) after sha256 verification against the mirror's SHA256SUMS; Zenodo hits download with checksum validation and retries. build_renewable_profiles consumes snakemake.input.godeeep_cf and never picks a file itself.
  • zenodo_downloader.py — every failure raises ZenodoDownloadError; nothing returns None or leaves partial files. The unregistered wind_125m_rcp45hotter record (20127545, HTTP 404) is declared with an empty year list so requesting it fails honestly instead of via the legacy silent-None path.
  • Compression pipeline (nrel_exclusion/compress_godeeep_*) used to build the 172-file sha256-verified mirror (solar + wind 80/100/125 m × 1980–2022 historical), plus 68 unit tests covering the registry, downloader, and compressor.
  • Docs — retrieval mechanism and a pre-aggregated data availability matrix (which datasets/scenarios/years exist on the local mirror vs. Zenodo).

Verification

  • 81 fast tests pass (68 new).
  • Parse-time probes: invalid hub height, out-of-range year, and the unpublished rcp45hotter wind record each fail loudly with actionable messages listing per-source availability.
  • End-to-end on Sherlock: historical 2019 / _125m / western / s75 build_renewable_profiles run retrieves via symlink + sha256 from the mirror and produces sane profiles (8760 hourly 2019 snapshots, CF ∈ [0,1], zero NaNs, full bus coverage; onwind's 73/75 buses reflect zero NREL wind capacity in LA + Inyo counties, pre-existing behavior).

Notes for reviewers

  • resolve_cf validates from config declarations alone — it does not stat the local root. An off-Sherlock user keeping the default config passes parse-time validation and fails at retrieve time with a missing-file error; docs tell them to remove/repoint the local source. Guidance, not enforcement, in this PR.
  • The mem_mb=32000 request for godeeep build_renewable_profiles measured ~17× actual peak (1.9 GB) at s75/western — left as-is since it may be sized for finer resolutions; happy to right-size.
  • 80 m wind data exists in the mirror archive but is deliberately not wired into VALID_WIND_HEIGHTS — the _80m silent default was the original GODEEEP: historical weather years other than 2012 cannot receive NREL land-access screening (unscreened fallback profiles) #803 bug; enabling it explicitly can be a follow-up.

🤖 Generated with Claude Code

…#803)

Replace the hardcoded Zenodo record table and in-script download inside
build_renewable_profiles with a declarative registry:

- godeeep_cf_registry.py: config-declared sources (local mirror / Zenodo),
  first source declaring the requested (dataset, year) wins, no fallbacks
  (no default hub height, no nearest-year substitution). All configured
  combinations resolve eagerly at snakemake parse time; failures aggregate
  into one error naming the dataset key, requested year and available
  years per source.
- rule retrieve_godeeep_cf + retrieve_godeeep_cf.py: dedicated DAG step
  that symlinks (or copies) local-mirror hits after verifying them against
  the mirror's SHA256SUMS manifest, or downloads from the declared Zenodo
  record with checksum validation.
- build_renewable_profiles.py consumes snakemake.input.godeeep_cf instead
  of downloading; year/scenario resolution shares the registry module.
- zenodo_downloader.py: every failure raises ZenodoDownloadError with
  retries; nothing returns None or leaves partial files.
- config.common.yaml declares the Oak mirror (historical 1980-2022, solar
  + wind 100m/125m) and the published Zenodo records; the unregistered
  wind_125m rcp45hotter record (404) is declared with an empty year list
  so requesting it fails honestly.
- Compression pipeline (nrel_exclusion/compress_godeeep_* helpers) and
  tests for the registry, downloader and compressor (68 tests).
- Docs: retrieval mechanism + pre-aggregated data availability matrix.

Verified end to end on Sherlock: historical 2019 _125m western s75 run
retrieves via symlink+sha256 from the mirror and produces sane profiles
(8760 hourly snapshots, CF within [0,1], full bus coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ktehranchi
ktehranchi merged commit cc33423 into develop Aug 31, 2026
1 of 3 checks passed
ktehranchi added a commit that referenced this pull request Aug 31, 2026
Resolves conflicts from #806 (GODEEEP CF registry, no fallback paths).
Resolution policy: the registry wins everywhere it overlaps —

- build_renewable_profiles.py / zenodo_downloader.py: theirs wholesale;
  the interim unscreened aggregated fallback (and its opt-in flag,
  configtable row, docs and 52 tests) is retired. Screened historical
  years 1980-2022 now come from the registry's SHA256-verified mirror,
  which resolves #803 properly.
- config.common.yaml: registry semantics for godeeep_wind_height (no
  default); our PUDL v2026.8.0 pin and nrel_caps_reassign block kept.
- docs: availability rewritten registry-era in data-generators.md,
  config-configuration.md and california-model.md (source × years
  matrix; every SERVM demand year 2000-2022 now pairs with a screened
  profile of the same year).
- config.california.yaml switched to renewable_scenarios: ['historical']
  + WY2019: the registry validates configured (dataset, year) pairs at
  parse time, and the previous rcp85cooler pairing with 2035/2045
  horizons (a documented landmine) became a hard parse failure. Verified
  parsing (35-job data_model DAG).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ktehranchi added a commit that referenced this pull request Aug 31, 2026
…organization

Conflicts were two comment blocks in config.default.yaml where the
reorg's one-owning-file pointer notes met #806's new registry prose;
resolved by combining both texts, keeping renewable.dataset owned by
config.common.yaml. The registry block itself and the _125m wind-height
change auto-merged into common.yaml alongside the reference land-access
default. Also declares the new top-level godeeep_cf_registry key in the
config schema (permissive - the registry module does its own strict
validation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant