GODEEEP CF registry: config-declared retrieval with no fallback paths - #806
Merged
Merged
Conversation
…#803) Replace the hardcoded Zenodo record table and in-script download inside build_renewable_profiles with a declarative registry: - godeeep_cf_registry.py: config-declared sources (local mirror / Zenodo), first source declaring the requested (dataset, year) wins, no fallbacks (no default hub height, no nearest-year substitution). All configured combinations resolve eagerly at snakemake parse time; failures aggregate into one error naming the dataset key, requested year and available years per source. - rule retrieve_godeeep_cf + retrieve_godeeep_cf.py: dedicated DAG step that symlinks (or copies) local-mirror hits after verifying them against the mirror's SHA256SUMS manifest, or downloads from the declared Zenodo record with checksum validation. - build_renewable_profiles.py consumes snakemake.input.godeeep_cf instead of downloading; year/scenario resolution shares the registry module. - zenodo_downloader.py: every failure raises ZenodoDownloadError with retries; nothing returns None or leaves partial files. - config.common.yaml declares the Oak mirror (historical 1980-2022, solar + wind 100m/125m) and the published Zenodo records; the unregistered wind_125m rcp45hotter record (404) is declared with an empty year list so requesting it fails honestly. - Compression pipeline (nrel_exclusion/compress_godeeep_* helpers) and tests for the registry, downloader and compressor (68 tests). - Docs: retrieval mechanism + pre-aggregated data availability matrix. Verified end to end on Sherlock: historical 2019 _125m western s75 run retrieves via symlink+sha256 from the mirror and produces sane profiles (8760 hourly snapshots, CF within [0,1], full bus coverage). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ktehranchi
added a commit
that referenced
this pull request
Aug 31, 2026
Resolves conflicts from #806 (GODEEEP CF registry, no fallback paths). Resolution policy: the registry wins everywhere it overlaps — - build_renewable_profiles.py / zenodo_downloader.py: theirs wholesale; the interim unscreened aggregated fallback (and its opt-in flag, configtable row, docs and 52 tests) is retired. Screened historical years 1980-2022 now come from the registry's SHA256-verified mirror, which resolves #803 properly. - config.common.yaml: registry semantics for godeeep_wind_height (no default); our PUDL v2026.8.0 pin and nrel_caps_reassign block kept. - docs: availability rewritten registry-era in data-generators.md, config-configuration.md and california-model.md (source × years matrix; every SERVM demand year 2000-2022 now pairs with a screened profile of the same year). - config.california.yaml switched to renewable_scenarios: ['historical'] + WY2019: the registry validates configured (dataset, year) pairs at parse time, and the previous rcp85cooler pairing with 2035/2045 horizons (a documented landmine) became a hard parse failure. Verified parsing (35-job data_model DAG). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ktehranchi
added a commit
that referenced
this pull request
Aug 31, 2026
…organization Conflicts were two comment blocks in config.default.yaml where the reorg's one-owning-file pointer notes met #806's new registry prose; resolved by combining both texts, keeping renewable.dataset owned by config.common.yaml. The registry block itself and the _125m wind-height change auto-merged into common.yaml alongside the reference land-access default. Also declares the new top-level godeeep_cf_registry key in the config schema (permissive - the registry module does its own strict validation). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #803.
What
Replaces the hardcoded Zenodo record table and the download hidden inside
build_renewable_profileswith a single declarative registry for GODEEEP compressed capacity-factor files:godeeep_cf_registry.py— sources (local mirror / Zenodo) are declared inconfig["godeeep_cf_registry"]with exactly the (dataset, year) pairs they hold. Sources are walked in config order; the first one declaring the request wins. There are no fallback paths: no default hub height, no nearest-year substitution, no unscreened-variant swap. All configured scenario × year × technology combinations are resolved eagerly at snakemake parse time; failures aggregate into one error naming the dataset key, the requested year, and the available years per source.rule retrieve_godeeep_cf— retrieval is now an explicit DAG step. Local-mirror hits are symlinked (or copied viacopy_local) after sha256 verification against the mirror'sSHA256SUMS; Zenodo hits download with checksum validation and retries.build_renewable_profilesconsumessnakemake.input.godeeep_cfand never picks a file itself.zenodo_downloader.py— every failure raisesZenodoDownloadError; nothing returnsNoneor leaves partial files. The unregisteredwind_125m_rcp45hotterrecord (20127545, HTTP 404) is declared with an empty year list so requesting it fails honestly instead of via the legacy silent-Nonepath.nrel_exclusion/compress_godeeep_*) used to build the 172-file sha256-verified mirror (solar + wind 80/100/125 m × 1980–2022 historical), plus 68 unit tests covering the registry, downloader, and compressor.Verification
_125m/ western / s75build_renewable_profilesrun retrieves via symlink + sha256 from the mirror and produces sane profiles (8760 hourly 2019 snapshots, CF ∈ [0,1], zero NaNs, full bus coverage; onwind's 73/75 buses reflect zero NREL wind capacity in LA + Inyo counties, pre-existing behavior).Notes for reviewers
resolve_cfvalidates from config declarations alone — it does not stat the local root. An off-Sherlock user keeping the default config passes parse-time validation and fails at retrieve time with a missing-file error; docs tell them to remove/repoint the local source. Guidance, not enforcement, in this PR.mem_mb=32000request for godeeepbuild_renewable_profilesmeasured ~17× actual peak (1.9 GB) at s75/western — left as-is since it may be sized for finer resolutions; happy to right-size.VALID_WIND_HEIGHTS— the_80msilent default was the original GODEEEP: historical weather years other than 2012 cannot receive NREL land-access screening (unscreened fallback profiles) #803 bug; enabling it explicitly can be a follow-up.🤖 Generated with Claude Code