Skip to content

Improve data retrieval and workflow robustness - #785

Open
danielelerede-oet wants to merge 13 commits into
PyPSA:developfrom
open-energy-transition:improve_workflow
Open

danielelerede-oet wants to merge 13 commits into
PyPSA:developfrom
open-energy-transition:improve_workflow

Conversation

@danielelerede-oet

@danielelerede-oet danielelerede-oet commented Aug 25, 2026

Copy link
Copy Markdown

Changes proposed in this Pull Request

This PR consolidates fixes identified while running the electricity and sector-coupled workflows.

  • Download the required PUDL Parquet files through explicit Snakemake rules and store them in a versioned local cache, avoiding direct remote access from DuckDB.
  • Pass the local PUDL inputs consistently to the power plant, demand, stock, and cost-building scripts.
  • Improve Zenodo downloads by supporting archived renewable-profile files, creating missing parent directories, validating downloads, and cleaning up incomplete or corrupted files.
  • Update obsolete NLR Data Catalog URLs.
  • Handle capacity plotting when a single- or multi-horizon network has no new extendable capacity.
  • Improve ERA5 cutout download reliability by using monthly, sequential requests.
  • Fix natural-gas data handling by selecting the requested year before temporal remapping and aggregating duplicate trade entries.
  • Accept both U.S. and U.S. Total in EIA gas-storage data.

The branch has been rebased onto develop; run-specific 2023 configuration files and downloaded PUDL data are not included.

Checklist

  • I tested my contribution locally and it seems to work fine.
  • Code and workflow changes are sufficiently documented.
  • Changed dependencies are added to envs/environment.yaml.
  • Changes in configuration options are added in all of config.default.yaml.
  • Changes in configuration options are also documented in doc/configtables/*.csv.

@danielelerede-oet
danielelerede-oet marked this pull request as draft August 25, 2026 18:32
@danielelerede-oet
danielelerede-oet marked this pull request as ready for review August 31, 2026 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant