Describe the feature you'd like to see
Harden input data retrieval steps by using a stable cache such as data.pypsa.org for all workflow data inputs not version-controlled via git.
Motivation
We do use large parts from the PyPSA-DE workflow in PyPSA-AT and run tests frequently on private pipelines and on public github actions. If remote files are not available from remote resources, the snakemake workflow fails early. This is the case even for data files that already exist in a local cache, because the retrieval rules check for the existence of remote resources regardless.
Another pro for using a cache is reproducibility: Input files may change e.g. the .xlsx files, and older results may turn irreproducible on input data updates.
List of resources I know of that sometimes fail
Are there reasons for the resources not being copied to a cache? Maybe licensing restricts storing copies?
Describe the feature you'd like to see
Harden input data retrieval steps by using a stable cache such as
data.pypsa.orgfor all workflow data inputs not version-controlled via git.Motivation
We do use large parts from the PyPSA-DE workflow in PyPSA-AT and run tests frequently on private pipelines and on public github actions. If remote files are not available from remote resources, the snakemake workflow fails early. This is the case even for data files that already exist in a local cache, because the retrieval rules check for the existence of remote resources regardless.
Another pro for using a cache is reproducibility: Input files may change e.g. the .xlsx files, and older results may turn irreproducible on input data updates.
List of resources I know of that sometimes fail
Are there reasons for the resources not being copied to a cache? Maybe licensing restricts storing copies?