Config reorganization: auto-loaded base config, tracked layers, schema validation

.github/workflows/main.yml

@@ -19,7 +19,7 @@ on:
   - cron: "0 5 * * TUE"    # weekly upstream-master regression
 
 env:
-  DATA_CACHE_NUMBER: 2
+  DATA_CACHE_NUMBER: 3
 
 jobs:
   fast-tests:
@@ -29,6 +29,8 @@ jobs:
     - uses: actions/setup-python@v5
       with:
         python-version: '3.11'
+    - name: Install GLPK (solver for the auto-marked unit tests)
+      run: sudo apt-get update -qq && sudo apt-get install -y -qq glpk-utils
     - name: Install test extras
       run: pip install -e '.[test]'
     - name: Run Tier A (static checks)
@@ -59,8 +61,8 @@ jobs:
       uses: actions/cache@v4
       with:
         path: |
-          data
-          cutouts
+          workflow/data
+          workflow/cutouts
         key: data-cutouts-${{ env.WEEK }}-${{ env.DATA_CACHE_NUMBER }}
     - name: Install test extras into the conda env
       run: pip install -e '.[test]'

.gitignore

@@ -11,8 +11,12 @@ dconf
 *.dot
 *.tar.gz
 
-# configuration files
-config/
+# per-user configuration files, seeded by init_pypsa_usa.sh. The canonical
+# templates live in workflow/repo_data/config/ and ARE tracked — the old
+# bare `config/` pattern matched that directory too and forced every template
+# edit through `git add -f`.
+/workflow/config/*
+!/workflow/config/.gitkeep
 
 # generated files
 results/
@@ -266,7 +270,6 @@ TSWLatexianTemp*
 
 # Exceptions to gitignore
 connect.sh
-/workflow/config/config.cluster.yaml
 /workflow/repo_data/dag.png
 !.pre-commit-config.yaml
 

CLAUDE.md

@@ -6,9 +6,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 PyPSA-USA is a Snakemake-orchestrated PyPSA workflow for capacity expansion, production-cost simulation, and power-flow analysis of the US bulk transmission system. Configuration is layered YAML; intermediate and final artifacts are netCDF/CSV/GeoJSON files produced by rules in `workflow/rules/*.smk` and Python scripts in `workflow/scripts/`.
 
+## Branch policy: work against `develop`, not `master`
+
+`develop` is the integration branch and is often well ahead of `master`. Any work on the model — code changes, config changes, reviews, audits, or analysis of "current state" — MUST reference `origin/develop`, not `master` or whatever the working tree happens to have checked out. Before drawing conclusions about how something works or proposing changes, run `git fetch origin develop` and check the file's state on `origin/develop` (`git show origin/develop:<path>` or check out a branch based on it). Base all feature branches and PRs on `develop`; never target `master` directly.
+
 ## Running the workflow
 
-**All `snakemake` invocations run from `workflow/`** — `cd workflow/` first. Snakemake auto-loads `config/config.cluster.yaml`, `config.common.yaml`, `config.plotting.yaml`, `config.api.yaml`, `config.sector.yaml` from `workflow/Snakefile`; the main configfile is passed via `--configfile`.
+**All `snakemake` invocations run from `workflow/`** — `cd workflow/` first. `workflow/Snakefile` auto-loads the whole layered base out of the tracked templates: `repo_data/config/config.{slurm,common,plotting,api,sector,default}.yaml`, then the optional per-user overlays `config/config.api.yaml` and `config/config.slurm.yaml`, then whatever is passed via `--configfile`. Because `config.default.yaml` is a loaded layer, a scenario config is a sparse **overlay** — it only needs the keys it changes (nested mappings merge; lists and scalars are replaced wholesale).
 
 ```bash
 cd workflow
@@ -20,9 +24,9 @@ Useful targets:
 - `rule data_model` — build everything up to the assembled-but-unsolved network (no solver).
 - `rule all` — full pipeline including solve and figures.
 - `--until <rule>` to stop early, `-R <rule>` to force re-execution.
-- Tutorial config (`config/config.tutorial.yaml`, CA only, simpl=75, clusters=4m, 2050) is the smallest meaningful end-to-end run.
+- Tutorial config (`repo_data/config/config.tutorial.yaml`, CA only, simpl=75, clusters=4m, 2050) is the smallest meaningful end-to-end run.
 
-HPC: edit `config/config.cluster.yaml` (account/partition/email) and `workflow/run_slurm.sh`, then `bash workflow/run_slurm.sh`.
+HPC: edit `config/config.slurm.yaml` (account/partition/email; it also holds the single per-rule `walltime:` block) and `workflow/run_slurm.sh`, then `bash workflow/run_slurm.sh`.
 
 ## Tests and lint
 
@@ -89,10 +93,12 @@ Defined in `workflow/Snakefile`:
 
 ## Configs
 
-- `config/config.default.yaml` — primary user-facing config (Western, default knobs).
-- `config/config.tutorial.yaml` — minimal CA-only smoke run.
-- `workflow/repo_data/config/` mirrors `config/` and is the source for `docs/source/configtables/` documentation.
-- Layered configs in `config/config.{cluster,common,api,plotting,sector}.yaml` are merged automatically by `Snakefile`; the main configfile only overrides what it needs to.
+- `workflow/repo_data/config/` is canonical and is what the Snakefile loads. It is also the source for `docs/source/configtables/` documentation.
+- `repo_data/config/config.default.yaml` — the scenario base layer: every user-facing knob, with defaults. Auto-loaded, and also the file users copy as a starting scenario.
+- `repo_data/config/config.tutorial.yaml`, `config.test.yaml` — sparse overlays (only keys that differ from the base). `config.equivalence*.yaml` are deliberately self-contained because the Tier C harness replays them against a pinned upstream anchor that does not load the base.
+- `workflow/config/` is untracked and holds only per-user files, seeded by `init_pypsa_usa.sh`: `config.default.yaml` (your scenario starting point), `config.api.yaml`, `config.slurm.yaml`. Do not add layered configs back into it.
+- `policy_constraints/` CSVs are read straight from `repo_data/config/policy_constraints/`.
+- The merged config is validated against `workflow/schemas/config.schema.yaml` at parse time (`snakemake.utils.validate`, `set_default=False`). The top level is open (snakemake/scenarios inject keys) but `electricity:`, `model_topology:`, `clustering:`, `solving:` etc. are closed, so a typo'd key fails loudly. Adding a config key means adding it to the schema.
 
 ## Things to know before changing rules
 

README.md

@@ -24,7 +24,7 @@ With [uv](https://docs.astral.sh/uv/) installed:
 ```bash
 git clone https://github.com/PyPSA/pypsa-usa.git
 cd pypsa-usa
-bash init_pypsa_usa.sh   # copy configuration templates into workflow/config
+bash init_pypsa_usa.sh   # seed the per-user config files into workflow/config
 cd workflow
 uv run snakemake -j1 --configfile config/config.default.yaml
 ```

docs/source/about-install.md

@@ -20,8 +20,14 @@ git clone git@github.com:PyPSA/pypsa-usa.git
 
 ## Step 2. Initialize Configuration files
 
-From the command line, run the script `init_pypsa_usa.sh` to copy configuration file
-templates into the `workflow/config` folder.
+From the command line, run the script `init_pypsa_usa.sh` to seed the per-user
+configuration files into the `workflow/config` folder.
+
+Only three files are copied — `config.default.yaml` (a starting point for your own
+scenario config), `config.api.yaml` (API keys) and `config.slurm.yaml` (HPC account
+settings). Every other configuration file is read by the workflow directly from the
+tracked `workflow/repo_data/config/` templates, so it can never fall out of sync with
+your checkout. The script is safe to re-run: existing files are left untouched.
 
 ```console
 bash init_pypsa_usa.sh
@@ -103,4 +109,6 @@ Snakemake's internal job scheduler, not the optimization solver).
 
 The PyPSA-USA workflow leverages the EIA API in several steps. The default configuration activates dynamic fuel-cost prices, which requires EIA API key. You can quickly get your key by completing this [form](https://www.eia.gov/opendata/register.php).
 
-The API key will be emailed to you, and you can copy the key into the `config.api.yaml` file.
+The API key will be emailed to you. Paste it into `workflow/config/config.api.yaml`, or
+export it as the `EIA_API_KEY` environment variable — the environment variable takes
+precedence over the YAML value, which keeps the key out of your files entirely.

docs/source/about-introduction.md

@@ -111,16 +111,16 @@ created under `workflow/` on first run.
 ```console
 ├── README.md
 ├── LICENSE.md
-├── init_pypsa_usa.sh            # one-time setup: copies default configs into place
+├── init_pypsa_usa.sh            # one-time setup: seeds the per-user config files
 ├── pyproject.toml               # python dependencies (uv); pins pypsa/atlite/linopy
 ├── docs                         # this documentation (sphinx + myst)
 ├── tests                        # static + integration test suites
 └── workflow
     ├── Snakefile                # entry point: wildcards, paths, top-level rules
     ├── rules                    # snakemake rule definitions (*.smk)
     ├── scripts                  # python scripts executed by the rules
-    ├── config                   # your run configuration (config.default.yaml, ...)
-    ├── repo_data                # small checked-in seed data (shapes, costs, dag)
+    ├── config                   # your per-user files (api keys, HPC, scenario configs)
+    ├── repo_data                # checked-in seed data + the canonical config templates
     ├── envs                     # conda environment specification
     ├── data                     # downloaded raw data bundles
     ├── cutouts                  # atlite weather cutouts (optional, large)

docs/source/about-usage.md

@@ -7,7 +7,16 @@
 
 ## Set Configuration
 
-To start, you'll want to set the proper network configuration for your studies purpose. The default configuration in `config/config.default.yaml` using the `western` interconnect and 33 nodes is a good place to start!
+To start, you'll want to set the proper network configuration for your studies purpose. The
+default configuration in `config/config.default.yaml` (seeded from the tracked template by
+`init_pypsa_usa.sh`) using the `western` interconnect and 33 nodes is a good place to start!
+
+Your config file only needs to carry the keys you actually change: the workflow always
+loads `repo_data/config/config.default.yaml` and the other layered files underneath it, and
+your `--configfile` is merged on top. So there are two equally valid ways to set up a run —
+copy the seeded `config/config.default.yaml` to `config/config.<scenario>.yaml` and edit it,
+or write a short file holding only your overrides. Nested mappings merge key by key, but
+lists and scalars are **replaced wholesale**, so any list you change must be restated in full.
 
 You can find more information on each configuration setting on the [configurations page](https://pypsa-usa.readthedocs.io/en/latest/config-configuration.html).
 
@@ -45,9 +54,9 @@ snakemake data_model -j1 --configfile config/config.default.yaml
 
 ## Running on HPC Cluster
 
-If you are running the workflow on an High-Performance Compute (HPC) cluster, you will first need to update the configuration settings in `config.cluster.yaml`. Update the account, partition, email, and chdir fields to match the information of your institutions cluster.
+If you are running the workflow on an High-Performance Compute (HPC) cluster, you will first need to update the configuration settings in `workflow/config/config.slurm.yaml` (seeded by `init_pypsa_usa.sh`). Update the account, partition, email, and chdir fields to match the information of your institutions cluster.
 
-Next, identify the name of the configuration file you would like to run by editing the `run_slurm.sh` script. The default value is the `--configfile config/config.default.yaml`.
+Next, identify the name of the configuration file you would like to run by editing the `--configfile` argument in the `run_slurm.sh` script; the path shipped in the script is only an example. The script also passes `--cluster-config config/config.slurm.yaml`, which is what resolves the `{cluster.*}` placeholders in its `sbatch` command line.
 
 To run, open a terminal within a login node of your cluster and run the script included in the `workflow` directory:
 
@@ -101,6 +110,7 @@ uv run snakemake -j4 -R build_shapes --until build_base_network --configfile con
 where `build_shapes` is forced to run, and `build_base_network` is the last rule you would like to run.
 
 ```{note}
-Every `snakemake` invocation must include `--configfile` (the Snakefile does not set a
-default configuration file). Omitting it fails with `KeyError: 'scenario'`.
+`--configfile` is optional: the Snakefile loads `repo_data/config/config.default.yaml` as
+the base layer, so omitting it simply runs the shipped defaults under `run: name: "Default"`.
+Pass `--configfile` to select your own scenario and give it its own `run: name:`.
 ```

docs/source/config-configuration.md

@@ -1,6 +1,47 @@
 (config)=
 # Configuration
 
+## How configuration is layered
+
+`workflow/Snakefile` loads its whole base out of the tracked
+`workflow/repo_data/config/` templates, in this order:
+
+| file | holds |
+| --- | --- |
+| `config.slurm.yaml` | HPC scheduler settings and the per-rule `walltime:` block |
+| `config.common.yaml` | renewable/atlite plumbing that rarely changes per scenario |
+| `config.plotting.yaml` | figure styling |
+| `config.api.yaml` | API-key placeholders |
+| `config.sector.yaml` | sector-coupling defaults |
+| `config.default.yaml` | the scenario base — every user-facing knob, with defaults |
+
+Two optional per-user overlays follow (`config/config.api.yaml`,
+`config/config.slurm.yaml`, seeded by `init_pypsa_usa.sh`), and whatever you
+pass with `--configfile` is merged last.
+
+Because `config.default.yaml` is itself a loaded layer, **your scenario config is
+an overlay, not a fork**: it only needs the keys it actually changes. Nested
+mappings merge key by key; lists and scalars are replaced wholesale, so a list
+you change must be restated in full.
+
+Every top-level key is owned by exactly one file — nothing is defined twice
+across layers.
+
+The one value that does not come from a file at all is the EIA API key: `workflow/Snakefile`
+reads `$EIA_API_KEY` from the environment and it takes precedence over `api: eia:` in
+`config/config.api.yaml`, which keeps the secret out of the checkout entirely.
+
+## Validation
+
+The merged configuration is checked against `workflow/schemas/config.schema.yaml`
+at parse time. The check covers the top-level structure, the enum-typed knobs
+(`foresight`, `electricity: retirement`, `renewable: dataset`, clustering
+algorithms, ATB scenario/model case, solver names, ...) and closes the
+well-bounded sections such as `electricity:` and `model_topology:` against
+unknown keys — so a misspelling like `retirment:` fails immediately instead of
+silently falling back to a default several rules later. The top level itself
+stays open, since Snakemake and the scenarios feature inject keys there.
+
 (run_cf)=
 ## `run`
 
@@ -680,13 +721,16 @@ networks.
 (walltime_cf)=
 ## `walltime`
 
-Per-rule wall-time overrides consumed as Snakemake `walltime` resources, used by the SLURM
-profile when submitting jobs to an HPC scheduler (see `config.cluster.yaml` and
-`workflow/run_slurm.sh`). Rules not listed here fall back to per-rule defaults defined in the
-workflow. Local runs ignore these values.
+Per-rule wall-time overrides consumed as Snakemake `walltime` resources and forwarded to
+`sbatch --time` by `workflow/run_slurm.sh`. This is the single source for per-rule wall times;
+it lives in `config.slurm.yaml` (renamed from `config.cluster.yaml`, whose name collided with
+the `{clusters}` wildcard and the `clustering:` section). Rules not listed here fall back to
+per-rule defaults defined in the workflow. Local runs ignore these values.
+
+Not to be confused with `solving: walltime:`, which caps the *solver*, not the scheduler job.
 
 ```{eval-rst}
-.. literalinclude:: ../../workflow/repo_data/config/config.default.yaml
+.. literalinclude:: ../../workflow/repo_data/config/config.slurm.yaml
    :language: yaml
    :start-after: # docs : WALLTIME
    :end-before: # docs :

docs/source/configtables/electricity.csv

@@ -10,7 +10,7 @@ extendable_carriers:,,,Capacity-expansion candidates. Carriers listed here get e
 -- Link,--,"e.g. {HVDC}",Adds extendable links at every connection where there are lines or HVDC links with zero initial capacity.
 ,,,
 SAFE_reservemargin,p.u.,float,"System-wide planning reserve margin as a fraction of peak load. Reserved setting — not currently enforced by an active constraint; the ``ERM`` option is the active resource-adequacy mechanism."
-SAFE_regional_reservemargins,--,path,"CSV of per-region planning reserve margins (``config/policy_constraints/SAFE_regional_prm.csv``). Reserved setting — not currently enforced by an active constraint."
+SAFE_regional_reservemargins,--,path,"CSV of per-region planning reserve margins (``repo_data/config/policy_constraints/SAFE_regional_prm.csv``). Reserved setting — not currently enforced by an active constraint."
 ,,,
 operational_reserve:,,,Settings for reserve requirements following `GenX <https://genxproject.github.io/GenX.jl/stable/Model_Reference/core/#Operational-Reserves>`_
 -- activate,bool,true or false,Whether to take operational reserve requirements into account during optimisation
@@ -21,10 +21,10 @@ operational_reserve:,,,Settings for reserve requirements following `GenX <https:
 erm:,,,Energy Reserve Margin settings (used when ERM opt is enabled). Ensures sufficient firm capacity to meet demand plus reserve margin at every timestep.
 -- <region>,--,float,"Reserve margin as a fraction (e.g., 0.15 for 15%). Use ``all`` for all regions, or specify region codes (state, interconnect, NERC region, or ReEDS zone). Defaults to ``all: 0.15`` if not specified."
 ,,,
-regional_Co2_limits,--,path,"CSV of per-region CO2 caps in tCO2/yr (``config/policy_constraints/regional_Co2_limits.csv``). Enforced when the ``REM`` keyword is present in the ``{opts}`` wildcard."
-technology_capacity_targets,--,path,"CSV of forced minimum/maximum capacity builds by technology and region (``config/policy_constraints/technology_capacity_targets.csv``). Enforced when the ``TCT`` keyword is present in the ``{opts}`` wildcard."
-portfolio_standards,--,path,"CSV of RPS/CES clean-energy fractions by region (``config/policy_constraints/portfolio_standards.csv``). Enforced when the ``RPS`` keyword is present in the ``{opts}`` wildcard (covers both RPS and CES targets)."
-transmission_interface_limits,--,path,"CSV of MW limits on flows across inter-regional transmission interfaces (``config/policy_constraints/transmission_interface_limits.csv``), paired with ``model_topology: interface_transmission_limits``. Reserved setting — not currently consumed by the workflow."
+regional_Co2_limits,--,path,"CSV of per-region CO2 caps in tCO2/yr (``repo_data/config/policy_constraints/regional_Co2_limits.csv``). Enforced when the ``REM`` keyword is present in the ``{opts}`` wildcard."
+technology_capacity_targets,--,path,"CSV of forced minimum/maximum capacity builds by technology and region (``repo_data/config/policy_constraints/technology_capacity_targets.csv``). Enforced when the ``TCT`` keyword is present in the ``{opts}`` wildcard."
+portfolio_standards,--,path,"CSV of RPS/CES clean-energy fractions by region (``repo_data/config/policy_constraints/portfolio_standards.csv``). Enforced when the ``RPS`` keyword is present in the ``{opts}`` wildcard (covers both RPS and CES targets)."
+transmission_interface_limits,--,path,"CSV of MW limits on flows across inter-regional transmission interfaces (``repo_data/config/policy_constraints/transmission_interface_limits.csv``), paired with ``model_topology: interface_transmission_limits``. Reserved setting — not currently consumed by the workflow."
 ,,,
 co2limit_enable,bool,true or false,"Switch to activate the system-wide CO2 cap below. Optional; defaults to false when unset. Can also be set via the ``Co2L`` keyword in the ``{opts}`` wildcard."
 co2limit,:math:`t_{CO_2}/a`,float,"System-wide cap on annual CO2 emissions, added as a global constraint in ``prepare_network``. Only applied when ``co2limit_enable`` is true."