diff --git a/docs/source/california-model.md b/docs/source/california-model.md new file mode 100644 index 00000000..7f83fc48 --- /dev/null +++ b/docs/source/california-model.md @@ -0,0 +1,299 @@ +(california-model)= +# California Model (CPUC SERVM) + +PyPSA-USA ships a maintained, runnable California-only configuration at +`workflow/repo_data/config/config.california.yaml` (copied to `workflow/config/` by +`init_pypsa_usa.sh`). This page is the reference for that configuration: what data goes into +it, which weather years are available for each ingredient, and which simplifications the +results carry. + +The model is a **California carve-out of the Western Interconnection**, not a standalone +network. The footprint is the four California ReEDS zones (`p8`, `p9`, `p10`, `p11`); the rest +of WECC is represented as capped import/export links rather than as buses. Load is the +California Public Utilities Commission's 2026 Integrated Resource Planning hourly forecast, +produced with the SERVM production-cost model. + +## Running it + +```bash +cd workflow + +# full pipeline (build + solve + figures) +uv run snakemake -j1 --configfile repo_data/config/config.california.yaml + +# data model only, no solver +uv run snakemake data_model -j1 --configfile repo_data/config/config.california.yaml +``` + +The first build downloads one ~118 MB SERVM load CSV per planning horizon from +`files.cpuc.ca.gov` (`retrieve_cpuc_servm_load`), the CPUC Baseline Generator List workbook +(`retrieve_cpuc_baseline_generators`), and the GODEEEP / NREL land-access artifacts from +Zenodo. `imports: costs: wholesale` calls the EIA API, so a key must be present in +`config/config.api.yaml`. + +Scenario settings as shipped: `interconnect: western`, `simpl: 75`, `clusters: 4`, +`ll: v1.0`, `opts: REM-3h`, `planning_horizons: [2030, 2035, 2040, 2045]`, `foresight: perfect`, +`sector: ""` (electricity only). + +## Data inventory + +| Ingredient | Source | Where it enters | Location | +| --- | --- | --- | --- | +| **Demand** | CPUC SERVM 2026 IRP hourly load — 6 California load regions, 9 forecast years, 25 stacked weather years, `Net Load` plus every published component | `demand: profile: servm` | `data/cpuc/servm/HourlyLoad_CA_Regions_V2025E_2224_Mon_{year}.csv`, retrieved from [files.cpuc.ca.gov](https://files.cpuc.ca.gov/energy/modeling/2026_servm_updates/) | +| **Demand → bus mapping** | 2020 Decennial Census county population, routed through the SERVM region map | `demand: bus_allocation: population`; `build_servm_load_weights` | `workflow/repo_data/CPUC/servm_region_map.csv` | +| **Existing fleet** | [PUDL](https://catalystcoop-pudl.readthedocs.io) `v2026.8.0` (EIA-860/923, FERC), announced retirements honored | `build_powerplants` → `resources/powerplants/powerplants.csv` | `pudl_path: s3://pudl.catalyst.coop/v2026.8.0` (`config.common.yaml`) | +| **Unit-commitment parameters** | WECC Anchor Data Set thermal tables, merged onto the EIA fleet and clamped to per-carrier bounds | `merge_ads_data` → `sanitize_uc_parameters` | `workflow/repo_data/plants/`, `UC_BOUNDS` in `build_powerplants.py` | +| **Renewable profiles** | [GODEEEP](https://www.pnnl.gov/projects/godeeep) capacity factors, weighted by NREL reV land-access availability | `renewable: dataset: godeeep`, `renewable_land_access: reference` | Zenodo (see [Weather years](ca-weather-years)) | +| **Transmission backbone** | ReEDS/NARIS zonal network | `model_topology: transmission_network: reeds` | `workflow/repo_data/ReEDS_Constraints/transmission/` | +| **Trade capacity** | NARIS AC flowgate ratings on the footprint boundary | `imports/exports: capacity_limit: true` | `transmission_capacity_init_AC_ba_NARIS2024.csv` (or `..._county_...` at county resolution) | +| **Trade interface caps** | CPUC RESOLVE aggregate CAISO interface limits | `model_topology: interface_transmission_limits: true` | `config/policy_constraints/transmission_interface_limits.csv` | +| **Trade prices** | EIA wholesale electricity prices for the weather year | `imports/exports: costs: wholesale` | EIA API (`config/config.api.yaml`) | +| **Out-of-state contracts** | CPUC ledger of physically out-of-state units serving California load (74 rows, 10,873 MW) | `electricity: remote_contracted_resources: enable: true` | `workflow/repo_data/CPUC/servm_out_of_state_units.csv` | +| **Emissions limit** | CARB 2022 Scoping Plan AB 32 trajectory, annual 2025-2050, import emissions factor 0.428 tCO2/MWh | `REM` token in `{opts}`; `electricity: regional_Co2_limits` | `config/policy_constraints/regional_Co2_limits.csv` (`CA_AB32` rows) | +| **Benchmark reference** | CPUC Baseline Generator List (`BaselineGeneratorList_CAISO.xlsx`), plus the out-of-state exclusion ledger | `run: benchmark_cpuc: true` | `data/cpuc/BaselineGeneratorList_CAISO.xlsx`, `workflow/repo_data/CPUC/servm_benchmark_regions.csv`, `servm_tech_map.csv` | +| **Costs** | NREL ATB (`Market` / `Moderate`) with IRA ITC/PTC modifiers; AEO reference fuel outlook | `costs:` block | see [Costs](data-costs.md) | + +### Demand regions + +SERVM reports six California load regions, mapped onto PyPSA-USA balancing areas by +`workflow/repo_data/CPUC/servm_region_map.csv`: `PGE`→`CISO-PGAE`, `SCE`→`CISO-SCE`, +`SDGE`→`CISO-SDGE`, `IID`→`IID`, `LADWP`→`LDWP`, and `NCNC`→`BANC` + `TIDC`. Four balancing +areas carry a *blank* region and are deliberately dropped with a log message rather than +hard-failing: `CISO-VEA` (a Nevada footprint) and the California slivers served by `PACW` +(Siskiyou/Del Norte/Modoc), `WALC` (Colorado River) and `NEVP` (Tahoe/CalNeva), none of which +the CPUC California-region files cover. An *unknown* balancing area still raises. + +Only the `Net Load` component is dispatched against. The full component split is preserved on +the `subsector` index level and written to +`resources//demand/{interconnect}/power_zonal_components_s{simpl}.parquet`. See the +[SERVM section](servm-demand) of the demand page for the full treatment. + +(ca-weather-years)= +## Weather years + +Load and renewable profiles are selected by **two independent keys**, and they must be kept +in step. Setting them to different years decorrelates load from wind and solar, which +understates both peak net load and the flexibility requirement. + +### Demand + +`electricity: demand: scenario: servm_weather_years` selects one of the **25 weather years +(2000-2024)** stacked inside each SERVM forecast-year file. It takes a list with **exactly one** +entry; multiple entries are reserved for stochastic scenarios and currently raise +`NotImplementedError`, because the demand output path is not weather-year specific. A mismatch +against the top-level `renewable_weather_years` is permitted but logs a warning in +`build_electrical_demand`. + +The *forecast* year is separate and comes from `scenario: planning_horizons`, which must be +drawn from the nine published SERVM years — 2026, 2028, 2030, 2032, 2035, 2037, 2040, 2042, +2045. SERVM demand is not interpolated or AEO-scaled between them. + +### Renewables + +Which renewable years are available depends on `renewable_scenarios`: + +- **Climate scenarios** (`rcp45hotter`, `rcp45cooler`, `rcp85hotter`, `rcp85cooler`) are indexed + by the **planning horizon**, not by a weather year. One Zenodo record per `(tech, scenario)` + publishes exactly three horizons: **2030, 2040 and 2050**. `renewable_weather_years` is not + consulted for the profile. +- **`historical`** is indexed by `renewable_weather_years[0]`. Every historical year flows + through the same screened NREL land-access path (`renewable_land_access`, plus optional + `_cec` / `_boem` overlays); availability depends on which registry source holds the + compressed per-cell file: + +| Source (first match wins) | Solar | Wind 100 m | Wind 125 m | +| --- | --- | --- | --- | +| Local (Oak) mirror, SHA256-verified | **1980-2022** | **1980-2022** | **1980-2022** | +| Zenodo records | 2012 | — | 2012 | + +A `(dataset, year)` combination no configured source declares fails at snakemake parse time +with the available years listed — no fallback, no default hub height, no nearest-year +substitution ([issue #803](https://github.com/PyPSA/pypsa-usa/issues/803) is resolved by the +mirror; the interim unscreened bus-aggregated fallback is retired). This makes every SERVM +demand weather year 2000–2022 pairable with a screened renewable profile of the same year; +SERVM years 2023–2024 currently have no matching GODEEEP profile. + +```{note} +`config.california.yaml` ships with `renewable_scenarios: ["rcp85cooler"]` and +`planning_horizons: [2030, 2035, 2040, 2045]`. Only 2030 and 2040 have a published GODEEEP +climate record; 2035 and 2045 have none. Use `planning_horizons: [2030, 2040]` under a climate +scenario, or switch to `renewable_scenarios: ["historical"]` (with the weather-year rules +above) for the other SERVM horizons. +``` + +## Spatial resolution + +The footprint is set once, by `model_topology: include: reeds_state: ['CA']` (equivalently +`reeds_zone: ['p8','p9','p10','p11']`). Two transmission resolutions are supported, and +`config.california.yaml` carries the second as a commented alternative block. + +| `topological_boundaries` | `clusters` | `simpl` | NARIS flowgate file | +| --- | --- | --- | --- | +| `reeds_zone` (shipped default) | `4` — the four California ReEDS zones | `75` | `transmission_capacity_init_AC_ba_NARIS2024.csv` | +| `county` | `58` — the 58 California counties | `'county'` (county-FIPS fast path) or any number ≥ 58 | `transmission_capacity_init_AC_county_NARIS2024.csv` | + +`clusters` cannot go below the number of zones in the footprint, which is why the zonal case is +pinned to 4. The county case is pinned to 58 because the county NARIS interface table carries +exactly 58 `p06xxx` nodes. `add_extra_components` swaps the flowgate file automatically from +`topological_boundaries`; no other key changes. See [Spatial Configuration](spatial) for the +`simpl`/`clusters` split. + +## Unit commitment + +`conventional: unit_commitment: true` is on by default in this configuration, and +`solving: options: linearized_unit_commitment: true` relaxes the binary commitment variables — +the model is an LP, not a MILP. With `opts: REM-3h` the problem also runs at 3-hour resolution. + +Commitment parameters (`min_up_time`, `min_down_time`, `ramp_limit_up`, `ramp_limit_down`, +`start_up_cost`, `minimum_load_mw`) come from the WECC Anchor Data Set thermal tables merged +onto the EIA fleet in `build_powerplants`. `sanitize_uc_parameters` then clamps every +committable row to per-carrier bounds (`UC_BOUNDS`, one band per carrier drawn from +NREL/EPRI/Intertek cycling literature and CAISO/WECC ADS typicals), filling missing values with +the carrier default and logging clamp/fill counts per carrier and parameter. The binding +invariant is `minimum_load_mw / p_nom <= min(summer_derate, winter_derate)`: a larger stable +minimum than the seasonal derate would leave the unit no feasible output above zero. +`add_electricity` re-clips the same fields as a second line of defence, and only committable +units are allowed to carry a non-zero `p_min_pu` (on a non-committable generator `p_min_pu` is +an unconditional must-run). + +Clustering aggregates commitment attributes deliberately: `start_up_cost` sums (the aggregate +starts as one unit, so a capacity-weighted average would understate cycling cost by roughly the +member count), ramp limits and stable minima take capacity-weighted averages, and `committable` +takes `any`. + +## Trade with the rest of WECC + +California is not modelled as an island. `electricity: imports` and `electricity: exports` are +both enabled, adding trade links at boundary buses; three bounds apply. + +1. **Per-path capacity** — `capacity_limit: true` rates each link from the NARIS AC flowgate + table for the active `topological_boundaries`. +2. **Aggregate interface caps** — `interface_transmission_limits: true` adds one per-snapshot + constraint per interface and direction from the RESOLVE table: + + | Interface | inside (`region_1`) | outside (`region_2`) | `flow_12` MW (export) | `flow_21` MW (import) | + | --- | --- | --- | --- | --- | + | `CA_NW` | p9, p10, p11 | p2, p5, p6, p7, p8 | 3,592 | 9,269 | + | `CA_SW` | p9, p10, p11 | p12, p13, p25, p27, p28, p30 | 10,901 | 10,463 | + | `CAISO_Imports` | p9, p10, p11 | all of the above | 9,728 | 10,208 | + +3. **Annual volume** — `volume_limit: 25` with `balancing_period: year` caps imported (and + separately exported) energy at 25 % of total demand, roughly CAISO's historical net-import + share. + +Imported energy is priced at EIA monthly wholesale prices and charged +`co2_emissions: 0.428` tCO2/MWh, the same import emissions factor the `CA_AB32` rows carry. +Exports earn the same wholesale price and are assigned zero emissions. + +Details of both constraint formulations are in +[Interface transmission limits](interface-transmission-limits) and +[Import and export volume limits](model-constraints.md#import-and-export-volume-limits). + +## Policy inputs + +`opts: [REM-3h]` activates **regional emissions limits** and 3-hour temporal resolution. The +`CA_AB32` rows of `config/policy_constraints/regional_Co2_limits.csv` give an annual CO2 budget +for California from the 2022 CARB Scoping Plan, from 46.6 MtCO2 in 2025 down to 8.68 MtCO2 in +2045, with imported energy charged at 0.428 tCO2/MWh. + +```{note} +`config.california.yaml` also populates `SAFE_reservemargin`, `SAFE_regional_reservemargins` +and `erm`. Neither constraint is active as shipped: the SAFE planning-reserve constraint is +switched on by a `SAFE` token in `{opts}` and the energy reserve margin by an `ERM` token, and +`opts` is `REM-3h`. Add the tokens (e.g. `REM-ERM-3h`) to bind them. +``` + +`technology_capacity_targets.csv` and `portfolio_standards.csv` are wired in but carry mostly +ReEDS-derived and example rows; see [Policies](data-policies.md). + +## CPUC baseline benchmark + +`run: benchmark_cpuc: true` adds `benchmark_cpuc_baseline` to the workflow targets, writing +`results//cpuc_benchmark/cpuc_capacity_benchmark.csv` and a deviation heatmap. It compares +installed capacity in `resources/powerplants/powerplants.csv` against the CPUC Baseline +Generator List, by region and technology, per horizon. + +The rule is deliberately **network-free**, so a fleet benchmark never drags in a network build. +`run: benchmark_cpuc_horizons: [2026]` is the shipped default: 2026 is the only pure +fleet-vs-fleet comparison, since the CPUC list is a baseline that stays roughly static at later +years while the model expands. An empty list falls back to `scenario: planning_horizons`. + +Two reconciliations happen before the sides are comparable. + +- **Region.** EIA reports every CAISO plant under the single code `CISO` with no sub-BA split, + so the benchmark runs at the coarsest resolution both sides support: `CAISO` (CPUC PGE + SCE + + SDGE vs. model `CISO`), `LADWP` (vs. `LDWP`), `IID`, and `NCNC` (vs. `BANC` + `TIDC`). The + model side is additionally restricted to `state == "CA"`, because EIA's `CISO` also covers + the Nevada `CISO-VEA` footprint. The collapse lives in `servm_benchmark_regions.csv`. +- **Technology.** Both sides map into a shared `compare_category` via `servm_tech_map.csv`, + which carries a `side` column so one file documents both directions. Anything unmapped + becomes its own `UNMAPPED:` row — a category is never silently dropped. + +Both sides are filtered by the same vintage rule: in service by December 31 of the horizon and +not retired by then. + +### The `EXCLUDED` row + +Rows of the CPUC list whose physical resource sits outside California are split off before +scoring and reported on a pseudo-region row, `EXCLUDED: out-of-state contracted`, broken down +by technology. Its `model_mw` and delta columns are blank: these are contractual ledger +entries that a physically located model cannot carry, so reporting them as model shortfall +would be misleading. + +Setting `electricity: remote_contracted_resources: enable: true` adds the EIA-identifiable +subset of exactly those units back into the model — 74 rows totalling 10,873 MW in +`servm_out_of_state_units.csv`, of which 8 rows / 1,910 MW have no `eia_plant_id` and are +skipped with a summary warning. **The benchmark keeps scoring them on the `EXCLUDED` row +regardless**, because it reads `powerplants.csv` rather than the network. When the option is +on, the MW on that row is what has been added back. See +[Out-of-State Contracted Resources](remote_contracted_resources) for the attachment rules and +their four deliberate simplifications. + +## Known caveats + +**Demand calendar and timezone** + +- SERVM strips are in **fixed Pacific Standard Time (UTC−8) with no DST transition**. This is + not stated in the source files; it was established empirically from the behind-the-meter PV + solar-noon centroid. The strips are rolled forward 8 hours to UTC. +- SERVM lays each year's 8760 hours on a **synthetic calendar that starts on a Monday**, so + weekday-versus-weekend hours do not line up with the real weekdays of the planning horizon. +- For a **leap weather year** the SERVM strip contains February 29 and omits December 31, while + PyPSA-USA's snapshots do the opposite. Every hour after February therefore lands one calendar + day earlier than in the source file. The strip is mapped positionally onto the network's own + snapshots, so each planning horizon must carry exactly 8760 snapshots — a truncated snapshot + window cannot be used with `profile: servm`. + +**Footprint** + +- The California slivers served by `PACW`, `WALC` and `NEVP`, and the Nevada `CISO-VEA` + footprint, carry no SERVM load and are dropped. +- `p8` appears in the `region_2` list of every RESOLVE interface row but is itself a California + zone. In a California-only model it is *inside* the network, so the internal `p8`-`p9` AC + corridor (~300 MW in the ReEDS/NARIS balancing-area table) carries no trade links and escapes + the `CAISO_Imports` cap. Simultaneous CAISO imports are understated by roughly that amount. + This is documented rather than corrected. + +**Fleet and benchmark comparability** + +- `Gas Cogen/CHP` is a CPUC-only category. EIA technology descriptions have no CHP concept, so + California gas cogeneration lands in `CCGT`/`OCGT` by prime mover. Expect the model to be + short in `Gas Cogen/CHP` and long in the two gas buckets by roughly the same amount. +- `Demand Response` and `Pumping Load` are CPUC-only resources with no PyPSA-USA counterpart; + they are kept as rows with `model_mw == 0`. +- The CPUC `Capmax MW` column is **nameplate** capacity as SERVM sees it, while the model side + totals `p_nom`. Seasonal derates are applied downstream in `add_electricity`, not in + `powerplants.csv`, so the benchmark compares nameplate to nameplate — but any summer-rating + comparison against a third source will not line up with either. +- `honor_planned_retirements: true` drops units at their EIA planned retirement date as of the + first investment period, so the fleet is smaller than an all-existing-units inventory. + +**Not implemented** + +- `conventional: ambient_derate` (CPUC SERVM unit-specific hourly thermal derates) is a + reserved phase-2 option. `enable: true` raises `NotImplementedError`. When it lands it + *replaces* the EIA-860 seasonal derate rather than stacking on it — stacking an ambient + derate on a seasonal derate, or on a UCAP-derated capacity credit, double-counts the same + thermal deficiency. +- `servm_weather_years` with more than one entry (stochastic weather scenarios) raises + `NotImplementedError`. diff --git a/docs/source/config-configuration.md b/docs/source/config-configuration.md index a644f2f8..24719947 100644 --- a/docs/source/config-configuration.md +++ b/docs/source/config-configuration.md @@ -91,8 +91,13 @@ network is aggregated to. `transmission_network` chooses between the ReEDS zonal TAMU synthetic nodal network; `topological_boundaries` sets the zone type used after clustering (county, REeDS zone, state, or balancing area). Use `include` to subset the modeled footprint to specific zones, states, or balancing authorities (mixed zone types are not supported), and -`aggregate` to pre-aggregate buses into larger regions. `interface_transmission_limits` applies -NARIS2024 inter-regional transfer capacity limits and requires the ReEDS backbone. +`aggregate` to pre-aggregate buses into larger regions. `interface_transmission_limits` switches +on the aggregate inter-regional transfer limits read from +`electricity: transmission_interface_limits` and requires the ReEDS backbone; the limits are +applied in `solve_network` as a per-snapshot cap on the total flow across each interface, and +constrain only the import/export links, so they are inert unless `electricity: imports` or +`electricity: exports` is enabled. See {ref}`spatial` for a worked California example +(`workflow/repo_data/config/config.california.yaml`) at both REeDS-zone and county resolution. ```{eval-rst} .. literalinclude:: ../../workflow/repo_data/config/config.default.yaml diff --git a/docs/source/config-spatial.md b/docs/source/config-spatial.md index 04fe7f71..f94deb9b 100644 --- a/docs/source/config-spatial.md +++ b/docs/source/config-spatial.md @@ -35,6 +35,49 @@ model_topology: Alternatively, you can use the code reeds_state: 'CA' option to achieve the same result by specifying the entire state. +A complete, maintained California configuration ships with the repository as +`workflow/repo_data/config/config.california.yaml`. It pairs the footprint below with CPUC +SERVM demand, the RESOLVE CAISO interface limits, and enabled imports/exports: + +```yaml +scenario: + interconnect: [western] + planning_horizons: [2030, 2035, 2040, 2045] # CPUC SERVM forecast years + clusters: [4] # p8, p9, p10, p11 + simpl: [75] + +model_topology: + transmission_network: 'reeds' + topological_boundaries: 'reeds_zone' + interface_transmission_limits: true # RESOLVE CAISO interface caps + include: + reeds_state: ['CA'] +``` + +`clusters` is pinned to 4 because the ReEDS zonal backbone cannot be clustered below the +number of zones in the footprint, and California holds exactly four. + +To run the same footprint at **county resolution**, switch `topological_boundaries` to +`county` and raise `clusters` to 58 — the number of California counties, and the number of +`p06xxx` nodes in the county NARIS interface table. `simpl: ['county']` selects the +county-FIPS fast path in `cluster_simpl`, so the resource layer is built directly on county +boundaries (a numeric `simpl` of at least 58 also works if you want more resource zones than +transmission nodes): + +```yaml +scenario: + clusters: [58] # 58 California counties + simpl: ['county'] # county-FIPS fast path + +model_topology: + topological_boundaries: 'county' +``` + +`add_extra_components` switches from the balancing-area NARIS flowgate file to the county one +(`transmission_capacity_init_AC_county_NARIS2024.csv`) automatically when +`topological_boundaries` is `county` — no other key needs changing. `config.california.yaml` +carries this same block as a commented alternative. + In addition to filtering by `reeds_zone` and `reeds_state`, you can filter by `reeds_ba`, `trans_reg`, and `nerc_reg` shown graphically below. diff --git a/docs/source/configtables/electricity.csv b/docs/source/configtables/electricity.csv index 792503da..7741f971 100644 --- a/docs/source/configtables/electricity.csv +++ b/docs/source/configtables/electricity.csv @@ -2,6 +2,7 @@ conventional_carriers,--,"Any subset of {nuclear, oil, OCGT, CCGT, coal, geothermal, biomass, waste}","List of conventional power plants to include in the model from ``resources/powerplants/powerplants.csv``. If an included carrier is also listed in ``extendable_carriers``, the capacity is taken as a lower bound." renewable_carriers,--,"Any subset of {onwind, offwind, offwind_floating, solar, hydro}",List of renewable generators to include in the model as existing capacity. retirement, --,One of ``economic`` or ``technical``,"Sets the retirement method for conventional generators. If ``technical`` all generators ``p_nom_min`` are set to ``p_nom`` to prevent selling off of the asset. Retirements are then tracked in post-processing. If ``economic`` existing plants have their ``p_nom_min`` set as ``0``, ``p_nom_max`` set to ``p_nom``, and capital costs set to fixed costs. Generators with ``p_nom`` are then added to handle capacity expansion.""" +honor_planned_retirements,bool,"{true, false}","Drop existing/proposed units at their EIA ``planned_generator_retirement_date`` (applies at the first investment period). ``false`` keeps every existing unit indefinitely (pre-v1 behavior)." ,,, extendable_carriers:,,,Capacity-expansion candidates. Carriers listed here get extendable ``p_nom`` and pay annualized capex; carriers listed only in the ``*_carriers`` lists above are fixed (brownfield-only). -- Generator,--,"Defaults: {solar, onwind, offwind_floating, OCGT, CCGT, CCGT-95CCS, coal, nuclear, hydrogen_ct}; also supported: {EGS, SMR, coal-95CCS}","Extendable generation carriers. Conventional generators can only be built/expanded where already existent today. If a listed conventional carrier is not included in the ``conventional_carriers`` list, the lower limit of the capacity expansion is set to 0." @@ -9,6 +10,10 @@ extendable_carriers:,,,Capacity-expansion candidates. Carriers listed here get e -- Store,--,"e.g. {H2_tank}",Adds extendable stores at every node/bus after clustering with zero initial capacity. Requires sector coupling. -- Link,--,"e.g. {HVDC}",Adds extendable links at every connection where there are lines or HVDC links with zero initial capacity. ,,, +remote_contracted_resources:,,,"California only. Re-adds the physically out-of-state resources the CPUC ledger attributes to California load regions (Palo Verde's SCE share, LADWP's Intermountain and Apex, the Hoover entitlements, ~2 GW of AZ/NV solar and battery contracts) as generators at California buses, at their CONTRACTED capacity, with techno-economics from the same PUDL fleet build. A California-scoped footprint otherwise drops them because they sit outside the model regions. Requires ``demand: profile: servm`` — the SERVM load weights decide which bus each region's contracts attach to. Modelling simplifications are documented under Generators & Storage Units." +-- enable,bool,"{true, false}","Attach the CA-contracted out-of-state units at CA buses. ``false`` (the default) is fully inert: no extra rule inputs are requested and the network is unchanged. NOTE: when this is on, the MW reported on the ``EXCLUDED`` row of the CPUC baseline benchmark is what has been added back into the model — the benchmark reads ``powerplants.csv`` rather than the network, so it keeps scoring these units as excluded." +-- file,--,path,"CPUC out-of-state contracted-unit ledger (default ``repo_data/CPUC/servm_out_of_state_units.csv``). Rows with an empty ``eia_plant_id`` (Mexicali TDM/LR2, Powerex/BC, ESJ Baja wind and a few pure entitlement rows) are skipped with a summary warning." +,,, SAFE_reservemargin,p.u.,float,"System-wide planning reserve margin as a fraction of peak load. Reserved setting — not currently enforced by an active constraint; the ``ERM`` option is the active resource-adequacy mechanism." SAFE_regional_reservemargins,--,path,"CSV of per-region planning reserve margins (``repo_data/config/policy_constraints/SAFE_regional_prm.csv``). Reserved setting — not currently enforced by an active constraint." ,,, @@ -24,7 +29,7 @@ erm:,,,Energy Reserve Margin settings (used when ERM opt is enabled). Ensures su regional_Co2_limits,--,path,"CSV of per-region CO2 caps in tCO2/yr (``repo_data/config/policy_constraints/regional_Co2_limits.csv``). Enforced when the ``REM`` keyword is present in the ``{opts}`` wildcard." technology_capacity_targets,--,path,"CSV of forced minimum/maximum capacity builds by technology and region (``repo_data/config/policy_constraints/technology_capacity_targets.csv``). Enforced when the ``TCT`` keyword is present in the ``{opts}`` wildcard." portfolio_standards,--,path,"CSV of RPS/CES clean-energy fractions by region (``repo_data/config/policy_constraints/portfolio_standards.csv``). Enforced when the ``RPS`` keyword is present in the ``{opts}`` wildcard (covers both RPS and CES targets)." -transmission_interface_limits,--,path,"CSV of MW limits on flows across inter-regional transmission interfaces (``repo_data/config/policy_constraints/transmission_interface_limits.csv``), paired with ``model_topology: interface_transmission_limits``. Reserved setting — not currently consumed by the workflow." +transmission_interface_limits,--,path,"CSV of MW limits on flows across inter-regional transmission interfaces (``repo_data/config/policy_constraints/transmission_interface_limits.csv``, columns ``interface, region_1, region_2, flow_12, flow_21``). Applied by ``solve_network`` when ``model_topology: interface_transmission_limits`` is ``true``, as a per-snapshot cap on the **aggregate** flow across each interface rather than a path-by-path limit: ``flow_12`` caps exports out of ``region_1`` into ``region_2``, ``flow_21`` caps imports in the opposite direction. Only the import/export ``Link`` components added by ``add_extra_components`` are constrained, so the caps are inert when ``imports``/``exports`` are disabled." ,,, co2limit_enable,bool,true or false,"Switch to activate the system-wide CO2 cap below. Optional; defaults to false when unset. Can also be set via the ``Co2L`` keyword in the ``{opts}`` wildcard." co2limit,:math:`t_{CO_2}/a`,float,"System-wide cap on annual CO2 emissions, added as a global constraint in ``prepare_network``. Only applied when ``co2limit_enable`` is true." @@ -33,12 +38,13 @@ gaslimit,MWh thermal,float,"Cap on annual gas-fired primary energy from gas carr ,,, demand:,,, -- bus_allocation,--,"One of {``population``, ``breakthrough``}","How zone-level demand is distributed to individual buses. ``population`` (default) weights buses by 2020 Decennial Census county populations (split evenly across each county's substations, then each substation's buses). ``breakthrough`` uses the legacy nominal-demand column (``Pd``) from the 2016-vintage Breakthrough Energy grid model." --- profile,--,"One of {``efs``, ``eia``, ``eer``}","Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand data. ``EER`` pulls future state-level profiles from the EER dataset; when selected, ``planning_horizons`` must be one of 2021, 2025, 2030, 2035, 2040, 2045, or 2050 and ``renewable_weather_years`` must contain exactly one year from 2007-2013 or 2016-2023." +-- profile,--,"One of {``efs``, ``eia``, ``eer``, ``servm``}","Datasource for electrical load data. ``EFS`` pulls future state level electrical demand data. ``EIA`` pulls historical balancing level electrical demand data. ``EER`` pulls future state-level profiles from the EER dataset; when selected, ``planning_horizons`` must be one of 2021, 2025, 2030, 2035, 2040, 2045, or 2050 and ``renewable_weather_years`` must contain exactly one year from 2007-2013 or 2016-2023. ``SERVM`` pulls CPUC SERVM hourly load for the six California load regions (California models only); when selected, ``planning_horizons`` must be one of 2026, 2028, 2030, 2032, 2035, 2037, 2040, 2042, or 2045, and each horizon must carry a full 8760-hour snapshot year. Only the ``Net Load`` component is dispatched against; the full component split is written to ``power_zonal_components_s{simpl}.parquet``. See the SERVM section of the demand data page." -- scenario:,,, -- -- efs_case,--,"One of {``reference``, ``medium``, ``high``}",(UNDER DEVELOPMENT) Extracts EFS data according to level of adoption -- -- efs_speed,--,"One of {``slow``, ``moderate``, ``fast``}",(UNDER DEVELOPMENT) Extracts EFS data according to speed of electrification -- -- eer_file,--,"One of {``demand_EER2025_100by2050.h5``, ``demand_EER2025_Baseline_AEO2023.h5``, ``demand_EER2025_IRAlow.h5``}",Selects the EER demand dataset file to download and use when ``profile`` is ``eer``. -- -- aeo,--,One of the AEO scenarios `here `_,(UNDER DEVELOPMENT) Scales future demand according to the AEO scenario +-- -- servm_weather_years,--,List with exactly one year from 2000-2024,"Weather year drawn from the stacked SERVM record when ``profile`` is ``servm``. Multiple entries are reserved for stochastic scenarios and currently raise ``NotImplementedError``. Keep equal to the top-level ``renewable_weather_years`` so load and renewable profiles share a weather year; a mismatch logs a warning." ,,, demand_response:,,,Settings to activate and configure demand response -- shift,per_unit,"float {0 <=, >= 1} or 'inf'",Allowable load to be shifted per snapshot. Set to 0 to turn off demand response. Set to 'inf' to not enforce capacity limits. diff --git a/docs/source/configtables/run.csv b/docs/source/configtables/run.csv index 0cce5f09..c1951015 100644 --- a/docs/source/configtables/run.csv +++ b/docs/source/configtables/run.csv @@ -1,6 +1,8 @@ -,Unit,Values,Description -name,--,any string,Specify a name for your run. Results will be stored under this name. -disable_progrssbar,bool,"{true, false}",Switch to select whether progressbar should be disabled. -shared_resources,bool,"{true, false}",Switch to select whether resources should be shared across runs. -shared_cutouts,bool,"{true, false}",Switch to select whether cutouts should be shared across runs. -validation,bool,"{true, false}",Switch to enable back-casting validation plotting +,Unit,Values,Description +name,--,any string,Specify a name for your run. Results will be stored under this name. +disable_progrssbar,bool,"{true, false}",Switch to select whether progressbar should be disabled. +shared_resources,bool,"{true, false}",Switch to select whether resources should be shared across runs. +shared_cutouts,bool,"{true, false}",Switch to select whether cutouts should be shared across runs. +validation,bool,"{true, false}",Switch to enable back-casting validation plotting +benchmark_cpuc,bool,"{true, false}",Switch to enable the CPUC Baseline Generator List capacity benchmark (California runs) +benchmark_cpuc_horizons,list,"years from the CPUC list horizon range","Benchmark year(s) for the CPUC capacity comparison; default [2026] compares today's fleet against today's baseline without any network build. Empty list falls back to scenario planning_horizons." diff --git a/docs/source/data-demand.md b/docs/source/data-demand.md index 89be90f8..5a1dea8a 100644 --- a/docs/source/data-demand.md +++ b/docs/source/data-demand.md @@ -21,6 +21,113 @@ the model years 2021, 2025, 2030, 2035, 2040, 2045, and 2050, and each profile i historical weather year, so `renewable_weather_years` must contain exactly one year from 2007-2013 or 2016-2023. +(servm-demand)= +### CPUC SERVM (California) + +`profile: servm` uses the hourly load forecast the California Public Utilities Commission +publishes for its 2026 Integrated Resource Planning cycle, produced with the SERVM +production-cost model. It is a **California-only** dataset — use it with a footprint scoped +to California (`model_topology: include: reeds_state: ['CA']`); the maintained entry point is +`workflow/repo_data/config/config.california.yaml`. The [California model](california-model.md) +page is the full reference for that configuration — every dataset that goes into it, the +weather-year options, and the caveats. + +The workflow retrieves one CSV per forecast year from + +``` +https://files.cpuc.ca.gov/energy/modeling/2026_servm_updates/HourlyLoad_CA_Regions_V2025E_2224_Mon_{year}.csv +``` + +(~118 MB each, via `retrieve_cpuc_servm_load`). **Nine forecast years are published: 2026, +2028, 2030, 2032, 2035, 2037, 2040, 2042, and 2045.** Unlike EFS, SERVM demand is *not* +interpolated or AEO-scaled between published years, so `scenario: planning_horizons` must be +drawn from that set — any other year raises in `ReadServm`. + +#### Regions + +SERVM reports six California load regions. Each maps onto the balancing areas PyPSA-USA +carries on its buses (`workflow/repo_data/CPUC/servm_region_map.csv`): + +| SERVM region | PyPSA-USA balancing area(s) | Notes | +| --- | --- | --- | +| `PGE` | `CISO-PGAE` | PG&E CAISO footprint | +| `SCE` | `CISO-SCE` | Includes Valley Electric Association's California load, per the CPUC data dictionary | +| `SDGE` | `CISO-SDGE` | San Diego Gas & Electric | +| `IID` | `IID` | Imperial Irrigation District | +| `LADWP` | `LDWP` | Los Angeles Department of Water and Power | +| `NCNC` | `BANC` + `TIDC` | Northern California non-CAISO: Balancing Authority of Northern California and Turlock Irrigation District | + +Four balancing areas carry a deliberately **empty** region in the mapping file, so their +(small) load shares are dropped with a log message, while any *unknown* balancing area +introduced by upstream relabeling still hard-fails: `CISO-VEA` (Valley Electric Association, a +Nevada footprint) and the California slivers served by `PACW` (Siskiyou/Del Norte/Modoc, whose +load lives in CPUC's non-CA PACW region file), `WALC` (WAPA Desert Southwest, Colorado River) +and `NEVP` (NV Energy, Tahoe/CalNeva) — none of which the CPUC California-region files cover. + +Because the SERVM regions are balancing areas rather than states, the demand is disaggregated +with a purpose-built weights table (`build_servm_load_weights`) instead of the generic +state/BA path: a cluster bus can straddle two SERVM regions (Los Angeles County holds both +`LDWP` and `CISO-SCE` buses), so a bus receives the sum of its share of every region it +overlaps. The underlying per-bus weights are still the `bus_allocation` weights described +below. + +The same weights table also places California's *out-of-state contracted supply*: with +`electricity: remote_contracted_resources: enable: true`, each resource the CPUC ledger +attributes to a SERVM region is attached at that region's max-LAF bus. See +[Out-of-State Contracted Resources](remote_contracted_resources) under Generators & Storage +Units. + +#### Components + +Each file publishes several load components per region (`Load`, `BTMPV`, `EV`, `DATA_CEN`, ...) +alongside `Net Load`. **Only `Net Load` is dispatched against by the model** — it is what +remains after behind-the-meter PV and other embedded resources. Every published component is +nonetheless carried through on the `subsector` index level and written to the zonal artifact +`resources//demand/{interconnect}/power_zonal_components_s{simpl}.parquet`, so the +component split stays available for reporting. Components that exist for only some regions +(`EV` and friends are published for PGE/SCE/SDGE only) align to `NaN` there. + +#### Weather years + +Each forecast-year file stacks 25 weather years (2000-2024) of a full hourly year. +`electricity: demand: scenario: servm_weather_years:` selects which one to use. It takes a +list with **exactly one** entry; multiple entries are reserved for stochastic scenarios +(phase 3) and currently raise `NotImplementedError`, because the demand output path is not +weather-year specific. + +**Set `servm_weather_years` equal to the top-level `renewable_weather_years`.** Drawing load +and wind/solar profiles from different weather years decorrelates them and will understate +both the peak-net-load and the flexibility need. A mismatch is permitted but logs a warning. + +Not every SERVM weather year has a matching renewable profile: which years the GODEEEP dataset +can supply, and at what cost in land screening, is tabulated under +[Historical weather-year availability](godeeep_weather_years) and summarised for California on +the [California model](california-model.md) page. + +#### Timezone and calendar caveats + +SERVM strips are in **fixed Pacific Standard Time (UTC−8) with no daylight-saving +transition**. This is not stated in the source files; it was verified empirically from the +behind-the-meter PV solar-noon centroid, which sits at hour 12.52 in December and 12.68 in +July — a DST-observing series would move by a full hour between the two. The strips are +rolled forward 8 hours to UTC before being attached. + +Two calendar misalignments are accepted, and are immaterial for an hourly +capacity-expansion model, but matter if you compare hour-for-hour against another source: + +1. **Monday-start synthetic calendar.** SERVM lays each year's 8760 hours on a synthetic + calendar that starts on a Monday, so weekday-versus-weekend hours do not line up with the + real weekdays of the planning horizon. +2. **Leap weather years.** For a leap *weather* year the SERVM strip contains February 29 and + omits December 31, while PyPSA-USA's snapshots do the opposite (`get_snapshots` drops + February 29 from leap planning horizons). Every hour after February therefore lands one + calendar day earlier than it sat in the source file. + +The strip is mapped **positionally** onto the network's own per-period snapshots rather than +onto a synthesised `date_range` — the latter would run a day short of December 31 for the +leap planning horizons (2028, 2032, 2040). As a consequence each planning horizon must carry +exactly 8760 snapshots; a truncated snapshot window cannot be used with `profile: servm`. + ## Demand Disaggregation All of the demand sources above arrive at a coarser resolution than the network: EIA930 @@ -68,8 +175,9 @@ horizons setting, and the electricity demand setting. If conducting historical s user must select a planning horizon in the past (2018-2023) and set `profile: eia`. If conducting forward-looking planning cases the user must set a future planning horizon — -2030, 2040, or 2050 with `profile: efs`, or any of 2021, 2025, 2030, 2035, 2040, 2045, and -2050 with `profile: eer`. +2030, 2040, or 2050 with `profile: efs`; any of 2021, 2025, 2030, 2035, 2040, 2045, and +2050 with `profile: eer`; or any of 2026, 2028, 2030, 2032, 2035, 2037, 2040, 2042, and 2045 +with `profile: servm` (California only). For planning horizons between the EFS data years, PyPSA-USA implements a scaling factor that interpolates between future years or scales historical demand using forecasts from the Annual @@ -81,11 +189,12 @@ scenario: electricity: demand: - profile: efs # efs, eia, eer + profile: efs # efs, eia, eer, servm scenario: efs_case: reference # reference, medium, high efs_speed: moderate # slow, moderate, rapid eer_file: demand_EER2025_100by2050.h5 # used when profile: eer + servm_weather_years: [2019] # used when profile: servm; exactly one year, 2000-2024 aeo: reference ``` diff --git a/docs/source/data-generators.md b/docs/source/data-generators.md index 77735383..00998982 100644 --- a/docs/source/data-generators.md +++ b/docs/source/data-generators.md @@ -25,6 +25,53 @@ In production cost-minimizing optimization models, a generator’s marginal cost - **Future Fuel Costs**: - Forecasted annual fuel prices are imported from the EIA's Annual Energy Outlook (AEO). +(remote_contracted_resources)= +## Out-of-State Contracted Resources (California) + +A California-scoped run (`model_topology: include: reeds_state: ['CA']`) attaches only the +plants that fall inside the model footprint, so every resource that is *physically* outside +California is dropped by `filter_plants_by_region` — including roughly 10.9 GW that the CPUC +ledger attributes to California load regions and that genuinely serves California load: SCE's +635 MW share of Palo Verde, LADWP's 1,185 MW of Intermountain and 566 MW of Apex, the Hoover +entitlements, and about 2 GW of Arizona and Nevada solar and battery contracts. + +Setting `electricity: remote_contracted_resources: enable: true` adds them back. For each row +of `workflow/repo_data/CPUC/servm_out_of_state_units.csv`, `add_electricity` looks the unit's +`eia_plant_id` up in the same PUDL fleet build that supplies the rest of the model (captured +*before* the regional filter runs, so the plant is still present), and attaches a single +generator — or a StorageUnit for battery contracts — at a California bus: + +- **Capacity** is the *contracted* MW (`capmax_mw`), capped at the live capacity the physical + plant actually has. Retirement and vintage filtering is respected: a contract whose plant has + no live units at the first investment period is skipped. +- **Bus** is the max-LAF bus of the row's SERVM region, taken from the same + `build_servm_load_weights` table the demand path uses. This is why the option requires + `demand: profile: servm`. +- **Techno-economics** (heat rate, efficiency, marginal cost, ramp rates, seasonal derates, + unit-commitment parameters) are capacity-weighted over the plant's constituent units, so a + contract inherits the same data the physical fleet carries. Generators are named + `R `, mirroring the `C` prefix on the conventional fleet, and are attached + **non-extendable**: they are existing contracts, not expansion candidates. + +Four simplifications are deliberate and worth knowing before reading results: + +1. **Remote wind and solar borrow the capacity-factor profile of their California attachment + bus**, not of their physical location. Desert-southwest solar is in reality better + correlated with CAISO's own solar than this implies. If the attachment bus has no profile + for that carrier, the network mean profile for the carrier is used; if the network has no + such profile at all the contract is dropped rather than given an implicit 100 % capacity + factor. +2. **Remote hydro (the Hoover entitlements) is attached as a firm, energy-unlimited + generator** carrying only the EIA-860 seasonal derate. CRSP hydrology, Lake Mead elevation + and the monthly energy schedules that actually bound those entitlements are not modelled. +3. **Rows with no `eia_plant_id` are skipped** — Mexicali TDM/LR2, Powerex/BC, the ESJ Baja + wind contracts and a few pure entitlement rows (about 1.9 GW). They have no PUDL + techno-economics to inherit and remain in import-machinery territory; the run logs a single + summary warning naming them and their MW. +4. **The CPUC baseline benchmark keeps scoring these units on its `EXCLUDED` row**, because it + reads `powerplants.csv` rather than the network. When this option is on, the MW on that row + is what has been added back into the model. + ## Renewable Resources (renewable_cfs)= @@ -42,19 +89,38 @@ The default capacity-factor source (`renewable.dataset: godeeep`) is the [GODEEE - **A historical period** (1980–2022) calibrated against observed weather. - **Four future climate scenarios** — `rcp45hotter`, `rcp45cooler`, `rcp85hotter`, `rcp85cooler` — under the RCP4.5 and RCP8.5 emissions pathways, downscaled with two GCM ensemble members per pathway. -- **Three planning horizons** (2030, 2040, 2050) per future scenario, drawn from contiguous 20-year (wind) or 40-year (solar) windows. +- **Three planning horizons** (2030, 2040, 2050) per future scenario, drawn from contiguous 20-year (wind) or 40-year (solar) windows. Under a climate scenario the profile year is the `planning_horizons` wildcard, not `renewable_weather_years`. The raw GODEEEP files are large (~4.4 GB per `(tech, scenario, year)` triple). PyPSA-USA consumes a uint8-quantized + zlib-compressed variant (~350 MB for solar, ~800 MB for wind). Each compressed file is placed by `rule retrieve_godeeep_cf` from the sources declared in the `godeeep_cf_registry` config block — a local mirror (on Sherlock, the full 1980–2022 historical archive at both hub heights) and Zenodo records keyed by `(tech, scenario)`. Only part of the matrix is on Zenodo; see [`renewable: godeeep`](godeeep_cf) for exactly which `(dataset, year)` pairs each source holds. GODEEEP capacity factors are re-aggregated to PyPSA-USA bus polygons using a runtime weighting step: -1. **Per-cell availability raster** — a fraction in [0, 1] for each 12 km GODEEEP cell, derived from NREL reV supply-curve availability scenarios. Three access scenarios are supported: `reference` (least restrictive, ~5× more permissive than Atlite+CORINE), `limited` (closest to the Atlite+CORINE baseline), and `open` (most permissive, for sensitivity studies). Optional overlays apply the California Energy Commission Wind/Solar BaseScreen (`_cec`, CA-only) and BOEM offshore wind planning areas (`_boem`, offshore). +1. **Per-cell availability raster** — a fraction in [0, 1] for each 12 km GODEEEP cell, derived from NREL reV supply-curve availability scenarios. Three access scenarios are supported, ordered from most to least restrictive: `limited` (the tightest siting regime, and the closest of the three to the Atlite+CORINE baseline), `reference` (NREL's intermediate regime, and the workflow's usual choice — it admits roughly two to three times the onshore land that `limited` does), and `open` (the most permissive, for sensitivity studies). Optional overlays apply the California Energy Commission Wind/Solar BaseScreen (`_cec`, CA-only) and BOEM offshore wind planning areas (`_boem`, offshore). 2. **Cell→bus mapping** computed once per bus layout from a county-level shapefile (cached on disk; ~14 min per interconnect at county resolution). 3. **Per-bus rollup** of `weight`, `p_nom_max`, `potential`, `average_distance`, and (for offshore) `underwater_fraction` from NREL supply-curve site locations within each bus polygon. The availability rasters and per-bus capacity rollups are published as a separate Zenodo record ([10.5281/zenodo.20127899](https://doi.org/10.5281/zenodo.20127899)) and downloaded on first run. -See [`renewable: godeeep`](godeeep_cf) under Model Configuration for the full set of config knobs. +(godeeep_weather_years)= +##### Historical weather-year availability + +With `renewable_scenarios: ["historical"]` the profile year is `renewable_weather_years[0]`. +Every historical year flows through the same screened NREL land-access path; what differs is +which registry **source** holds the compressed per-cell file: + +| Source (first match wins) | Solar | Wind 100 m | Wind 125 m | +| --- | --- | --- | --- | +| Local (Oak) mirror, SHA256-verified | **1980-2022** | **1980-2022** | **1980-2022** | +| Zenodo records | 2012 | — | 2012 | + +A `(dataset, year)` combination no configured source declares fails at snakemake parse time with +the available years listed — there is no fallback, no default hub height, and no nearest-year +substitution. (The unscreened bus-aggregated archives that briefly served other years are +retired; [issue #803](https://github.com/PyPSA/pypsa-usa/issues/803) is resolved by the mirror.) + +See [`renewable: godeeep`](godeeep_cf) under Model Configuration for the full set of config +knobs, and the [California model](california-model.md) page for how the two paths interact with a +CPUC SERVM demand year. #### Atlite (legacy alternative) diff --git a/docs/source/data-transmission.md b/docs/source/data-transmission.md index 4930c6df..63761b9f 100644 --- a/docs/source/data-transmission.md +++ b/docs/source/data-transmission.md @@ -38,6 +38,37 @@ While representative of the US electricity system, the TAMU network is synthetic See the [Spatial Configuration](./config-spatial.md) page for information on how to choose between networks. ``` +## Interface Transmission Limits + +The path-by-path ratings above are complemented by **interface** limits: aggregate MW caps on +the total simultaneous flow across a bundle of paths. PyPSA-USA ships the CPUC RESOLVE +interface table at `config/policy_constraints/transmission_interface_limits.csv`, which rates +the CAISO import/export capability against the rest of WECC: + +| Interface | `region_1` (inside) | `region_2` (outside) | `flow_12` (MW) | `flow_21` (MW) | +| --- | --- | --- | --- | --- | +| `CA_NW` | p9, p10, p11 | p2, p5, p6, p7, p8 | 3,592 | 9,269 | +| `CA_SW` | p9, p10, p11 | p12, p13, p25, p27, p28, p30 | 10,901 | 10,463 | +| `CAISO_Imports` | p9, p10, p11 | all of the above | 9,728 | 10,208 | + +**Flow orientation:** `flow_12` is the cap on flow *out of* `region_1` (exports), `flow_21` the +cap on flow *into* `region_1` (imports). Enable the table with +`model_topology: interface_transmission_limits: true`; the constraint formulation is described +in [Model Constraints](./model-constraints.md#interface-transmission-limits). + +```{warning} +The interface caps are applied to the virtual `imports` / `exports` links created by +`add_extra_components`, so they bind only when `electricity: imports` / `electricity: exports` +are enabled, and they only see flow that crosses the boundary of the modeled footprint. + +`p8` (northeastern California) appears in the `region_2` list of every RESOLVE row but is +itself a California zone. In a California-only model it is therefore *inside* the network, and +the internal `p8`-`p9` AC corridor — about 300 MW in the ReEDS/NARIS balancing-area table — +carries no trade links and escapes the `CAISO_Imports` cap. Simultaneous CAISO imports are +understated by roughly that amount. This gap is documented rather than corrected: closing it +would require constraining internal AC lines alongside the trade links. +``` + (transmission-data)= ### Data ```{eval-rst} diff --git a/docs/source/datatables/demand.csv b/docs/source/datatables/demand.csv index 18b33f81..ef55c509 100644 --- a/docs/source/datatables/demand.csv +++ b/docs/source/datatables/demand.csv @@ -1,3 +1,4 @@ Characteristic,Data Source,Spatial Scale,Temporal Scale Historical Demand,GridEmissions (EIA930),Balancing Area,Hourly (2018- 2023) Future Demand,NREL Electrification Futures Study (EFS),States,"Hourly (2030, 2040, 2050)" +Future Demand (California),CPUC SERVM 2026 IRP Hourly Load,Six California load regions (PGE/SCE/SDGE/IID/LADWP/NCNC),"Hourly (2026, 2028, 2030, 2032, 2035, 2037, 2040, 2042, 2045; weather years 2000-2024)" diff --git a/docs/source/index.md b/docs/source/index.md index f41051cf..e20ba25d 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -41,6 +41,7 @@ model-workflow model-components model-constraints model-network-schema +california-model ``` ```{toctree} diff --git a/docs/source/model-constraints.md b/docs/source/model-constraints.md index ad510003..42443b67 100644 --- a/docs/source/model-constraints.md +++ b/docs/source/model-constraints.md @@ -52,6 +52,7 @@ Regions used by the policy constraints may be specified as state codes, ReEDS zo | [Bidirectional link coupling](#bidirectional-link-coupling) | Equal capacity expansion of paired forward/reverse links | always active | [bidirectional_link.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/opts/bidirectional_link.py) | | [Demand-response capacity](#demand-response-capacity) | Shifted load bounded by a fixed share of nominal load per bus and snapshot | `electricity: demand_response: shift` | [sector.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/opts/sector.py) | | [Import/export volume limits](#import-and-export-volume-limits) | Traded energy bounded by a share of demand per balancing period | `electricity: imports/exports: volume_limit` | [interchange.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/opts/interchange.py) | +| [Interface transmission limits](#interface-transmission-limits) | Aggregate MW cap on the total flow across a bundle of transmission paths | `model_topology: interface_transmission_limits`; `electricity: transmission_interface_limits` | [interfaces.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/opts/interfaces.py) | | [National emission cap](#national-emission-cap-co2l) | System-wide CO2 cap via PyPSA `GlobalConstraint` | `Co2L` opts token; `electricity: co2limit` | [prepare_network.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/prepare_network.py) | | [Natural gas limit](#natural-gas-limit-ch4l) | Cap on annual gas-fired primary energy | `CH4L` opts token; `electricity: gaslimit` | [prepare_network.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/prepare_network.py) | | [Emission pricing](#emission-pricing-ep) | CO2 price added to marginal costs (objective, not a constraint) | `Ep` opts token; `costs: emission_prices` | [prepare_network.py](https://github.com/PyPSA/pypsa-usa/blob/master/workflow/scripts/prepare_network.py) | @@ -307,6 +308,41 @@ percent, and {math}`d_t` total AC load in period {math}`\tau`. In sector studies measured as the flow into the end-use sectors and the bound becomes a linear constraint in both trade and demand variables. +(interface-transmission-limits)= +## Interface transmission limits + +A transmission *interface* is a bundle of paths between two groups of regions that is rated in +aggregate rather than path-by-path — CAISO's simultaneous import capability being the canonical +example. Setting `model_topology: interface_transmission_limits: true` reads the interface table +at `electricity: transmission_interface_limits` (columns +`interface, region_1, region_2, flow_12, flow_21`) and adds one per-snapshot constraint per +interface and direction: + +**Trigger:** `model_topology: interface_transmission_limits: true`. + +\begin{align*} + &\ \hspace{1cm} \sum_{\ell \in I^{\rightarrow}} p_{\ell,t} \;\leq\; F_{12} + \hspace{0.5cm} \forall_t + \hspace{1cm} + \sum_{\ell \in I^{\leftarrow}} p_{\ell,t} \;\leq\; F_{21} + \hspace{0.5cm} \forall_t +\end{align*} + +where {math}`F_{12}` (`flow_12`) caps flow **out of** `region_1` into `region_2` and +{math}`F_{21}` (`flow_21`) caps flow in the opposite direction. {math}`I^{\rightarrow}` and +{math}`I^{\leftarrow}` are selected by bus membership and carrier, never by link name. + +```{important} +Only the `imports` / `exports` links added by `add_extra_components` are constrained, so the +limits are a **no-op when `electricity: imports` and `electricity: exports` are both disabled**. +A `region_2` entry that is itself inside the modeled footprint contributes no trade links, so +internal AC lines between it and `region_1` escape the cap. In the shipped +`CAISO_Imports` row this applies to `p8`, which is a California zone: in a California-only run +the internal `p8`-`p9` corridor (~300 MW in the ReEDS/NARIS balancing-area table) is not +counted against the CAISO import cap, understating simultaneous imports by roughly that +corridor's rating. This is documented, not corrected. +``` + (national-emission-cap-co2l)= ## National emission cap (Co2L) diff --git a/docs/source/release-notes.md b/docs/source/release-notes.md index 42d50689..cbc8ddd0 100644 --- a/docs/source/release-notes.md +++ b/docs/source/release-notes.md @@ -72,6 +72,36 @@ The full engineering change-log, including per-change expected effects on model results, is maintained in the repository at [`docs/CHANGELOG-v1-epic.md`](https://github.com/PyPSA/pypsa-usa/blob/master/docs/CHANGELOG-v1-epic.md). +### California / CPUC SERVM + +- **New demand source `electricity: demand: profile: servm`** — CPUC SERVM 2026 IRP hourly + load for the six California load regions (PGE, SCE, SDGE, IID, LADWP, NCNC), retrieved + per forecast year from files.cpuc.ca.gov. Nine forecast years are published (2026, 2028, + 2030, 2032, 2035, 2037, 2040, 2042, 2045) and `planning_horizons` is restricted to them. + `electricity: demand: scenario: servm_weather_years` picks one weather year out of the + stacked 2000-2024 record. Only `Net Load` is dispatched; the full component split is + written to a new component-resolved zonal artifact + (`power_zonal_components_s{simpl}.parquet`), which is now produced for every demand + profile. See {ref}`servm-demand`. +- **SERVM load-allocation weights** — a new `build_servm_load_weights` rule composes the + base→substation→cluster busmaps into a fractional `(SERVM region, bus)` table, so a cluster + that straddles two regions (Los Angeles County holds both LDWP and CISO-SCE buses) receives + the sum of its share of each. +- **Interface transmission limits are live.** `model_topology: interface_transmission_limits` + and `electricity: transmission_interface_limits` were previously dead keys. They now apply + the RESOLVE interface table as a per-snapshot cap on the *aggregate* flow across each + interface. The constraint scopes to the import/export links, so it is inert when trade is + disabled; the resulting understatement for `region_2` entries inside the footprint (notably + `p8` in California-only runs) is documented in {doc}`data-transmission`. +- **New maintained config `config.california.yaml`** — a runnable California-only model on + SERVM demand with the CAISO interface caps and imports/exports enabled, at REeDS-zone + resolution (`clusters: 4`) with a commented county-resolution alternative + (`clusters: 58`, `simpl: county`). +- **Phase-2 hook `conventional: ambient_derate`** — reserved for CPUC SERVM unit-specific + ambient-temperature derates. It is not implemented; enabling it raises `NotImplementedError` + in `add_electricity`. When it lands it replaces the EIA-860 seasonal derate rather than + stacking on it. + ### Documentation - New Model Description section ({doc}`model-workflow`, {doc}`model-components`, diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 74b6d713..b3d4f864 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -92,6 +92,118 @@ def busmap_s(self) -> Path: return self.base / "busmaps" / self.interconnect / f"busmap_s{self.simpl}.csv" +@dataclass(frozen=True) +class ServmArtifacts: + """Paths to the CPUC SERVM demand artifacts produced by the Tier B build. + + The ``interconnect`` and ``simpl`` defaults MUST match + ``workflow/repo_data/config/config.test.california.yaml``'s ``scenario`` + section. If you change one, change both. + """ + + run_name: str + base: Path # resources/{run_name}/ + interconnect: str = "western" + simpl: str = "20" + + @property + def elec_s(self) -> Path: + """Simplified network the weights and demand were built against.""" + return self.base / "networks" / self.interconnect / f"elec_s{self.simpl}.nc" + + @property + def elec_s_dem(self) -> Path: + """Simplified network with SERVM demand attached.""" + return self.base / "networks" / self.interconnect / f"elec_s{self.simpl}_dem.nc" + + @property + def weights(self) -> Path: + """(bus, servm_region, laf) allocation table from build_servm_load_weights.""" + return self.base / "demand" / self.interconnect / f"servm_load_weights_s{self.simpl}.csv" + + @property + def demand(self) -> Path: + """Per-bus hourly demand CSV written by build_electrical_demand.""" + return self.base / "demand" / self.interconnect / f"power_electricity_s{self.simpl}.csv" + + @property + def zonal_components(self) -> Path: + """Component-resolved zonal demand parquet (pre-disaggregation).""" + return self.base / "demand" / self.interconnect / f"power_zonal_components_s{self.simpl}.parquet" + + +def _run_snakemake(configfile: str, until: str, run_name: str) -> None: + """Run one ``snakemake --until `` build against ``workflow/``.""" + cmd = [ + "snakemake", + "--until", + until, + "--configfile", + configfile, + "--config", + f"run={{name: '{run_name}', shared_cutouts: true}}", + "-j", + str(os.cpu_count() or 2), + # Force greedy scheduler to avoid the ILP scheduler's cbc dependency + # (cbc is shipped non-executable in some envs and causes PermissionError). + "--scheduler", + "greedy", + # In a fresh checkout/worktree that shares data/ via symlink, snakemake's + # empty provenance DB marks every input-less retrieve rule as "code has + # changed" and re-downloads (rewriting the shared tree). mtime-only + # triggers treat existing retrieve outputs as up to date. + "--rerun-triggers", + "mtime", + "--quiet", + ] + try: + result = subprocess.run( + cmd, + cwd=WORKFLOW_DIR, + capture_output=True, + text=True, + timeout=600, + ) + except subprocess.TimeoutExpired as e: + raw_stderr = e.stderr + if isinstance(raw_stderr, bytes): + raw_stderr = raw_stderr.decode("utf-8", errors="replace") + stderr_tail = "\n".join(raw_stderr.splitlines()[-100:]) if raw_stderr else "" + pytest.fail( + f"snakemake build timed out after {e.timeout}s\nstderr (last 100 lines):\n{stderr_tail}", + ) + if result.returncode != 0: + pytest.fail( + f"snakemake build failed (exit {result.returncode}):\n" + f"stderr (last 100 lines):\n" + "\n".join(result.stderr.splitlines()[-100:]), + ) + + +@pytest.fixture(scope="session") +def servm_built(tmp_path_factory) -> ServmArtifacts: + """Run ``snakemake --until add_demand`` on the SERVM test config once per session. + + Stops at ``add_demand`` rather than ``cluster_network``: the SERVM + artifacts under test (weights, per-bus demand CSV, zonal components) are all + produced at or before that stage, and the extra clustering work is already + covered by the ``built`` fixture. + + Skips like ``built`` when the data dirs are missing. This build also + downloads one ~118 MB CPUC SERVM load file the first time it runs. + """ + missing = [d for d in DATA_DIRS if not d.exists()] + if missing: + pytest.skip( + "Integration tests require populated data dirs; missing: " + ", ".join(str(m) for m in missing), + ) + run_name = f"pytest_servm_{tmp_path_factory.mktemp('servm_run').name}" + _run_snakemake("repo_data/config/config.test.california.yaml", "add_demand", run_name) + return ServmArtifacts( + run_name=run_name, + base=WORKFLOW_DIR / "resources" / run_name, + ) + + @pytest.fixture(scope="session") def built(tmp_path_factory) -> BuiltArtifacts: """Run ``snakemake --until cluster_network`` once per session and expose artifact paths. @@ -119,6 +231,12 @@ def built(tmp_path_factory) -> BuiltArtifacts: # (cbc is shipped non-executable in some envs and causes PermissionError). "--scheduler", "greedy", + # In a fresh checkout/worktree that shares data/ via symlink, snakemake's + # empty provenance DB marks every input-less retrieve rule as "code has + # changed" and re-downloads (rewriting the shared tree). mtime-only + # triggers treat existing retrieve outputs as up to date. + "--rerun-triggers", + "mtime", "--quiet", ] try: diff --git a/tests/integration/test_artifacts.py b/tests/integration/test_artifacts.py index e5d752cc..bf5ca57a 100644 --- a/tests/integration/test_artifacts.py +++ b/tests/integration/test_artifacts.py @@ -28,8 +28,11 @@ def test_file_exists(self, built): def test_bus_count_reasonable(self, built): """Substation count falls in the band expected for the test config.""" n = pypsa.Network(str(built.elec_b)) - # CA-only Western yields O(50) substations after aggregation - assert 10 < len(n.buses) < 200, ( + # On the ReEDS transport path aggregate_to_substations keeps TAMU bus + # granularity (CA-only Western ~2,000 buses); the real reduction + # happens at cluster_resources. Bound catches config drift, not a + # target substation count. + assert 10 < len(n.buses) < 4000, ( f"unexpected substation count {len(n.buses)} — config.test.yaml may have drifted" ) diff --git a/tests/integration/test_servm_demand_artifacts.py b/tests/integration/test_servm_demand_artifacts.py new file mode 100644 index 00000000..ba5612de --- /dev/null +++ b/tests/integration/test_servm_demand_artifacts.py @@ -0,0 +1,151 @@ +"""Tier B — assert shape and orientation of the CPUC SERVM demand artifacts. + +Built from ``workflow/repo_data/config/config.test.california.yaml`` by the +``servm_built`` fixture (``snakemake --until add_demand``). These are structural +checks — conservation of the allocation weights, alignment of the per-bus demand +CSV with the network it was built against, and the PST->UTC roll. No numerical +regression against the CPUC reference case here; that is the (separate) +benchmark rule ``run.benchmark_cpuc`` reserves. +""" + +from __future__ import annotations + +import pandas as pd +import pypsa +import pytest + +pytestmark = pytest.mark.integration + +# The six CPUC SERVM California load regions (ReadServm.REGIONS). +SERVM_REGIONS = {"IID", "LADWP", "NCNC", "PGE", "SCE", "SDGE"} + + +class TestServmLoadWeights: + """Assertions on servm_load_weights_s{simpl}.csv (build_servm_load_weights).""" + + def test_file_exists(self, servm_built): + """The weights table was produced by snakemake.""" + assert servm_built.weights.exists(), f"missing {servm_built.weights}" + + def test_columns(self, servm_built): + """The table carries the three columns WriteServm reads.""" + weights = pd.read_csv(servm_built.weights) + assert {"bus", "servm_region", "laf"}.issubset(weights.columns) + + def test_laf_sums_to_one_per_region(self, servm_built): + """Allocation factors conserve each region's demand exactly.""" + weights = pd.read_csv(servm_built.weights) + totals = weights.groupby("servm_region")["laf"].sum() + assert not totals.empty, "no SERVM regions in the weights table" + pd.testing.assert_series_equal( + totals, + pd.Series(1.0, index=totals.index, name="laf"), + check_exact=False, + rtol=1e-6, + ) + + def test_regions_are_known(self, servm_built): + """Only the six published SERVM regions appear (CISO-VEA is excluded).""" + weights = pd.read_csv(servm_built.weights) + unknown = set(weights.servm_region.unique()) - SERVM_REGIONS + assert not unknown, f"unexpected SERVM region(s) {sorted(unknown)}" + + def test_buses_exist_in_network(self, servm_built): + """Every weighted bus is a bus of the network the weights were built for.""" + weights = pd.read_csv(servm_built.weights) + n = pypsa.Network(str(servm_built.elec_s)) + unknown = set(weights.bus.astype(str)) - set(n.buses.index.astype(str)) + assert not unknown, f"weights reference {len(unknown)} bus(es) not in elec_s{servm_built.simpl}.nc" + + +class TestServmDemand: + """Assertions on power_electricity_s{simpl}.csv (build_electrical_demand).""" + + def test_file_exists(self, servm_built): + """The per-bus demand CSV was produced by snakemake.""" + assert servm_built.demand.exists(), f"missing {servm_built.demand}" + + def test_columns_are_network_buses(self, servm_built): + """Demand columns are buses of the network, and there is at least one.""" + demand = pd.read_csv(servm_built.demand, index_col=0) + n = pypsa.Network(str(servm_built.elec_s)) + assert len(demand.columns) > 0 + unknown = set(demand.columns.astype(str)) - set(n.buses.index.astype(str)) + assert not unknown, f"demand columns are not network buses: {sorted(unknown)[:10]}" + + def test_row_count_matches_snapshots(self, servm_built): + """One row per network snapshot. + + Compares against the demand-attached network: ``elec_s{simpl}.nc`` + still carries PyPSA's default ``['now']`` index — snapshots are only + set when ``add_demand`` attaches the load. + """ + demand = pd.read_csv(servm_built.demand, index_col=0) + n = pypsa.Network(str(servm_built.elec_s_dem)) + assert len(demand) == len(n.snapshots), ( + f"demand has {len(demand)} rows, network has {len(n.snapshots)} snapshots" + ) + + def test_demand_is_positive(self, servm_built): + """No NaNs, no negative load, and a non-trivial total.""" + demand = pd.read_csv(servm_built.demand, index_col=0) + assert not demand.isna().any().any() + assert (demand.to_numpy() >= 0).all(), "SERVM Net Load produced negative demand" + assert demand.to_numpy().sum() > 0 + + def test_attached_to_network(self, servm_built): + """add_demand attached the SERVM load to elec_s{simpl}_dem.nc.""" + n = pypsa.Network(str(servm_built.elec_s_dem)) + assert len(n.loads) > 0 + assert not n.loads_t.p_set.isna().any().any() + assert n.loads_t.p_set.to_numpy().sum() > 0 + + def test_peak_hour_consistent_with_pst(self, servm_built): + """The annual peak lands in the CA afternoon/evening once rolled to UTC. + + ``ReadServm`` rolls the fixed-PST strip forward by 8 hours, so the + snapshot index is effectively UTC. A California system peak sits in the + late afternoon local time; 15:00-20:00 PST maps to UTC hours 23-04. The + band is deliberately wide — it is testing the direction and magnitude of + the roll, not the exact peak hour. (Peak *date* is not asserted: SERVM + lays its hours on a synthetic Monday-start calendar, and a leap weather + year shifts every post-February hour by one calendar day.) + """ + demand = pd.read_csv(servm_built.demand, index_col=0, parse_dates=True) + peak_hour = demand.sum(axis=1).idxmax().hour + assert peak_hour in { + 23, + 0, + 1, + 2, + 3, + 4, + }, f"annual peak at UTC hour {peak_hour}; expected 23-04 for a PST-rolled California profile" + + +class TestServmZonalComponents: + """Assertions on power_zonal_components_s{simpl}.parquet.""" + + def test_file_exists(self, servm_built): + """The component-resolved zonal artifact was produced.""" + assert servm_built.zonal_components.exists(), f"missing {servm_built.zonal_components}" + + def test_regions_are_columns(self, servm_built): + """Columns are the SERVM regions; every region carries nonzero energy.""" + zonal = pd.read_parquet(servm_built.zonal_components) + assert set(zonal.columns) <= SERVM_REGIONS, f"unexpected column(s) {sorted(set(zonal.columns) - SERVM_REGIONS)}" + net_load = zonal.xs("Net Load", level="subsector") + assert (net_load.sum() > 0).all(), ( + f"zero annual energy in region(s) {sorted(net_load.columns[net_load.sum() <= 0])}" + ) + + def test_net_load_component_present(self, servm_built): + """``Net Load`` — the only component the model dispatches against — is kept.""" + zonal = pd.read_parquet(servm_built.zonal_components) + assert "Net Load" in zonal.index.get_level_values("subsector") + + def test_components_beyond_net_load_are_kept(self, servm_built): + """The zonal artifact stays component-resolved (BTMPV, EV, ... survive).""" + zonal = pd.read_parquet(servm_built.zonal_components) + components = set(zonal.index.get_level_values("subsector")) + assert components - {"Net Load"}, "zonal artifact collapsed to Net Load only" diff --git a/tests/static/test_dag_dryrun.py b/tests/static/test_dag_dryrun.py index 84176912..d92df700 100644 --- a/tests/static/test_dag_dryrun.py +++ b/tests/static/test_dag_dryrun.py @@ -19,34 +19,56 @@ WORKFLOW_DIR = Path(__file__).resolve().parents[2] / "workflow" +# County-resolution overrides for the California config. `config.california.yaml` +# ships this same block commented out; snakemake's ``--config`` performs a +# recursive dict update, so only the listed sub-keys are replaced. California has +# 58 counties, and ``simpl: county`` selects the county-FIPS fast path in +# ``cluster_simpl``. +CALIFORNIA_COUNTY_OVERRIDE = [ + "scenario={clusters: [58], simpl: ['county']}", + "model_topology={topological_boundaries: 'county'}", +] + + @pytest.mark.fast @pytest.mark.parametrize( - "configfile,target", + "configfile,target,overrides", [ - ("repo_data/config/config.tutorial.yaml", "cluster_network"), - ("repo_data/config/config.tutorial.yaml", "solve_network"), - ("repo_data/config/config.default.yaml", "cluster_network"), - ("repo_data/config/config.test.yaml", "cluster_network"), + ("repo_data/config/config.tutorial.yaml", "cluster_network", []), + ("repo_data/config/config.tutorial.yaml", "solve_network", []), + ("repo_data/config/config.default.yaml", "cluster_network", []), + ("repo_data/config/config.test.yaml", "cluster_network", []), + ("repo_data/config/config.california.yaml", "cluster_network", []), + ("repo_data/config/config.california.yaml", "solve_network", []), + ( + "repo_data/config/config.california.yaml", + "cluster_network", + CALIFORNIA_COUNTY_OVERRIDE, + ), ], + ids=lambda v: "+".join(v) if isinstance(v, list) else v, ) -def test_snakemake_dryrun_resolves(configfile, target): +def test_snakemake_dryrun_resolves(configfile, target, overrides): + cmd = [ + "snakemake", + "-n", + "--configfile", + configfile, + "--until", + target, + "--quiet", + ] + if overrides: + cmd += ["--config", *overrides] result = subprocess.run( - [ - "snakemake", - "-n", - "--configfile", - configfile, - "--until", - target, - "--quiet", - ], + cmd, cwd=WORKFLOW_DIR, capture_output=True, text=True, timeout=60, ) assert result.returncode == 0, ( - f"snakemake -n failed for {configfile} --until {target}\n" + f"snakemake -n failed for {configfile} --until {target} (overrides={overrides})\n" f"stderr:\n{result.stderr}\n" f"stdout (last 50 lines):\n" + "\n".join(result.stdout.splitlines()[-50:]) ) diff --git a/workflow/Snakefile b/workflow/Snakefile index 68dceae2..192e3fc0 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -305,6 +305,26 @@ def validation_figures(wildcards): return [] +BENCHMARK_CPUC_OUTPUTS = [ + "cpuc_capacity_benchmark.csv", + "cpuc_capacity_deviation.pdf", + "cpuc_capacity_composition.pdf", +] + + +def benchmark_figures(wildcards): + """CPUC Baseline Generator List capacity benchmark, opt-in via run.benchmark_cpuc. + + Gated on its own flag rather than on the demand profile: the fleet + benchmark is useful for any California run, whatever drives its load. + Network-free by design — it needs only powerplants.csv and the CPUC + workbook, so enabling it never triggers a model build. + """ + if not config["run"].get("benchmark_cpuc", False): + return [] + return [RESULTS + "cpuc_benchmark/" + figure for figure in BENCHMARK_CPUC_OUTPUTS] + + rule all: input: #"repo_data/dag.jpg", @@ -312,6 +332,7 @@ rule all: electricity_figures, sector_figures, validation_figures, + benchmark_figures, rule data_model: diff --git a/workflow/repo_data/CPUC/servm_benchmark_regions.csv b/workflow/repo_data/CPUC/servm_benchmark_regions.csv new file mode 100644 index 00000000..c273e14a --- /dev/null +++ b/workflow/repo_data/CPUC/servm_benchmark_regions.csv @@ -0,0 +1,8 @@ +servm_region,eia_ba_code,benchmark_region,note +PGE,CISO,CAISO,EIA reports every CAISO plant under the single BA code CISO; PGE/SCE/SDGE cannot be split without geography +SCE,CISO,CAISO,rolled into CAISO for the same reason +SDGE,CISO,CAISO,rolled into CAISO for the same reason +LADWP,LDWP,LADWP, +IID,IID,IID, +NCNC,BANC,NCNC,Balancing Authority of Northern California +NCNC,TIDC,NCNC,Turlock Irrigation District diff --git a/workflow/repo_data/CPUC/servm_out_of_state_units.csv b/workflow/repo_data/CPUC/servm_out_of_state_units.csv new file mode 100644 index 00000000..d3eb6795 --- /dev/null +++ b/workflow/repo_data/CPUC/servm_out_of_state_units.csv @@ -0,0 +1,76 @@ +cpuc_unit_name,servm_region,compare_category,capmax_mw,physical_location,eia_plant_id,evidence +PRIMM_2_SSSBT1,SCE,Battery,200.0,"Clark County, NV",67734,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 67734 Silver State South Storage [SSSB], NV, OP 2025-04-01, 200 MW/800 MWh, BA=CISO pseudo-tie. CPUC's own EIA Plant Name field names the same plant." +MCFLND_5_MSCBT1,SDGE,Battery,185.0,"Yuma County, AZ",67498,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 67498 McFarland C [MCFRC], AZ, OP 2025-05-01, 185 MW/740 MWh, BA=CISO. CPUC's own EIA Plant Name field names the same plant." +ATLASC_5_ACBBT2,SCE,Battery,180.0,"La Paz County, AZ",63798,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 63798 Atlas [ATLB1], AZ La Paz County, BA=AZPS; TransGrid Atlas complex interconnecting to CAISO over the Delaney-Colorado River 500 kV line." +MCFLND_5_MSBSX2_LESR,SDGE,Battery,150.0,"Yuma County, AZ",66637,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 66637 McFarland B Solar and Storage [BESS], AZ, OP 2024-12-01, 150 MW/600 MWh, BA=CISO." +ATLASC_5_ACBBT6,SCE,Battery,150.0,"La Paz County, AZ",68324,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 68324 Atlas IX [ALT9], AZ, OP 2025-10-01, 150.0 MW/600 MWh -- exact MW match; CAISO GIRID places the ATLASC_5_ACBBT* family in La Paz County AZ (queue #1402)." +ATLASC_5_ACBBT3,SCE,Battery,120.0,"La Paz County, AZ",63798,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 63798 Atlas [ATLB1], AZ La Paz County, BA=AZPS; TransGrid Atlas complex interconnecting to CAISO over the Delaney-Colorado River 500 kV line." +SRAPTA_5_DECBT1,SDGE,Battery,112.5,"Yuma County, AZ",67973,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 67973 Sierra Pinta BESS [SP01], AZ Yuma County, 113 MW, OP 2025-10-01; CAISO queue + Leeward Renewable Energy identify SRAPTA as Sierra Pinta Energy Storage, interconnecting over Delaney-Colorado River 500 kV." +TAHOE_2_THOBT3,SCE,Battery,110.0,"Clark County, NV",,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): CAISO Public Queue row 'TAHOE', 300 MW, Clark County NEVADA, POI SCE Eldorado 230 kV (not Lake Tahoe). No EIA-860 or 860M record; the three CPUC blocks sum to the queue's 300 MW." +TAHOE_2_THOBT2,SCE,Battery,109.0,"Clark County, NV",,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): CAISO Public Queue row 'TAHOE', 300 MW, Clark County NEVADA, POI SCE Eldorado 230 kV (not Lake Tahoe). No EIA-860 or 860M record; the three CPUC blocks sum to the queue's 300 MW." +MCFLND_5_MFSBT1,SDGE,Battery,100.0,"Yuma County, AZ",66636,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 66636 McFarland A Solar and Storage [BESS], AZ, OP 2023-12-01, 100 MW/400 MWh, BA=CISO." +TOWNSITE_2_MEADDYN_LESR,SCE,Battery,90.0,"Clark County, NV",60654,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 60654 Townsite Solar Project Hybrid [GEN02], NV, OP 2021-12-01, 90 MW/360 MWh. CPUC's own EIA Plant Name field names the same plant; RID token MEADDYN = Mead dynamic schedule." +CNTFLT_5_CFSBT1,SCE,Battery,89.0,"La Paz County, AZ",68466,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 68466 Centennial Flats [CFB1/CFB2/CFB3], AZ La Paz County, BA=AZPS, COD 2026." +CNTFLT_5_CFSBT2,SCE,Battery,89.0,"La Paz County, AZ",68466,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 68466 Centennial Flats [CFB1/CFB2/CFB3], AZ La Paz County, BA=AZPS, COD 2026." +CNTFLT_5_CFSBT3,SCE,Battery,89.0,"La Paz County, AZ",68466,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 68466 Centennial Flats [CFB1/CFB2/CFB3], AZ La Paz County, BA=AZPS, COD 2026." +YELPIN_2_YP2BT1,SCE,Battery,65.0,"Clark County, NV",66357,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 66357 Yellow Pine Solar [YPES], NV Clark County, OP 2023-07-01, BA=NEVP. CPUC's own EIA Plant Name field names the same plant." +TAHOE_2_THOBT1,SCE,Battery,60.0,"Clark County, NV",,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): CAISO Public Queue row 'TAHOE', 300 MW, Clark County NEVADA, POI SCE Eldorado 230 kV (not Lake Tahoe). No EIA-860 or 860M record; the three CPUC blocks sum to the queue's 300 MW." +YELPIN_2_YP2BT2,SCE,Battery,53.0,"Clark County, NV",66357,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 66357 Yellow Pine Solar [YPES], NV Clark County, OP 2023-07-01, BA=NEVP. CPUC's own EIA Plant Name field names the same plant." +YELPIN_2_YP2BT3,SCE,Battery,32.0,"Clark County, NV",66357,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 66357 Yellow Pine Solar [YPES], NV Clark County, OP 2023-07-01, BA=NEVP. CPUC's own EIA Plant Name field names the same plant." +MSOLAR_2_MS4BT1,SDGE,Battery,10.0,"Maricopa County, AZ",65962,"deep-chase battery resolution (battery_chase_resolved.csv, corrected_disposition=out_of_state_confirmed): EIA 65962 Mesquite Solar 4, LLC [MS4B], AZ, OP 2024-01-01, 10 MW/47.7 MWh, BA=CISO." +TERMEX_2_PL1X3,SDGE,Gas CC,625.0,"Mexicali, Baja California, MEXICO",,"CPUC Long Name = 'TDM' = Termoelectrica de Mexicali, Intergen's 625 MW CC in Mexicali, Baja California, exporting its full output to CAISO over the La Rosita-Imperial Valley 230 kV line. SERVM Region SDGE. No EIA-860 record (Mexican plant)." +MRCHNT_2_PL1X3,SCE,Gas CC,494.58,"Boulder City, Clark County, NV",55077,"CPUC's own 'EIA Plant Name' field resolves to EIA 55077 Desert Star Energy Center, state=NV, BA=CISO, 537.3 MW (formerly El Dorado Energy). A Nevada plant that EIA reports under the CISO balancing authority, so the model's state=='CA' restriction correctly excludes it while the CPUC carries it in SCE." +LAROA2_2_UNITA1,SDGE,Gas CC,322.0,"Mexicali, Baja California, MEXICO",,"CPUC Long Name = 'LR2' = La Rosita Power Complex unit 2, Mexicali, Baja California; SERVM Region SDGE; same cross-border 230 kV path as TDM. No EIA-860 record (Mexican plant)." +Ipp_Repower_1_CC,LADWP,Gas CC,422.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +Ipp_Repower_2_CC,LADWP,Gas CC,422.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +Apex_CC1a,LADWP,Gas CC,397.95,"Apex, Clark County, NV",55514,"CPUC 'EIA Plant Name' = 'Apex Generating Station' (CC1a) -> EIA 55514, state=NV, BA=NEVP, 600.6 MW; CC1b is the duct-fired block of the same LADWP-owned Nevada plant." +Apex_CC1b,LADWP,Gas CC,168.08,"Apex, Clark County, NV",55514,"CPUC 'EIA Plant Name' = 'Apex Generating Station' (CC1a) -> EIA 55514, state=NV, BA=NEVP, 600.6 MW; CC1b is the duct-fired block of the same LADWP-owned Nevada plant." +INTMNT_3_CC_3_ANAHEIM,LADWP,Gas CC,67.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +INTMNT_3_CC_4_ANAHEIM,LADWP,Gas CC,67.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +Intermountain_CC1_ADS,LADWP,Gas CC,37.41,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +Intermountain_CC2_ADS,LADWP,Gas CC,37.41,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +INTMNT_3_CC_3_RIVERSIDE,LADWP,Gas CC,37.09,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +INTMNT_3_CC_4_RIVERSIDE,LADWP,Gas CC,37.09,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +INTMNT_3_CC_3_PASADENA,LADWP,Gas CC,29.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +INTMNT_3_CC_4_PASADENA,LADWP,Gas CC,29.0,"Delta, Millard County, UT",6481,"CPUC 'EIA Plant Name' = 'Intermountain Power Project' -> EIA 6481, state=UT, BA=LDWP. The IPP Repower CC and the Anaheim / Riverside / Pasadena participation shares are all entitlements in the same Utah plant delivered to LADWP over the DC intertie." +Yuma_Axis_1_ST,IID,Gas CT/ICE/Steam,86.7,"Yuma, Yuma County, AZ",120,"Steam half of the same IID Yuma Axis entitlement as Yuma_Axis_1_GT (EIA 120 Yucca, Yuma AZ); the CPUC row carries no EIA Plant Name but shares the unit-name stem and the 1959 in-service date of the Yucca steam units." +Yuma_Axis_1_GT,IID,Gas CT/ICE/Steam,21.0,"Yuma, Yuma County, AZ",120,"CPUC 'EIA Plant Name' = 'Yucca', EIA Generator ID 'GT21' -> EIA 120 Yucca, state=AZ, BA=AZPS, 422.4 MW. IID's Yuma Axis entitlement in the Arizona Yucca plant." +CONTRL_1_OXBOW,SCE,Geothermal,60.0,"Dixie Valley, Churchill County, NV",52015,"CPUC 'EIA Plant Name' = 'Dixie Valley', Long Name 'Dixie Valley Geo'; RID token OXBOW = Oxbow Geothermal, the plant's original operator. PUDL carries EIA 52015 'Terra-Gen Dixie Valley', state=NV, BA=CISO, 70.9 MW. No plant named Dixie Valley exists in California." +Cape_Geothermal_1,SCE,Geothermal,32.0,"Beaver County, UT",68068,"CPUC 'EIA Plant Name' = 'Cape Generating Station', EIA Generator ID 'CP1G1' -> EIA 68068 Cape Generating Station, state=UT, BA=LDWP, 129 MW (Fervo Energy's Cape Station). SERVM Region SCE." +Fish_Lake_Geothermal,SCE,Geothermal,13.0,"Fish Lake Valley, Esmeralda County, NV",66618,"CPUC 'EIA Plant Name' = 'Fish Lake Geothermal' -> EIA 66618 Fish Lake Geothermal, state=NV, BA=NEVP, 26.5 MW; the only EIA plant of that name, and no California geothermal plant carries it. SERVM Region SCE." +Enel_Cove_Fort_2_GIA_BT,LADWP,Geothermal,12.5,"Cove Fort, Beaver County, UT",58570,"Unit name states the plant: Enel Green Power's Cove Fort geothermal plant, EIA 58570, state=UT, BA=PACE, 25 MW. LADWP entitlement; no California plant named Cove Fort exists." +BCTSYS_5_PWXDYN,PGE,Hydro,428.0,"British Columbia, CANADA",,"CAISO dynamic-schedule pseudo-unit. RID decodes as BCTSYS = BC Transmission System, PWX = Powerex (BC Hydro's marketing arm); SERVM Region PGE, SERVM Tech 'Hydro', Insvdt 1989-06-01 (the Pacific DC Intertie era Powerex contract). No EIA record exists because the resource is Canadian." +HOOVER_2_SCEPT,SCE,Hydro,287.0,"Black Canyon, Colorado River, AZ/NV border",154;8902,"Hoover Dam entitlement shares (SCE, Metropolitan Water District, Anaheim, Valley Electric). EIA files Hoover as 154 'Hoover Dam (AZ)' and 8902 'Hoover Dam (NV)'; no part of the plant is in California. The CPUC Long Names read 'Hoover DAM - SCE Share' / 'HOOVER MWD' / 'Hoover Power Plant' / 'HOOVER VEA'." +HOOVER_2_MWDPT,SCE,Hydro,249.0,"Black Canyon, Colorado River, AZ/NV border",154;8902,"Hoover Dam entitlement shares (SCE, Metropolitan Water District, Anaheim, Valley Electric). EIA files Hoover as 154 'Hoover Dam (AZ)' and 8902 'Hoover Dam (NV)'; no part of the plant is in California. The CPUC Long Names read 'Hoover DAM - SCE Share' / 'HOOVER MWD' / 'Hoover Power Plant' / 'HOOVER VEA'." +HOOV_2_ANAPT,SCE,Hydro,40.39,"Black Canyon, Colorado River, AZ/NV border",154;8902,"Hoover Dam entitlement shares (SCE, Metropolitan Water District, Anaheim, Valley Electric). EIA files Hoover as 154 'Hoover Dam (AZ)' and 8902 'Hoover Dam (NV)'; no part of the plant is in California. The CPUC Long Names read 'Hoover DAM - SCE Share' / 'HOOVER MWD' / 'Hoover Power Plant' / 'HOOVER VEA'." +HOOVER_2_VEAPT,SCE,Hydro,16.0,"Black Canyon, Colorado River, AZ/NV border",154;8902,"Hoover Dam entitlement shares (SCE, Metropolitan Water District, Anaheim, Valley Electric). EIA files Hoover as 154 'Hoover Dam (AZ)' and 8902 'Hoover Dam (NV)'; no part of the plant is in California. The CPUC Long Names read 'Hoover DAM - SCE Share' / 'HOOVER MWD' / 'Hoover Power Plant' / 'HOOVER VEA'." +PVERDE_5_SCEDYN,SCE,Nuclear,635.0,"Wintersburg, Maricopa County, AZ",6008,"CPUC 'EIA Plant Name' = 'Palo Verde' -> EIA 6008 Palo Verde, state=AZ, BA=SRP, 4209.6 MW. SCE's 15.8% Palo Verde entitlement carried as a CAISO dynamic schedule (RID suffix SCEDYN)." +MCFLND_5_MSBSX2_SUN,SDGE,Solar,300.0,"Yuma County, AZ",66637,"CPUC's own 'EIA Plant Name' field resolves to EIA 66637 McFarland B Solar and Storage, state=AZ, BA=CISO (450 MW at the plant)." +AGUCAL_5_SOLAR1,SDGE,Solar,290.0,"Yuma County, AZ",57373,"CPUC's own 'EIA Plant Name' field resolves to EIA 57373 Agua Caliente Solar Project, state=AZ, BA=CISO, 637.7 MW." +PRIMM_2_SOLAR1,SCE,Solar,250.0,"Clark County, NV",58644,"CPUC's own 'EIA Plant Name' field resolves to EIA 58644 Silver State Solar Power South, state=NV, BA=CISO, 260.1 MW." +MCFLND_5_MFSSR1,SDGE,Solar,200.0,"Yuma County, AZ",66636,"CPUC's own 'EIA Plant Name' field resolves to EIA 66636 McFarland A Solar and Storage, state=AZ, BA=CISO." +WHITEW_5_WWRSR1,SDGE,Solar,179.0,"Maricopa County, AZ",60572,"CPUC's own 'EIA Plant Name' field resolves to EIA 60572 White Wing Solar, state=AZ, BA=CISO, 179.0 MW -- exact MW match." +Centennial_Flats_Solar_3,SCE,Solar,166.6,"La Paz County, AZ",68466,"Long Name 'Centennial Flats' resolves to EIA 68466 Centennial Flats, state=AZ La Paz County, BA=AZPS, 767.1 MW across solar+storage; same node as the CNTFLT_5_CFSBT* batteries." +CNTFLT_5_CFSSR1,SCE,Solar,166.6,"La Paz County, AZ",68466,"Long Name 'Centennial Flats' resolves to EIA 68466 Centennial Flats, state=AZ La Paz County, BA=AZPS, 767.1 MW across solar+storage; same node as the CNTFLT_5_CFSBT* batteries." +CNTFLT_5_CFSSR2,SCE,Solar,166.6,"La Paz County, AZ",68466,"Long Name 'Centennial Flats' resolves to EIA 68466 Centennial Flats, state=AZ La Paz County, BA=AZPS, 767.1 MW across solar+storage; same node as the CNTFLT_5_CFSBT* batteries." +MSOLAR_2_SOLAR1,SDGE,Solar,165.0,"Maricopa County, AZ",57707,"CPUC's own 'EIA Plant Name' field resolves to EIA 57707 Mesquite Solar 1, state=AZ, BA=CISO, 170 MW." +COPMT2_2_SOLAR2,SCE,Solar,155.0,"Clark County, NV",58017,"CPUC's own 'EIA Plant Name' field resolves to EIA 58017 Copper Mountain Solar 2, state=NV (Boulder City), BA=CISO, 154 MW." +MSOLAR_2_SOLAR3,SDGE,Solar,152.0,"Maricopa County, AZ",60308,"CPUC's own 'EIA Plant Name' field resolves to EIA 60308 Mesquite Solar 3, LLC, state=AZ, BA=CISO, 150 MW." +SUNST2_5_SS2SR1,SDGE,Solar,150.0,"Maricopa County, AZ",63440,"CPUC's own 'EIA Plant Name' field resolves to EIA 63440 Sun Streams 2, state=AZ, BA=CISO, 150 MW -- exact MW match." +YELPIN_2_YP2SR1,SCE,Solar,125.0,"Clark County, NV",67091,"CPUC's own 'EIA Plant Name' field resolves to EIA 67091 Yellow Pine II, state=NV, BA=NEVP, 210 MW; the three CPUC blocks (125/60/65) sum to 250 MW at the same NV site." +MSOLAR_2_SOLAR2,SDGE,Solar,100.81,"Maricopa County, AZ",60307,"CPUC's own 'EIA Plant Name' field resolves to EIA 60307 Mesquite Solar 2, LLC, state=AZ, BA=CISO." +SUNSLR_1_SSVSR1,SCE,Solar,100.0,"Nye County, NV (Amargosa Valley)",59826,"CPUC's own 'EIA Plant Name' field resolves to EIA 59826 Sunshine Valley Solar, state=NV, BA=CISO, 103.5 MW vs CPUC 100 MW." +COPMT4_2_SOLAR4,SCE,Solar,92.0,"Clark County, NV",59814,"CPUC's own 'EIA Plant Name' field resolves to EIA 59814 Copper Mountain Solar 4, LLC, state=NV, BA=CISO, 93.6 MW." +TOWNSITE_2_MEADDYN_SUN,SCE,Solar,90.0,"Clark County, NV",60654,"CPUC's own 'EIA Plant Name' field resolves to EIA 60654 Townsite Solar Project Hybrid, state=NV, BA=WALC, 270 MW; solar half of the Mead dynamic-schedule hybrid whose battery half is TOWNSITE_2_MEADDYN_LESR." +YELPIN_2_YP2SR3,SCE,Solar,65.0,"Clark County, NV",67091,"CPUC's own 'EIA Plant Name' field resolves to EIA 67091 Yellow Pine II, state=NV, BA=NEVP, 210 MW; the three CPUC blocks (125/60/65) sum to 250 MW at the same NV site." +YELPIN_2_YP2SR2,SCE,Solar,60.0,"Clark County, NV",67091,"CPUC's own 'EIA Plant Name' field resolves to EIA 67091 Yellow Pine II, state=NV, BA=NEVP, 210 MW; the three CPUC blocks (125/60/65) sum to 250 MW at the same NV site." +MSOLAR_2_MS4SR4,SDGE,Solar,52.5,"Maricopa County, AZ",65962,"CPUC's own 'EIA Plant Name' field resolves to EIA 65962 Mesquite Solar 4, LLC, state=AZ, BA=CISO." +COPMTN_2_SOLAR1,SCE,Solar,48.0,"Clark County, NV",57205,"CPUC's own 'EIA Plant Name' field resolves to EIA 57205 CM48 (Copper Mountain 48 MW block), state=NV, BA=CISO -- exact MW match." +COPMTN_2_CM10,SCE,Solar,10.0,"Clark County, NV",56944,"CPUC's own 'EIA Plant Name' field resolves to EIA 56944 CM10 (Copper Mountain 10 MW block), state=NV, BA=CISO -- exact MW match." +MCFLND_5_MFAUXL1,SDGE,Solar,6.0,"Yuma County, AZ",67498,"6 MW auxiliary solar block on the MCFLND_5 intertie node, whose other members (MFSSR1/MFSBT1/MSBSX2/MSCBT1) are all EIA-confirmed at the McFarland A/B/C complex in Yuma County AZ (EIA 66636/66637/67498, all BA=CISO). No separate EIA record." +K_Road_Moapa,LADWP,Solar,250.0,"Clark County, NV (Moapa River Indian Reservation)",57859,"CPUC's own 'EIA Plant Name' field resolves to EIA 57859 Moapa Southern Paiute, state=NV, BA=NEVP, 250 MW -- exact MW match. LADWP's K Road Moapa PPA." +Copper_Mountain_3_PV_NT,LADWP,Solar,105.0,"Clark County, NV",58915,"CPUC's own 'EIA Plant Name' field resolves to EIA 58915 Copper Mountain Solar 3, state=NV, BA=LDWP, 255 MW; the two CPUC blocks are LADWP's 105 MW shares." +Copper_Mountain_3_02,LADWP,Solar,105.0,"Clark County, NV",58915,"CPUC's own 'EIA Plant Name' field resolves to EIA 58915 Copper Mountain Solar 3, state=NV, BA=LDWP, 255 MW; the two CPUC blocks are LADWP's 105 MW shares." +ENERSJ_2_WIND,SDGE,Wind,151.0,"La Rumorosa, Baja California, MEXICO",,"CPUC 'EIA Plant Name' = 'Energia Sierra Suarez' (workbook typo) and Long Name = 'ESJ Wind Energy'; SERVM Region SDGE. Energia Sierra Juarez is InterGen/IEnova's wind farm in Baja California, Mexico, delivering to SDG&E over the ESJ 230 kV cross-border line. No EIA-860 record exists because the plant is Mexican." +ENERSJ_5_ESJWD2,SDGE,Wind,105.0,"La Rumorosa, Baja California, MEXICO",,CPUC Long Name = 'Energia Sierra Juarez Wind 2'; SERVM Region SDGE; same cross-border 230 kV interconnection as ENERSJ_2_WIND. No EIA-860 record (Mexican plant). +CMRRON_2_CMNWD1,SDGE,Wind,300.0,"La Rumorosa, Tecate, Baja California, MEXICO",,"CPUC Long Name = 'Cimarron Wind' = Cimarron, the third phase of Sempra Infrastructure's Energia Sierra Juarez (ESJ) wind complex in La Rumorosa, Baja California, Mexico; ~320 MW nameplate / 300 MW contracted under a 20-year PPA to Silicon Valley Power (City of Santa Clara), delivered over the same existing cross-border 230 kV line as ENERSJ_2_WIND and ENERSJ_5_ESJWD2 into the East County (ECO) Substation in San Diego County. Began generating October 2025, COD expected 1H 2026, consistent with the CPUC 2025-12-01 in-service date; workbook SERVM Region = SDGE and SERVM shape = WIND_V2025E_EIA_ARIZONA (CPUC itself uses a proxy profile). No EIA-860 record exists because the plant is Mexican. Sources: https://www.sempra.com/newsroom/press-releases/sempra-infrastructure-launches-cimarron-wind-project ; https://www.power-technology.com/projects/cimarron-wind-farm-baja-california-mexico/ ; Sempra Form 10-Q FY2025 Q3 https://www.sec.gov/Archives/edgar/data/1032208/000103220825000065/sre-20250930.htm" diff --git a/workflow/repo_data/CPUC/servm_region_map.csv b/workflow/repo_data/CPUC/servm_region_map.csv new file mode 100644 index 00000000..bc2b3c24 --- /dev/null +++ b/workflow/repo_data/CPUC/servm_region_map.csv @@ -0,0 +1,12 @@ +balancing_area,servm_region,note +CISO-PGAE,PGE, +CISO-SCE,SCE, +CISO-SDGE,SDGE, +IID,IID, +LDWP,LADWP, +BANC,NCNC, +TIDC,NCNC, +CISO-VEA,,Nevada footprint; excluded from CA-only networks +PACW,,California sliver served by PacifiCorp West (Siskiyou/Del Norte/Modoc); load lives in CPUC's non-CA PACW region file +WALC,,California sliver served by WAPA Desert Southwest (Colorado River); not covered by the CPUC CA-region files +NEVP,,California sliver served by NV Energy (Tahoe/CalNeva); not covered by the CPUC CA-region files diff --git a/workflow/repo_data/CPUC/servm_tech_map.csv b/workflow/repo_data/CPUC/servm_tech_map.csv new file mode 100644 index 00000000..5f8bd6f8 --- /dev/null +++ b/workflow/repo_data/CPUC/servm_tech_map.csv @@ -0,0 +1,52 @@ +side,source_category,compare_category,note +cpuc,CC,Gas CC, +cpuc,CT,Gas CT/ICE/Steam, +cpuc,ICE,Gas CT/ICE/Steam, +cpuc,Steam,Gas CT/ICE/Steam,CA gas steam boilers; EIA maps 'Natural Gas Steam Turbine' to OCGT so it cannot be split out on the model side +cpuc,Cogen,Gas Cogen/CHP,CPUC-only bucket; EIA/PyPSA carriers have no CHP concept so these units land in CCGT/OCGT on the model side +cpuc,Coal,Coal, +cpuc,Nuclear,Nuclear, +cpuc,Geothermal,Geothermal, +cpuc,Biomass/Wood,Biomass, +cpuc,Biogas,Biomass, +cpuc,Hydro,Hydro, +cpuc,PSH,PSH, +cpuc,Wind,Wind, +cpuc,OOS_Wind,Wind,out-of-state wind contracted to a CA region; absent from the CA subset of the 2026 baseline +cpuc,Solar_1Axis,Solar, +cpuc,Solar_Fixed,Solar, +cpuc,Solar_2Axis,Solar, +cpuc,Solar_Thermal,Solar, +cpuc,Paired_Solar_1Axis,Solar, +cpuc,Paired_Solar_Fixed,Solar, +cpuc,Hybrid_Solar_1Axis,Solar, +cpuc,Hybrid_Solar_Fixed,Solar, +cpuc,Battery_4h,Battery, +cpuc,Paired_BattStorage,Battery, +cpuc,Hybrid_BattStorage,Battery, +cpuc,DR,Demand Response,CPUC-only; PyPSA-USA has no demand-response resource +cpuc,Pumping_Load,Pumping Load,CPUC-only; modelled as load rather than as a resource in PyPSA-USA +pypsa,CCGT,Gas CC, +pypsa,CCGT-95CCS,Gas CC, +pypsa,OCGT,Gas CT/ICE/Steam,EIA 'Natural Gas Steam Turbine' / 'Combustion Turbine' / 'Internal Combustion Engine' all map here +pypsa,hydrogen_ct,Gas CT/ICE/Steam, +pypsa,coal,Coal, +pypsa,nuclear,Nuclear, +pypsa,geothermal,Geothermal, +pypsa,EGS,Geothermal, +pypsa,biomass,Biomass, +pypsa,waste,Biomass,municipal solid waste; CPUC folds it into Biogas +pypsa,hydro,Hydro,pumped storage is split off before mapping via prime_mover_code == PS +pypsa,PHS,PSH, +pypsa,onwind,Wind, +pypsa,offwind,Wind, +pypsa,offwind_floating,Wind, +pypsa,solar,Solar, +pypsa,battery,Battery, +pypsa,2hr_battery_storage,Battery, +pypsa,4hr_battery_storage,Battery, +pypsa,6hr_battery_storage,Battery, +pypsa,8hr_battery_storage,Battery, +pypsa,10hr_battery_storage,Battery, +pypsa,oil,Oil,model-only; CPUC has no oil bucket for CA +pypsa,other,Other,"model-only catch-all (flywheels, other gases, compressed air)" diff --git a/workflow/repo_data/config/config.california.yaml b/workflow/repo_data/config/config.california.yaml new file mode 100644 index 00000000..05bd6d25 --- /dev/null +++ b/workflow/repo_data/config/config.california.yaml @@ -0,0 +1,74 @@ +# ==================================================================== +# PyPSA-USA — California (CPUC SERVM 2026) +# ==================================================================== +# CA-only network carved out of the Western Interconnect, with WECC +# imports/exports at the boundary and demand from the CPUC 2026 SERVM +# update (hourly, per CA load region, weather years 2000-2024 via +# `electricity: demand: scenario: servm_weather_years`). +# +# cd workflow +# uv run snakemake -j1 --configfile repo_data/config/config.california.yaml +# +# This is a SPARSE OVERLAY over the auto-loaded layered base — only keys +# that differ from the base belong here. Lists are replaced wholesale. +# See docs/source/california-model.md for the full data inventory. +# ==================================================================== + +run: + name: "california" + benchmark_cpuc: true # CPUC Baseline Generator List capacity benchmark (network-free; horizon from run.benchmark_cpuc_horizons) + +scenario: + # must be CPUC SERVM forecast years: {2026, 2028, 2030, 2032, 2035, 2037, 2040, 2042, 2045} + planning_horizons: [2030, 2035, 2040, 2045] + clusters: [4] # one bus per CA ReEDS zone (p8/p9/p10/p11) + scope: "total" + + # ---- county resolution (uncomment to switch) -------------------- + # California has 58 counties; `simpl: county` selects the county-FIPS + # fast path in cluster_simpl, and the imports machinery automatically + # picks the county-level NARIS flowgate file. + # clusters: [58] + # simpl: ['county'] + +model_topology: + interface_transmission_limits: true # aggregate RESOLVE CAISO import/export caps + include: + reeds_state: ['CA'] + # topological_boundaries: 'county' # pair with the county block above + +# Historical profiles matching the SERVM demand weather year. The registry's +# Oak mirror serves screened years 1980-2022 (Zenodo alone carries 2012 only). +# Climate scenarios (rcp*) publish horizons 2030/2040/2050 only, so they cannot +# back this config's 2035/2045 planning horizons — the registry fails at parse +# time if you switch to one without restricting planning_horizons. +# Keep `renewable_weather_years` == `electricity.demand.scenario.servm_weather_years` +# (both default to 2019 in the base) so load and renewables share a weather year. +renewable_scenarios: ['historical'] + +electricity: + extendable_carriers: + Generator: [solar, onwind, offwind_floating, OCGT, CCGT, CCGT-95CCS, nuclear, hydrogen_ct] + remote_contracted_resources: + enable: true # attach CA-contracted out-of-state units (Palo Verde, Intermountain, Hoover, ...) at CA buses + demand: + profile: servm + imports: + enable: true + volume_limit: 25 # % of annual CA load + balancing_period: year + exports: + enable: true + volume_limit: 25 + balancing_period: year + +conventional: + unit_commitment: true # linearized UC (solving.options.linearized_unit_commitment in the base) + +clustering: + aggregation_strategies: + generators: + p_min_pu: 'capacity_weighted_average' # keep the UC feasibility invariant through clustering + +solving: + mem: 30000 diff --git a/workflow/repo_data/config/config.common.yaml b/workflow/repo_data/config/config.common.yaml index f93ae5ae..44aa705e 100644 --- a/workflow/repo_data/config/config.common.yaml +++ b/workflow/repo_data/config/config.common.yaml @@ -37,7 +37,7 @@ # Points the build scripts at a specific PUDL release. Update the # version tag when you want fresher EIA-860/EIA-923/CEMS tables. # docs : PUDL -pudl_path: s3://pudl.catalyst.coop/v2025.5.0 # versioned S3 prefix to PUDL parquet outputs (or a local file:// path) +pudl_path: s3://pudl.catalyst.coop/v2026.8.0 # versioned S3 prefix to PUDL parquet outputs (or a local file:// path) # ==================================================================== diff --git a/workflow/repo_data/config/config.default.yaml b/workflow/repo_data/config/config.default.yaml index 2eb14113..c23dd312 100644 --- a/workflow/repo_data/config/config.default.yaml +++ b/workflow/repo_data/config/config.default.yaml @@ -48,6 +48,8 @@ run: shared_resources: false # true = reuse resources/ across all runs (DAG-keyed); false = isolate per run.name shared_cutouts: true # true = share atlite cutouts across runs (recommended; cutouts are large and slow) validation: false # true = wire in back-casting validation plots (historical-only) + benchmark_cpuc: false # true = compare installed capacity against the CPUC Baseline Generator List (California runs) + benchmark_cpuc_horizons: [2026] # benchmark year(s); 2026 = today's fleet vs today's baseline (network-free). Empty list = use scenario planning_horizons # NB: the whole `renewable:` block — including `renewable.dataset` # (atlite | godeeep), which used to be duplicated here — lives in @@ -157,6 +159,7 @@ electricity: conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal, biomass, waste] # plant types attached as existing capacity from EIA-860 renewable_carriers: [onwind, offwind_floating, solar, hydro] # variable renewables attached as existing capacity retirement: economic # economic = allow early retirement at fixed cost; technical = lock p_nom_min to p_nom + honor_planned_retirements: true # drop existing/proposed units at their EIA planned_generator_retirement_date (false = keep every existing unit indefinitely) # Capacity-expansion candidates. Carriers listed here get extendable # p_nom and pay annualized capex; carriers listed only above are @@ -167,6 +170,17 @@ electricity: Store: [] # candidate Store carriers (e.g. H2_tank); requires sector coupling Link: [] # candidate Link carriers (e.g. HVDC) + # ---------------- Out-of-state contracted resources (California) ---------------- + # Adds the physically out-of-state resources the CPUC ledger attributes to + # California load regions (Palo Verde's SCE share, LADWP's Intermountain and + # Apex, the Hoover entitlements, ~2 GW of AZ/NV solar and battery contracts) + # back into a California-scoped model as generators at California buses, at + # their CONTRACTED capacity. Requires `demand: profile: servm` (the SERVM load + # weights decide which bus each region's contracts attach to). + remote_contracted_resources: + enable: false # attach CA-contracted out-of-state units at CA buses + file: repo_data/CPUC/servm_out_of_state_units.csv + # ---------------- Reserves & resource adequacy ---------------- SAFE_reservemargin: 0.14 # SAFE planning reserve margin (fraction of peak load); used with SAFE opt SAFE_regional_reservemargins: 'repo_data/config/policy_constraints/SAFE_regional_prm.csv' @@ -188,13 +202,20 @@ electricity: # ---------------- Demand ---------------- demand: - profile: efs # demand time series source; efs (EIA-EFS) | eia (historical actuals) | eer (EER scenarios) + profile: efs # demand time series source; efs (EIA-EFS) | eia (historical actuals) | eer (EER scenarios) | servm (CPUC SERVM, California only) bus_allocation: population # per-bus demand weight; population (2020 census counties) | breakthrough (legacy BE Pd) - scenario: # EFS/EER scenario knobs (ignored if profile=eia) + scenario: # EFS/EER/SERVM scenario knobs (ignored if profile=eia) efs_case: reference # reference | medium | high efs_speed: moderate # slow | moderate | rapid eer_file: demand_EER2025_100by2050.h5 # EER h5 (profile=eer): demand_EER2025_100by2050 | demand_EER2025_Baseline_AEO2023 | demand_EER2025_IRAlow aeo: reference # AEO (EIA Annual Energy Outlook) scaling case; reference | high | low + # ---- profile: servm (CPUC SERVM 2026, California regions) ---- + # Weather year drawn from the stacked 2000-2024 SERVM record. Single-entry + # list = deterministic; multiple entries are reserved for stochastic + # scenarios (phase 3) and currently raise NotImplementedError. + # RECOMMENDED: keep equal to top-level `renewable_weather_years` so load + # and renewable profiles share a weather year (mismatch logs a warning). + servm_weather_years: [2019] demand_response: # price-responsive shiftable load; 0 = disabled shift: 0 # fraction of hourly load that can shift in time @@ -233,6 +254,20 @@ conventional: pudl: true # PUDL receipts-based fuel cost time series (monthly resolution) wholesale: true # CAISO/wholesale natural-gas hub prices (overrides PUDL for NG) + # PHASE 2 (NOT IMPLEMENTED): ambient-temperature capacity derates for thermal + # units, from the CPUC SERVM unit-specific hourly derate profiles. Setting + # `enable: true` raises NotImplementedError in add_electricity — the profiles + # are not retrieved or applied yet (see the reserved + # `retrieve_cpuc_thermal_derate` rule in workflow/rules/retrieve.smk). + # When implemented it REPLACES the EIA-860 summer/winter seasonal derate + # rather than stacking on it; stacking an ambient derate on a seasonal derate + # (or on a UCAP-derated capacity credit) double-counts the same thermal + # deficiency and must never be done. + ambient_derate: + enable: false # PHASE 2 — leave false + source: cpuc_servm # CPUC SERVM unit-specific hourly derates + replaces_seasonal_derate: true # never stack with the EIA-860 seasonal derate + # ==================================================================== # LINES — AC transmission lines @@ -373,11 +408,11 @@ clustering: generators: build_year: 'capacity_weighted_average' lifetime: 'capacity_weighted_average' - start_up_cost: 'capacity_weighted_average' + start_up_cost: sum # aggregate acts as one unit: starting it starts the members; cap-weighted average understates cycling cost ~N-fold min_up_time: 'capacity_weighted_average' min_down_time: 'capacity_weighted_average' - ramp_limit_up: max - ramp_limit_down: max + ramp_limit_up: capacity_weighted_average # max would make the cluster as flexible as its single nimblest member + ramp_limit_down: capacity_weighted_average committable: any vom_cost: mean fuel_cost: mean diff --git a/workflow/repo_data/config/config.test.california.yaml b/workflow/repo_data/config/config.test.california.yaml new file mode 100644 index 00000000..3db93864 --- /dev/null +++ b/workflow/repo_data/config/config.test.california.yaml @@ -0,0 +1,61 @@ +# ==================================================================== +# PyPSA-USA — SERVM Test Configuration (Tier B integration harness) +# ==================================================================== +# CA-only Western slice with CPUC SERVM demand, built to `add_demand` +# by the tests/integration `servm_built` fixture. DO NOT use as a +# user-facing entry point. +# +# SPARSE OVERLAY over the auto-loaded layered base — only keys that +# differ belong here; lists are replaced wholesale. +# +# Snapshots stay the base's full 8760-hour year: `ReadServm` maps each +# SERVM hourly strip positionally onto the network's own snapshots and +# raises unless the planning horizon carries exactly 8760 of them, so a +# short-window harness is impossible. Runtime is held down by the small +# `simpl` / `clusters` values instead. +# ==================================================================== + +run: + name: "test-california" + disable_progressbar: true + +scenario: + clusters: [4m] + simpl: [20] + scope: "total" + planning_horizons: [2030] # must be a CPUC SERVM forecast year + +model_topology: + interface_transmission_limits: true + include: + reeds_state: ['CA'] + +electricity: + renewable_carriers: [onwind, solar, hydro] + extendable_carriers: + Generator: [solar, onwind, OCGT, CCGT] + StorageUnit: [4hr_battery_storage] + remote_contracted_resources: + enable: true # Tier B coverage of the remote-attach path + demand: + profile: servm + imports: + enable: true + volume_limit: 25 + balancing_period: year + exports: + enable: true + volume_limit: 25 + balancing_period: year + +conventional: + unit_commitment: true + +clustering: + aggregation_strategies: + generators: + p_min_pu: 'capacity_weighted_average' # keep the UC feasibility invariant through clustering + +solving: + options: + transmission_losses: 0 diff --git a/workflow/rules/build_electricity.smk b/workflow/rules/build_electricity.smk index 566d3982..9274be16 100644 --- a/workflow/rules/build_electricity.smk +++ b/workflow/rules/build_electricity.smk @@ -362,6 +362,15 @@ def eer_demand_file(): return filename +SERVM_LOAD_FILE = "cpuc/servm/HourlyLoad_CA_Regions_V2025E_2224_Mon_{year}.csv" + + +def servm_demand_files(): + """One CPUC SERVM hourly-load file per planning horizon.""" + horizons = sorted(set(config["scenario"]["planning_horizons"])) + return [DATA + SERVM_LOAD_FILE.format(year=year) for year in horizons] + + def demand_raw_data(wildcards): # get profile to use end_use = wildcards.end_use @@ -393,6 +402,8 @@ def demand_raw_data(wildcards): return DATA + f"nrel_efs/EFSLoadProfile_{efs_case}_{efs_speed}.csv" elif profile == "eer": return DATA + f"eer/{eer_demand_file()}" + elif profile == "servm": + return servm_demand_files() elif profile == "ferc": return [ DATA + "pudl/out_ferc714__hourly_estimated_state_demand.parquet", @@ -428,13 +439,23 @@ def demand_raw_data(wildcards): def demand_disaggregate_data(wildcards): - """CLIU county-level industrial loads are the only disaggregation input. + """Extra per-profile input needed to spread zonal demand over buses. - All other end uses disaggregate by population and need no extra file. + CLIU county-level industrial loads serve the industry end use; the SERVM + profile needs its precomputed (region, bus) allocation weights. Everything + else disaggregates by population and needs no extra file. """ - if wildcards.end_use != "industry": - return [] - return DATA + "industry_load/2014_update_20170910-0116.csv" + if wildcards.end_use == "industry": + return DATA + "industry_load/2014_update_20170910-0116.csv" + if ( + wildcards.end_use == "power" + and config["electricity"]["demand"]["profile"] == "servm" + ): + return ( + DEMAND + + f"{wildcards.interconnect}/servm_load_weights_s{wildcards.simpl}.csv" + ) + return [] def demand_scaling_data(wildcards): @@ -457,6 +478,8 @@ def demand_scaling_data(wildcards): return [] elif profile == "eer": return [] + elif profile == "servm": + return [] else: return "" @@ -470,14 +493,20 @@ rule build_electrical_demand: profile_year=pd.to_datetime(config["snapshots"]["start"]).year, planning_horizons=config["scenario"]["planning_horizons"], renewable_weather_years=config["renewable_weather_years"], + servm_weather_years=config["electricity"]["demand"]["scenario"].get( + "servm_weather_years", [] + ), snapshots=config["snapshots"], pudl_path=config_provider("pudl_path"), input: network=NETWORKS + "{interconnect}/elec_s{simpl}.nc", demand_files=demand_raw_data, + dissagregate_files=demand_disaggregate_data, demand_scaling_file=demand_scaling_data, output: elec_demand=DEMAND + "{interconnect}/{end_use}_electricity_s{simpl}.csv", + zonal_components=DEMAND + + "{interconnect}/{end_use}_zonal_components_s{simpl}.parquet", log: LOGS + "{interconnect}/{end_use}_build_demand_s{simpl}.log", benchmark: @@ -732,6 +761,27 @@ rule build_powerplants: "../scripts/build_powerplants.py" +def remote_contracted_resource_files(wildcards): + """CPUC out-of-state contracted units, plus what is needed to place them. + + Only requested when ``electricity.remote_contracted_resources.enable`` is on, + so the option is fully inert (and pulls no extra inputs) by default. The + SERVM load weights supply the (region -> California bus) attribution and the + tech map reconciles the ledger's `compare_category` with model carriers. + """ + cfg = config["electricity"].get("remote_contracted_resources", {}) or {} + if not cfg.get("enable", False): + return {} + return { + "remote_resources": cfg.get( + "file", "repo_data/CPUC/servm_out_of_state_units.csv" + ), + "servm_tech_map": "repo_data/CPUC/servm_tech_map.csv", + "servm_load_weights": DEMAND + + f"{wildcards.interconnect}/servm_load_weights_s{wildcards.simpl}.csv", + } + + rule add_electricity: params: length_factor=config["lines"]["length_factor"], @@ -743,8 +793,10 @@ rule add_electricity: costs=config["costs"], planning_horizons=config["scenario"]["planning_horizons"], eia_api=config["api"]["eia"], + remote_contracted=config["electricity"].get("remote_contracted_resources", {}), input: unpack(dynamic_fuel_price_files), + unpack(remote_contracted_resource_files), **( { # For GODEEEP future scenarios: pass all horizon-specific profiles @@ -894,6 +946,26 @@ rule cluster_resources: "../scripts/cluster_simpl.py" +rule build_servm_load_weights: + input: + network=NETWORKS + "{interconnect}/elec_base_network.nc", + busmap_b=BUSMAPS + "{interconnect}/busmap_b.csv", + busmap_s=BUSMAPS + "{interconnect}/busmap_s{simpl}.csv", + region_map="repo_data/CPUC/servm_region_map.csv", + output: + weights=DEMAND + "{interconnect}/servm_load_weights_s{simpl}.csv", + log: + LOGS + "{interconnect}/build_servm_load_weights_s{simpl}.log", + threads: 1 + resources: + mem_mb=lambda wildcards, input, attempt: (input.size // 200000) * attempt * 2, + walltime=config_provider( + "walltime", "build_servm_load_weights", default="00:20:00" + ), + script: + "../scripts/build_servm_load_weights.py" + + rule cluster_network: params: cluster_network=config_provider("clustering", "cluster_network"), diff --git a/workflow/rules/retrieve.smk b/workflow/rules/retrieve.smk index 77ef1136..64c6bed4 100644 --- a/workflow/rules/retrieve.smk +++ b/workflow/rules/retrieve.smk @@ -103,6 +103,46 @@ rule retrieve_eer_demand_data: "../scripts/retrieve_eer_data.py" +CPUC_SERVM_URL = "https://files.cpuc.ca.gov/energy/modeling/2026_servm_updates/" + + +rule retrieve_cpuc_servm_load: + wildcard_constraints: + servm_year="2026|2028|2030|2032|2035|2037|2040|2042|2045", + params: + url=lambda wildcards: CPUC_SERVM_URL + + f"HourlyLoad_CA_Regions_V2025E_2224_Mon_{wildcards.servm_year}.csv", + output: + DATA + "cpuc/servm/HourlyLoad_CA_Regions_V2025E_2224_Mon_{servm_year}.csv", + resources: + mem_mb=5000, + log: + "logs/retrieve/retrieve_cpuc_servm_load_{servm_year}.log", + retries: 2 + script: + "../scripts/retrieve_cpuc_data.py" + + +rule retrieve_cpuc_baseline_generators: + params: + url=CPUC_SERVM_URL + "BaselineGeneratorList_CAISO.xlsx", + output: + DATA + "cpuc/BaselineGeneratorList_CAISO.xlsx", + resources: + mem_mb=5000, + log: + "logs/retrieve/retrieve_cpuc_baseline_generators.log", + retries: 2 + script: + "../scripts/retrieve_cpuc_data.py" + + +# RESERVED (phase 2): `retrieve_cpuc_thermal_derate` will pull the CPUC SERVM +# unit-specific ambient-temperature derate profiles that back the +# `conventional.ambient_derate` config hook. Until it lands, enabling that key +# raises NotImplementedError in add_electricity. + + sector_datafiles = [ # heating sector "population/DECENNIALDHC2020.P1-Data.csv", diff --git a/workflow/rules/solve_electricity.smk b/workflow/rules/solve_electricity.smk index 0dbc73b6..3e487a84 100644 --- a/workflow/rules/solve_electricity.smk +++ b/workflow/rules/solve_electricity.smk @@ -26,6 +26,7 @@ rule solve_network: safer_reeds="repo_data/config/policy_constraints/reeds/prm_annual.csv", rps_reeds="repo_data/config/policy_constraints/reeds/rps_fraction.csv", ces_reeds="repo_data/config/policy_constraints/reeds/ces_fraction.csv", + interface_limits="repo_data/config/policy_constraints/transmission_interface_limits.csv", pop_layout=pop_layout_input, ev_policy=ev_policy_input, output: diff --git a/workflow/rules/validate.smk b/workflow/rules/validate.smk index 691e8ef4..7ef9fa41 100644 --- a/workflow/rules/validate.smk +++ b/workflow/rules/validate.smk @@ -9,6 +9,7 @@ rule solve_network_validation: safer_reeds="repo_data/config/policy_constraints/reeds/prm_annual.csv", rps_reeds="repo_data/config/policy_constraints/reeds/rps_fraction.csv", ces_reeds="repo_data/config/policy_constraints/reeds/ces_fraction.csv", + interface_limits="repo_data/config/policy_constraints/transmission_interface_limits.csv", output: network=RESULTS + "{interconnect}/networks/elec_s{simpl}_c{clusters}_ec_l{ll}_{opts}_{sector}_operations.nc", @@ -66,3 +67,34 @@ rule plot_validation_figures: mem_mb=5000, script: "../scripts/plot_validation_production.py" + + +# Compares installed capacity against the CPUC Baseline Generator List, +# aggregated by SERVM benchmark region and technology, per planning horizon. +# Gated by run.benchmark_cpuc (see benchmark_figures in the Snakefile). +# Deliberately network-free: a fleet benchmark (default horizon 2026 = today's +# plants) compares powerplants.csv against the CPUC workbook and must not drag +# in a network build. run.benchmark_cpuc_horizons overrides the scenario +# planning_horizons so the benchmark year is decoupled from the study years. +rule benchmark_cpuc_baseline: + params: + planning_horizons=config_provider("scenario", "planning_horizons"), + benchmark_horizons=config_provider("run", "benchmark_cpuc_horizons", default=[]), + input: + powerplants="resources/powerplants/powerplants.csv", + cpuc_baseline=DATA + "cpuc/BaselineGeneratorList_CAISO.xlsx", + region_map="repo_data/CPUC/servm_benchmark_regions.csv", + tech_map="repo_data/CPUC/servm_tech_map.csv", + out_of_state="repo_data/CPUC/servm_out_of_state_units.csv", + output: + comparison=RESULTS + "cpuc_benchmark/cpuc_capacity_benchmark.csv", + heatmap=RESULTS + "cpuc_benchmark/cpuc_capacity_deviation.pdf", + composition=RESULTS + "cpuc_benchmark/cpuc_capacity_composition.pdf", + log: + LOGS + "benchmark_cpuc_baseline.log", + threads: 1 + resources: + walltime="00:20:00", + mem_mb=5000, + script: + "../scripts/benchmark_cpuc_baseline.py" diff --git a/workflow/schemas/config.schema.yaml b/workflow/schemas/config.schema.yaml index dd2c5621..faabe09a 100644 --- a/workflow/schemas/config.schema.yaml +++ b/workflow/schemas/config.schema.yaml @@ -55,6 +55,8 @@ properties: shared_resources: {type: boolean} shared_cutouts: {type: boolean} validation: {type: boolean} + benchmark_cpuc: {type: boolean} + benchmark_cpuc_horizons: {type: array, items: {type: integer}} scenarios: description: Optional multi-scenario driver; see rules/common.smk config_provider. type: object @@ -216,6 +218,15 @@ properties: conventional_carriers: {type: array, items: {type: string}} renewable_carriers: {type: array, items: {type: string}} retirement: {enum: [economic, technical]} + honor_planned_retirements: {type: boolean} + + remote_contracted_resources: + description: Attach CA-contracted out-of-state units at CA buses (see data-generators.md). + type: object + additionalProperties: false + properties: + enable: {type: boolean} + file: {type: string} extendable_carriers: type: object @@ -249,7 +260,7 @@ properties: type: object additionalProperties: false properties: - profile: {enum: [efs, eia, eer]} + profile: {enum: [efs, eia, eer, servm]} bus_allocation: {enum: [population, breakthrough]} scenario: type: object @@ -259,6 +270,7 @@ properties: efs_speed: {enum: [slow, moderate, rapid]} eer_file: {type: string} aeo: {enum: [reference, high, low]} + servm_weather_years: {type: array, items: {type: integer, minimum: 2000, maximum: 2024}} demand_response: type: object @@ -295,6 +307,15 @@ properties: properties: unit_commitment: {type: boolean} must_run: {type: boolean} + ambient_derate: + description: PHASE-2 stub — enabling raises NotImplementedError. + type: object + additionalProperties: false + properties: + enable: {type: boolean} + source: {type: string} + scope: {type: array, items: {type: string}} + replaces_seasonal_derate: {type: boolean} dynamic_fuel_price: type: object additionalProperties: false diff --git a/workflow/scripts/add_electricity.py b/workflow/scripts/add_electricity.py index 56324251..c0c45b88 100755 --- a/workflow/scripts/add_electricity.py +++ b/workflow/scripts/add_electricity.py @@ -35,6 +35,17 @@ # see filter_plants_by_region / _drop_distant_seam_plants. SEAM_PLANT_MAX_KM = 100.0 +# Index prefix for California's out-of-state CONTRACTED resources, mirroring the +# "C" prefix attach_conventional_generators puts on the physical fleet. +REMOTE_PREFIX = "R " +# Carriers whose availability comes from a per-bus capacity-factor profile +# (attach_wind_and_solar). `hydro` is deliberately absent: its profiles come from +# the breakthrough dataset and remote hydro is handled as a firm unit instead. +VRE_PROFILE_CARRIERS = ("solar", "onwind", "offwind", "offwind_floating") +# Duration (hours) assumed for a remote battery contract whose plant reports no +# energy_storage_capacity_mwh. +DEFAULT_REMOTE_DURATION_H = 4.0 + def sanitize_carriers(n, config): """ @@ -221,6 +232,7 @@ def load_powerplants( plants_fn, investment_periods: list[int], interconnect: str | None = None, + honor_planned_retirements: bool = True, ) -> pd.DataFrame: plants = pd.read_csv( plants_fn, @@ -242,9 +254,30 @@ def load_powerplants( "current_planned_generator_operating_date", ].dt.year - # If operational_status is existing or proposed, replace generator_retirement_date with 1/1/2100 - retirement_date = pd.to_datetime("2100-01-01") - plants.loc[plants.operational_status.isin(["existing", "proposed"]), "generator_retirement_date"] = retirement_date + # Existing/proposed units have no *actual* retirement date yet. Honor the + # EIA announced date where one exists (e.g. the CAISO OTC steamers Ormond + # Beach / Alamitos / Huntington Beach, ~2.9 GW that would otherwise stay + # in the fleet forever); units without an announcement fall back to the + # far-future sentinel. Announced retirements only bite at the first + # investment period: after clustering, build_year/lifetime are aggregated + # capacity-weighted, so per-period exogenous retirement inside a cluster + # is not representable. + far_future = pd.to_datetime("2100-01-01") + live_mask = plants.operational_status.isin(["existing", "proposed"]) + if honor_planned_retirements: + planned = pd.to_datetime(plants["planned_generator_retirement_date"]) + plants.loc[live_mask, "generator_retirement_date"] = planned[live_mask].fillna(far_future) + announced_out = live_mask & planned.notna() & (planned.dt.year <= investment_periods[0]) + if announced_out.any(): + logger.info( + "Honoring announced retirements: dropping %d units (%.0f MW) with a planned " + "retirement date on or before %d.", + announced_out.sum(), + plants.loc[announced_out, "p_nom"].sum(), + investment_periods[0], + ) + else: + plants.loc[live_mask, "generator_retirement_date"] = far_future # Handle NaT values plants.loc[plants.generator_retirement_date.isna(), "generator_retirement_date"] = pd.to_datetime("1900-01-01") @@ -575,6 +608,15 @@ def attach_conventional_generators( if unit_commitment: for attr in committable_fields: plants[attr] = plants[attr].astype(float).fillna(defaults[attr]) + # p_min_pu is the unit's minimum stable level. For a conventional unit + # p_max_pu is the seasonal derate (apply_seasonal_capacity_derates below), + # so p_min_pu must not exceed the *tighter* of the two seasons: above it + # the only feasible commitment is status = 0 and the unit's capacity + # silently disappears for that whole season. + # build_powerplants.sanitize_uc_parameters already enforces + # this at the source; the clip here is the second line of defence for + # hand-edited or externally supplied powerplants.csv files, and the 0.95 + # factor leaves 5 % headroom against the rounding applied to p_max_pu. plants["p_min_pu"] = ( (plants.minimum_load_mw / plants.p_nom) .clip( @@ -585,6 +627,19 @@ def attach_conventional_generators( .fillna(0) .mul(0.95) ) + # PyPSA defaults `up_time_before` to 1, i.e. every unit is assumed to have + # been online in the snapshot before the horizon. Any unit with + # min_up_time > 1 is then forced to stay online at the start of the + # horizon, and its whole minimum stable level has to be absorbed by the + # very first snapshots — a January night, the annual load minimum. With + # `solving.options.load_shedding: false` that is an outright infeasibility, + # not a cost. This is a greenfield planning model with no warm-start + # state, so start every committable unit with no prior commitment + # obligation in either direction (`down_time_before` already defaults to 0). + plants["up_time_before"] = 0 + # Only committable units may carry a non-zero p_min_pu: on a + # non-committable generator p_min_pu is an unconditional must-run. + committable_fields = [*committable_fields, "p_min_pu", "up_time_before"] else: for attr in committable_fields: plants[attr] = defaults[attr] @@ -963,6 +1018,530 @@ def attach_phs_storage( ) +def _parse_eia_plant_ids(value) -> list[int]: + """Parse the ``eia_plant_id`` cell of the CPUC out-of-state ledger. + + The column is usually a single EIA plant id, but a contract can span + several physical plants (Hoover is EIA 154 *and* 8902, the NV and AZ + halves of the same dam) in which case the ids are ``;``-joined. Empty + cells (Mexico/BC units, pure entitlement rows) yield an empty list. + """ + if value is None or (isinstance(value, float) and np.isnan(value)) or pd.isna(value): + return [] + ids = [] + for token in str(value).replace(",", ";").split(";"): + token = token.strip() + if not token: + continue + try: + ids.append(int(float(token))) + except ValueError: + logger.warning(f"Unparseable eia_plant_id token '{token}' in the remote contracted-resource file.") + return ids + + +def _servm_region_buses(weights: pd.DataFrame, n: pypsa.Network) -> pd.Series: + """Map each SERVM region onto the single network bus carrying most of its load. + + ``build_servm_load_weights`` writes a long (bus, servm_region, laf) table; the + max-LAF bus is the region's load centroid and is where a contracted + out-of-state resource is attributed. Buses absent from the network (a + coarser ``{simpl}`` than the weights file was built for) are dropped first. + """ + w = weights.copy() + w["bus"] = w["bus"].astype(str) + w["laf"] = pd.to_numeric(w["laf"], errors="coerce") + w = w.dropna(subset=["laf"]) + + known = w[w["bus"].isin(n.buses.index)] + if len(known) < len(w): + logger.warning( + f"SERVM load weights reference {len(w) - len(known)} bus(es) absent from the network; ignoring them " + "when picking remote-resource attachment buses.", + ) + if known.empty: + raise ValueError( + "None of the buses in the SERVM load-weights file are present in the network; cannot attribute " + "remote contracted resources to California buses.", + ) + # Deterministic tie-break: highest laf, then lexicographically largest bus. + ranked = known.sort_values(["servm_region", "laf", "bus"]) + return ranked.groupby("servm_region").tail(1).set_index("servm_region")["bus"] + + +def _carrier_category_maps(tech_map: pd.DataFrame | None) -> tuple[dict, dict]: + """Build (model carrier -> compare_category, compare_category -> carrier) maps. + + Both come from ``repo_data/CPUC/servm_tech_map.csv`` (``side == "pypsa"``); + the reverse map keeps the first carrier listed for a category, which the file + orders so that the canonical model carrier comes first (Gas CC -> CCGT, + Battery -> battery, ...). + """ + if tech_map is None or tech_map.empty: + return {}, {} + pypsa_side = tech_map[tech_map["side"] == "pypsa"] + carrier_to_category = dict(zip(pypsa_side["source_category"], pypsa_side["compare_category"], strict=False)) + category_to_carrier: dict[str, str] = {} + for carrier, category in carrier_to_category.items(): + category_to_carrier.setdefault(category, carrier) + return carrier_to_category, category_to_carrier + + +def _cost(costs: pd.DataFrame, carrier: str, field: str, default: float = np.nan) -> float: + """Look up a cost-table field, tolerating carriers the table does not cover.""" + try: + return float(costs.at[carrier, field]) + except (KeyError, TypeError, ValueError): + return default + + +def _select_remote_constituents( + row: pd.Series, + plants_prefilter: pd.DataFrame, + carrier_to_category: dict, + category_to_carrier: dict, +) -> tuple[pd.DataFrame, str, str]: + """Pick the powerplants.csv rows that back one CPUC contract row. + + Returns ``(constituent rows, carrier, note)``. The plant's live generator + rows are first restricted to those whose model carrier maps to the contract's + ``compare_category``; within that subset the dominant (max summed ``p_nom``) + carrier wins. When the plant carries no unit of the contracted category at all + -- e.g. the 6 MW solar entitlement ``MCFLND_5_MFAUXL1`` pointed at EIA 67498, + which powerplants.csv knows only as a battery -- the category's canonical + carrier is used with the whole plant as the techno-economic donor, rather than + silently converting a solar contract into a battery. + """ + ids = _parse_eia_plant_ids(row.get("eia_plant_id")) + live = plants_prefilter[plants_prefilter["plant_id_eia"].isin(ids)] + if live.empty: + return live, "", "no_live_units" + + live = live.copy() + live["compare_category"] = live["carrier"].map(carrier_to_category) + matched = live[live["compare_category"] == row["compare_category"]] + if not matched.empty: + carrier = matched.groupby("carrier")["p_nom"].sum().idxmax() + return matched[matched["carrier"] == carrier], carrier, "ok" + + carrier = category_to_carrier.get(row["compare_category"]) + if carrier is None: + carrier = live.groupby("carrier")["p_nom"].sum().idxmax() + logger.warning( + f"Remote contracted unit '{row['cpuc_unit_name']}' is categorised '{row['compare_category']}' but EIA plant(s) " + f"{ids} carry no unit of that category (carriers present: {sorted(set(live['carrier']))}). Attaching it as " + f"'{carrier}' with the whole plant as the techno-economic donor.", + ) + return live, carrier, "category_fallback" + + +def _remote_unit_attributes( + constituents: pd.DataFrame, + p_nom: float, + carrier: str, + costs: pd.DataFrame, +) -> dict: + """Capacity-weighted techno-economics for one remote contracted unit.""" + fields = [ + "efficiency", + "marginal_cost", + "heat_rate", + "summer_derate", + "winter_derate", + "ramp_limit_up", + "ramp_limit_down", + "min_up_time", + "min_down_time", + "start_up_cost", + "fuel_cost", + ] + attrs = {field: weighted_avg(constituents, field, "p_nom") for field in fields} + + plant_p_nom = float(constituents["p_nom"].sum()) + scale = (p_nom / plant_p_nom) if plant_p_nom > 0 else 0.0 + # start-up cost is an absolute $/start, so it scales with the contracted share + attrs["start_up_cost"] = float(np.nan_to_num(attrs["start_up_cost"])) * scale + + min_load = float(constituents["minimum_load_mw"].fillna(0).sum()) + attrs["min_load_pu"] = (min_load / plant_p_nom) if plant_p_nom > 0 else 0.0 + + for field, fallback in ( + ("efficiency", _cost(costs, carrier, "efficiency", 1.0)), + ("marginal_cost", _cost(costs, carrier, "marginal_cost", 0.0)), + ("heat_rate", _cost(costs, carrier, "heat_rate_mmbtu_per_mwh", 0.0)), + ("summer_derate", 1.0), + ("winter_derate", 1.0), + ("fuel_cost", 0.0), + ): + if pd.isna(attrs[field]): + attrs[field] = fallback + + build_year = weighted_avg(constituents, "build_year", "p_nom") + attrs["build_year"] = int(build_year) if not pd.isna(build_year) else 0 + + storage_mwh = constituents["energy_storage_capacity_mwh"].astype(float).sum() + attrs["duration"] = ( + (storage_mwh / plant_p_nom) if (plant_p_nom > 0 and storage_mwh > 0) else DEFAULT_REMOTE_DURATION_H + ) + return attrs + + +def _remote_vre_profile(n: pypsa.Network, bus: str, carrier: str) -> pd.Series | None: + """Per-unit availability for a remote VRE unit attached at ``bus``. + + Preference order: the attachment bus's own profile for that carrier, then the + mean profile of that carrier across every bus that has one. Returns ``None`` + when the network carries no profile for the carrier at all -- attaching a + resource with an implicit flat 100 % capacity factor would be worse than + dropping it. + """ + profiles = n.generators_t.p_max_pu + same_carrier = n.generators.index[n.generators.carrier == carrier] + available = [gen for gen in same_carrier if gen in profiles.columns] + if not available: + return None + own = f"{bus} {carrier}" + if own in available: + return profiles[own] + logger.info( + f"No {carrier} profile at bus '{bus}'; using the network mean {carrier} profile " + f"({len(available)} buses) for the remote contracted unit(s) attached there.", + ) + return profiles[available].mean(axis=1) + + +def _apply_remote_seasonal_derates(n: pypsa.Network, derates: pd.DataFrame): + """Seasonal (EIA-860) p_max_pu for firm remote units, mirroring apply_seasonal_capacity_derates. + + ``apply_seasonal_capacity_derates`` runs earlier in ``main`` and keys off the + ``plants`` frame, so it cannot see these generators; the same summer/winter + split is reproduced here for exactly the columns added by this module. + """ + if derates.empty: + return + sns_dt = n.snapshots.get_level_values(1) + summer_sns = sns_dt[sns_dt.month.isin([6, 7, 8])] + winter_sns = sns_dt[~sns_dt.month.isin([6, 7, 8])] + + p_max_pu = pd.DataFrame(1.0, index=sns_dt, columns=derates.index) + p_max_pu.loc[summer_sns, derates.index] *= derates["summer_derate"].astype(float) + p_max_pu.loc[winter_sns, derates.index] *= derates["winter_derate"].astype(float) + p_max_pu = broadcast_investment_horizons_index(n, p_max_pu).round(3) + # Round only the new columns: the VRE profiles already in the frame are + # attached at full precision and must not be re-rounded here. + n.generators_t.p_max_pu = pd.concat([n.generators_t.p_max_pu, p_max_pu], axis=1) + + +def attach_remote_contracted_resources( + n: pypsa.Network, + plants_prefilter: pd.DataFrame, + remote_df: pd.DataFrame, + weights: pd.DataFrame, + costs: pd.DataFrame, + conventional_carriers: list, + extendable_carriers: dict, + tech_map: pd.DataFrame | None = None, + unit_commitment: bool = False, +) -> pd.DataFrame: + """Attach California's out-of-state CONTRACTED resources at California buses. + + The CPUC ledger attributes ~10.9 GW of physically out-of-state capacity to + California load regions (Palo Verde's SCE share, LADWP's Intermountain and + Apex, the Hoover entitlements, ~2 GW of AZ/NV solar and battery contracts, + ...). A California-scoped run drops those plants in + :func:`filter_plants_by_region` because they sit outside the model footprint, + which understates the fleet that actually serves California load. This + function adds them back as generators / storage units at California buses, at + their CONTRACTED capacity, with techno-economics taken from the same PUDL + fleet build (``plants_prefilter``, captured before the regional filter). + + Parameters + ---------- + n + Network at the ``elec_s{simpl}_dem`` stage; VRE profiles must already be + attached (this function copies them for remote VRE contracts). + plants_prefilter + ``load_powerplants`` output BEFORE ``filter_plants_by_region``, so the + out-of-state plants are still present. Retirement and vintage filtering + has already been applied, and is respected: a contract whose plant has no + live rows is skipped. + remote_df + ``repo_data/CPUC/servm_out_of_state_units.csv``. + weights + ``build_servm_load_weights`` output — the (bus, servm_region, laf) table + that decides which California bus a region's contracts attach to. + tech_map + ``repo_data/CPUC/servm_tech_map.csv``, used to reconcile the ledger's + ``compare_category`` with model carriers. + + Notes + ----- + Simplifications, all deliberate: + + * ``p_nom`` is ``min(capmax_mw, live plant capacity)`` — a contract can never + exceed what the physical plant has, and the ledger's contracted share is + otherwise taken at face value. + * Remote VRE units borrow the CAPACITY FACTOR PROFILE OF THEIR CALIFORNIA + ATTACHMENT BUS, not of their physical location. Desert-southwest solar is + better-correlated with CAISO's own solar than this implies. + * Remote HYDRO (the Hoover entitlements) is attached as a firm, + energy-unlimited generator carrying only the EIA-860 seasonal derate. CRSP + hydrology, Lake Mead elevation and the monthly energy schedules that + actually bound those entitlements are NOT modelled. + * Ledger rows with no ``eia_plant_id`` (Mexicali TDM/LR2, Powerex/BC, + ESJ Baja wind, a few pure entitlement rows) are skipped: they have no PUDL + techno-economics to inherit and remain in import-machinery territory. + * Contracts are attached NON-EXTENDABLE. They represent existing contracted + capacity, not expansion candidates; ``extendable_carriers`` deliberately + does not make them expandable. + + Returns + ------- + pd.DataFrame + One row per ledger row with its disposition (``status``, ``carrier``, + ``bus``, ``p_nom``), for logging and testing. + """ + region_bus = _servm_region_buses(weights, n) + carrier_to_category, category_to_carrier = _carrier_category_maps(tech_map) + + missing_regions = sorted(set(remote_df["servm_region"]) - set(region_bus.index)) + if missing_regions: + raise ValueError( + f"SERVM regions {missing_regions} appear in the remote contracted-resource file but have no bus in the " + "SERVM load-weights table; cannot attribute their contracts to a California bus.", + ) + + records: list[dict] = [] + units: list[dict] = [] + for _, row in remote_df.iterrows(): + name = str(row["cpuc_unit_name"]) + record = { + "cpuc_unit_name": name, + "servm_region": row["servm_region"], + "compare_category": row["compare_category"], + "capmax_mw": float(row["capmax_mw"]), + "status": "", + "carrier": "", + "bus": "", + "p_nom": 0.0, + } + if not _parse_eia_plant_ids(row.get("eia_plant_id")): + record["status"] = "skipped_no_eia_id" + records.append(record) + continue + + constituents, carrier, note = _select_remote_constituents( + row, + plants_prefilter, + carrier_to_category, + category_to_carrier, + ) + if note == "no_live_units": + logger.info( + f"Remote contracted unit '{name}' ({record['capmax_mw']:.1f} MW) skipped: EIA plant(s) " + f"{_parse_eia_plant_ids(row.get('eia_plant_id'))} have no live generator rows in powerplants.csv " + "(retired, not yet built, or absent from the fleet).", + ) + record["status"] = "skipped_no_live_units" + records.append(record) + continue + + bus = region_bus[row["servm_region"]] + p_nom = float(min(record["capmax_mw"], constituents["p_nom"].sum())) + attrs = _remote_unit_attributes(constituents, p_nom, carrier, costs) + + record.update( + { + "status": "attached" if note == "ok" else "attached_category_fallback", + "carrier": carrier, + "bus": bus, + "p_nom": p_nom, + }, + ) + records.append(record) + units.append({"name": REMOTE_PREFIX + name, "carrier": carrier, "bus": bus, "p_nom": p_nom, **attrs}) + + logger.info( + f"Remote contracted unit '{name}' -> bus '{bus}' ({row['servm_region']} max-LAF bus), carrier " + f"'{carrier}', {p_nom:.1f} MW of a contracted {record['capmax_mw']:.1f} MW.", + ) + + summary = pd.DataFrame(records) + if not units: + logger.warning("Remote contracted resources enabled but no ledger row could be attached.") + return summary + + unit_df = pd.DataFrame(units).set_index("name") + if unit_df.index.has_duplicates: + raise ValueError( + f"Duplicate cpuc_unit_name(s) in the remote contracted-resource file: " + f"{sorted(unit_df.index[unit_df.index.duplicated()])}", + ) + add_missing_carriers(n, sorted(set(unit_df["carrier"]))) + + batteries = unit_df[unit_df["carrier"] == "battery"] + vre = unit_df[unit_df["carrier"].isin(VRE_PROFILE_CARRIERS)] + firm = unit_df.drop(index=batteries.index.union(vre.index)) + + _attach_remote_firm(n, firm, costs, conventional_carriers, unit_commitment) + dropped_vre = _attach_remote_vre(n, vre, costs) + _attach_remote_batteries(n, batteries, costs) + + if dropped_vre: + summary.loc[summary["cpuc_unit_name"].isin(dropped_vre), "status"] = "skipped_no_profile" + summary.loc[summary["cpuc_unit_name"].isin(dropped_vre), "p_nom"] = 0.0 + + attached = summary[summary["status"].str.startswith("attached")] + skipped = summary[~summary["status"].str.startswith("attached")] + logger.info( + f"Remote contracted resources: attached {len(attached)} of {len(summary)} CPUC ledger rows, " + f"{attached['p_nom'].sum():.1f} MW of a contracted {summary['capmax_mw'].sum():.1f} MW.\n" + f"By carrier [MW]:\n{attached.groupby('carrier')['p_nom'].sum().round(1).to_string()}", + ) + if not skipped.empty: + detail = ", ".join(f"{r.cpuc_unit_name} ({r.capmax_mw:.1f} MW, {r.status})" for r in skipped.itertuples()) + logger.warning( + f"Remote contracted resources: {len(skipped)} ledger row(s) totalling " + f"{skipped['capmax_mw'].sum():.1f} MW were NOT added to the model: {detail}.", + ) + return summary + + +def _attach_remote_firm( + n: pypsa.Network, + firm: pd.DataFrame, + costs: pd.DataFrame, + conventional_carriers: list, + unit_commitment: bool, +): + """Attach firm (thermal, nuclear, geothermal, hydro) remote contracts as generators.""" + if firm.empty: + return + + committable = bool(unit_commitment) and bool(firm["carrier"].isin(conventional_carriers).any()) + kwargs = {} + if committable: + commit_flags = firm["carrier"].isin(conventional_carriers) + # p_min_pu is a hard must-run on a non-committable generator, so it is + # only carried for units that are actually committable. The 0.95 factor + # and the seasonal-derate clip mirror attach_conventional_generators. + p_min_pu = ( + firm["min_load_pu"] + .clip(upper=np.minimum(firm["summer_derate"], firm["winter_derate"]), lower=0) + .astype(float) + .fillna(0) + .mul(0.95) + .where(commit_flags, 0.0) + ) + defaults = n.components["Generator"].defaults["default"] + kwargs = { + "committable": commit_flags, + "p_min_pu": p_min_pu, + "start_up_cost": firm["start_up_cost"].astype(float).fillna(defaults["start_up_cost"]), + "min_up_time": firm["min_up_time"].astype(float).fillna(defaults["min_up_time"]), + "min_down_time": firm["min_down_time"].astype(float).fillna(defaults["min_down_time"]), + # greenfield planning model: no warm-start commitment obligation + "up_time_before": 0, + } + + n.add( + "Generator", + firm.index, + carrier=firm["carrier"], + bus=firm["bus"], + p_nom=firm["p_nom"], + p_nom_min=firm["p_nom"], + p_nom_extendable=False, + ramp_limit_up=firm["ramp_limit_up"], + ramp_limit_down=firm["ramp_limit_down"], + efficiency=firm["efficiency"].round(3), + marginal_cost=firm["marginal_cost"], + capital_cost=firm["carrier"].map(lambda c: _cost(costs, c, "annualized_capex_fom", 0.0)), + build_year=firm["build_year"].astype(int), + lifetime=firm["carrier"].map(lambda c: _cost(costs, c, "lifetime", np.inf)), + **kwargs, + ) + n.generators.loc[firm.index, "vom_cost"] = firm["carrier"].map( + lambda c: _cost(costs, c, "opex_variable_per_mwh", 0.0), + ) + n.generators.loc[firm.index, "fuel_cost"] = firm["fuel_cost"] + n.generators.loc[firm.index, "heat_rate"] = firm["heat_rate"] + + _apply_remote_seasonal_derates(n, firm[["summer_derate", "winter_derate"]]) + + +def _attach_remote_vre(n: pypsa.Network, vre: pd.DataFrame, costs: pd.DataFrame) -> list[str]: + """Attach remote wind/solar contracts, borrowing the attachment bus's CF profile. + + Returns the ledger names that had to be dropped for want of any profile. + """ + dropped: list[str] = [] + if vre.empty: + return dropped + + profiles = {} + for name, unit in vre.iterrows(): + profile = _remote_vre_profile(n, unit["bus"], unit["carrier"]) + if profile is None: + logger.error( + f"Remote contracted unit '{name}' (carrier '{unit['carrier']}', {unit['p_nom']:.1f} MW) dropped: the " + "network carries no capacity-factor profile for that carrier, so no availability can be assigned.", + ) + dropped.append(name[len(REMOTE_PREFIX) :]) + continue + profiles[name] = profile + + vre = vre.loc[list(profiles)] + if vre.empty: + return dropped + + p_max_pu = pd.DataFrame(profiles).reindex(columns=vre.index) + n.add( + "Generator", + vre.index, + carrier=vre["carrier"], + bus=vre["bus"], + p_nom=vre["p_nom"], + p_nom_min=vre["p_nom"], + p_nom_extendable=False, + efficiency=1.0, + marginal_cost=vre["carrier"].map(lambda c: _cost(costs, c, "marginal_cost", 0.0)), + capital_cost=vre["carrier"].map(lambda c: _cost(costs, c, "annualized_capex_fom", 0.0)), + build_year=vre["build_year"].astype(int), + lifetime=vre["carrier"].map(lambda c: _cost(costs, c, "lifetime", np.inf)), + p_max_pu=p_max_pu, + ) + return dropped + + +def _attach_remote_batteries(n: pypsa.Network, batteries: pd.DataFrame, costs: pd.DataFrame): + """Attach remote battery contracts as non-extendable storage units.""" + if batteries.empty: + return + efficiency = 0.85**0.5 + n.add( + "StorageUnit", + batteries.index, + carrier="battery", + bus=batteries["bus"], + p_nom=batteries["p_nom"], + p_nom_max=batteries["p_nom"], + p_nom_min=0, + p_nom_extendable=False, + capital_cost=_cost(costs, "4hr_battery_storage", "opex_fixed_per_kw", 0.0) * 1e3, + # attach_battery_storage divides the reported duration by the dispatch + # efficiency so the DELIVERABLE energy matches the reported MWh; mirror it + # so remote and in-state batteries are sized on the same convention. + max_hours=batteries["duration"] / efficiency, + build_year=batteries["build_year"].astype(int), + lifetime=_cost(costs, "4hr_battery_storage", "lifetime", np.inf), + efficiency_store=efficiency, + efficiency_dispatch=efficiency, + cyclic_state_of_charge=True, + cyclic_state_of_charge_per_period=True, # pypsa v1 flipped this default to False + ) + + def broadcast_investment_horizons_index(n: pypsa.Network, df: pd.DataFrame): """ Broadcast the index of a dataframe to match the potentially multi-indexed @@ -988,6 +1567,22 @@ def broadcast_investment_horizons_index(n: pypsa.Network, df: pd.DataFrame): return df +def check_ambient_derate_not_enabled(conventional: dict) -> None: + """Guard the phase-2 ``conventional.ambient_derate`` hook. + + Ambient-temperature derates are meant to REPLACE the EIA-860 seasonal + derate applied by :func:`apply_seasonal_capacity_derates`, never to stack + on top of it (or on top of a UCAP-derated capacity credit) — stacking + double-counts the same thermal deficiency. Until the CPUC SERVM + unit-specific hourly derate profiles are retrieved and applied, enabling + the key is a hard error rather than a silent no-op. + """ + if (conventional or {}).get("ambient_derate", {}).get("enable", False): + raise NotImplementedError( + "conventional.ambient_derate is a phase-2 hook; CPUC unit-specific hourly derate profiles are not yet retrieved or applied.", + ) + + def apply_seasonal_capacity_derates( n: pypsa.Network, plants: pd.DataFrame, @@ -1222,7 +1817,21 @@ def main(snakemake): snakemake.input["powerplants"], n.investment_periods, interconnect=interconnection, + honor_planned_retirements=snakemake.config["electricity"].get("honor_planned_retirements", True), ) + + # California's out-of-state CONTRACTED resources are physically outside a + # CA-scoped footprint, so filter_plants_by_region below drops them. Stash the + # (small) slice of the fleet the CPUC ledger points at BEFORE that filter runs; + # attach_remote_contracted_resources adds them back at CA buses later. + remote_contracted = dict(getattr(params, "remote_contracted", None) or {}) + remote_df = None + plants_prefilter = None + if remote_contracted.get("enable", False): + remote_df = pd.read_csv(snakemake.input["remote_resources"]) + remote_ids = {pid for value in remote_df["eia_plant_id"] for pid in _parse_eia_plant_ids(value)} + plants_prefilter = plants[plants["plant_id_eia"].isin(remote_ids)].copy() + # A run scoped with model_topology.include tiles regions over the footprint only; # the seam-plant fallback in filter_plants_by_region must then be distance-bounded. include_filter = snakemake.config.get("model_topology", {}).get("include") or {} @@ -1255,6 +1864,7 @@ def main(snakemake): renewable_carriers, unit_commitment=params.conventional["unit_commitment"], ) + check_ambient_derate_not_enabled(params.conventional) apply_seasonal_capacity_derates( n, plants, @@ -1311,6 +1921,22 @@ def main(snakemake): ["hydro"], ) + # Runs last among the attach_* steps: remote VRE contracts copy the + # capacity-factor profile of their California attachment bus, which only + # exists once attach_wind_and_solar has run. + if remote_df is not None: + attach_remote_contracted_resources( + n, + plants_prefilter, + remote_df, + pd.read_csv(snakemake.input["servm_load_weights"]), + costs, + conventional_carriers, + extendable_carriers, + tech_map=pd.read_csv(snakemake.input["servm_tech_map"]), + unit_commitment=params.conventional["unit_commitment"], + ) + update_p_nom_max(n) # apply regional multipliers to capital cost data diff --git a/workflow/scripts/add_extra_components.py b/workflow/scripts/add_extra_components.py index 3d5cf424..061ddc93 100644 --- a/workflow/scripts/add_extra_components.py +++ b/workflow/scripts/add_extra_components.py @@ -994,8 +994,11 @@ def _add_import_export_links( zone_inside = row.r if row.r in zones_in_model else row.rr zone_outside = row.r if row.r not in zones_in_model else row.rr - # extremely crude cashing for generating cost timeseries :| - if zone_outside not in costs: + # extremely crude caching for generating cost timeseries :| + # keyed by the INSIDE zone — checking the outside zone here skipped + # the write whenever an earlier row's inside zone happened to match, + # leaving costs[zone_inside] unset (KeyError on county networks). + if zone_inside not in costs: if isinstance(fuel_costs, float | int): costs[zone_inside] = fuel_costs elif isinstance(fuel_costs, pd.DataFrame): @@ -1325,6 +1328,18 @@ def apply_ucap(n: pypsa.Network, ucap_config: dict) -> None: # Apply to static p_max_pu n.generators.loc[mask, "p_max_pu"] *= ucap_factor + # Committable units carry a minimum stable level (p_min_pu) that must stay + # below the derated p_max_pu, otherwise the unit's dispatch box is empty and + # the MILP is infeasible. UCAP shrinks the whole usable range of the unit, so + # scale p_min_pu by the same factor rather than leaving it at the ICAP level. + if "committable" in n.generators: + com_mask = mask & n.generators.committable.fillna(False).astype(bool) + if com_mask.any(): + n.generators.loc[com_mask, "p_min_pu"] *= ucap_factor + com_varying = [g for g in n.generators.index[com_mask] if g in n.generators_t.p_min_pu.columns] + if com_varying: + n.generators_t.p_min_pu[com_varying] *= ucap_factor + # Apply to time-varying p_max_pu if present gens_with_time_varying = [g for g in gen_names if g in n.generators_t.p_max_pu.columns] if gens_with_time_varying: @@ -1630,7 +1645,7 @@ def main(snakemake) -> None: f"'exports.costs' must be 'wholesale', name of a carrier, or a float/int. Received: {export_costs}", ) - add_elec_imports_exports(n, "exports", export_flowgates, fuel_costs, co2_emissions) + add_elec_imports_exports(n, "exports", export_flowgates, fuel_costs, co2_emissions, zone_col) if snakemake.config["scenario"]["sector"] == "E": co2_storage = snakemake.config.get("co2", {}).get("storage", False) diff --git a/workflow/scripts/benchmark_cpuc_baseline.py b/workflow/scripts/benchmark_cpuc_baseline.py new file mode 100644 index 00000000..06f923e7 --- /dev/null +++ b/workflow/scripts/benchmark_cpuc_baseline.py @@ -0,0 +1,803 @@ +# BY PyPSA-USA Authors +"""Benchmark the modelled California fleet against the CPUC Baseline Generator List. + +The CPUC publishes, alongside the SERVM hourly load, a unit-level baseline +generator list covering the whole WECC. Its California rows are the reference +fleet the CPUC's own SERVM/RESOLVE runs start from, so comparing installed +capacity against PyPSA-USA's fleet is a direct check on whether the model starts +a California study from a plausible resource mix. + +Two axes have to be reconciled before the sides are comparable. + +Region +------ +The CPUC tags every unit with a ``SERVM Region`` (PGE, SCE, SDGE, IID, LADWP, +NCNC). PyPSA-USA's ``powerplants.csv`` carries ``balancing_authority_code_eia``, +which is the *EIA* balancing-authority code — and EIA reports every CAISO plant +under the single code ``CISO`` with no sub-BA breakdown. There is therefore no +column on the model side that separates PGE from SCE from SDGE, and inventing +one (point-in-polygon against the clustered BA shapes) would need a +``resources/`` artifact that this rule does not consume. + +So the benchmark is run at the coarsest region resolution both sides support: + +* ``CAISO`` — CPUC PGE + SCE + SDGE vs. model BA ``CISO`` (the CPUC data + dictionary defines CAISO as exactly PGE + SCE + SDGE) +* ``LADWP`` — CPUC LADWP vs. model BA ``LDWP`` +* ``IID`` — CPUC IID vs. model BA ``IID`` +* ``NCNC`` — CPUC NCNC vs. model BAs ``BANC`` + ``TIDC`` + +The mapping lives in ``repo_data/CPUC/servm_benchmark_regions.csv`` so the +collapse is explicit and editable rather than hard-coded here. The model side is +additionally restricted to ``state == "CA"``: EIA's ``CISO`` also covers the +Valley Electric (CISO-VEA) footprint, which sits in Nevada and is excluded from +the SERVM California regions. + +Technology +---------- +SERVM splits technology far finer than PyPSA carriers do (nine solar buckets, a +paired/hybrid storage split, a CHP bucket). Both sides are therefore mapped into +a common ``compare_category`` by ``repo_data/CPUC/servm_tech_map.csv``, which +carries a ``side`` column (``cpuc`` / ``pypsa``) so one file documents both +directions. Anything unmapped on either side becomes ``UNMAPPED:`` and is +reported as its own row — a category is never silently dropped, because a silent +drop is exactly how a benchmark starts agreeing with itself. + +Two mappings deserve to be flagged when reading the output: + +* ``Gas Cogen/CHP`` is CPUC-only. EIA technology descriptions have no CHP + concept, so California gas cogeneration lands in ``CCGT``/``OCGT`` by prime + mover. Expect the model to be short in ``Gas Cogen/CHP`` and long in the two + gas buckets by roughly the same amount. +* ``Demand Response`` and ``Pumping Load`` are CPUC-only resources with no + PyPSA-USA counterpart; they are kept as rows with ``model_mw == 0``. + +Vintage and retirement +---------------------- +Both sides are filtered by the same rule (:func:`filter_active`): in service by +December 31 of the horizon, and not retired by then. The model side reproduces +``add_electricity.load_powerplants`` exactly (see +:func:`prepare_model_plants`) so this benchmark measures the fleet the model +actually builds, not a differently-filtered idealisation of it. +""" + +import logging + +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd +import pypsa +import seaborn as sns +from _helpers import configure_logging + +logger = logging.getLogger(__name__) + +sns.set_theme("paper", style="whitegrid") + +DPI = 300 + +CPUC_SHEET = "BaselineGeneratorList" +#: The workbook's first row is blank; the real header is the second one. +CPUC_HEADER_ROW = 1 + +#: SERVM regions that make up California. Every other ``SERVM Region`` value in +#: the workbook is a non-CA WECC balancing area. +CA_SERVM_REGIONS = ("IID", "LADWP", "NCNC", "PGE", "SCE", "SDGE") + +CPUC_CAPACITY_COL = "Capmax MW" +CPUC_REGION_COL = "SERVM Region" +CPUC_TECH_COL = "SERVM Tech Category" +CPUC_INSV_COL = "Insvdt" +CPUC_RETIRE_COL = "RetireDate" + +#: Sentinel used by :func:`filter_active` for "no retirement date on record". +NEVER_RETIRES = pd.Timestamp("2262-01-01") + +UNMAPPED_PREFIX = "UNMAPPED:" + +#: Pseudo-region prefix for :func:`reconcile_totals`' informational rows. +RECONCILE_PREFIX = "RECONCILE:" + +#: Pseudo-region for CPUC units whose physical resource sits outside California +#: (AZ/NV/UT solar, batteries and CCs, Hoover shares, Palo Verde, Mexicali CCs, +#: Powerex dynamic hydro, Baja wind). CPUC's SERVM regions are contractual +#: ledgers; a physically-located model can never carry these, so they are +#: reported on this row instead of being scored as model shortfall. +CPUC_UNIT_COL = "Unit Name" + + +# --------------------------------------------------------------------------- # +# inputs +# --------------------------------------------------------------------------- # + + +def read_cpuc_baseline(xlsx_path: str) -> pd.DataFrame: + """ + Read the California rows of the CPUC Baseline Generator List. + + The workbook opens with a blank row, so the header is row 2 (``header=1``). + The published sheet covers all of WECC; only units whose ``SERVM Region`` is + one of :data:`CA_SERVM_REGIONS` are kept. + + ``Capmax MW`` is the nameplate capacity of the unit as SERVM sees it. Paired + and hybrid resources are published as separate rows per component (a + ``Paired_Solar_1Axis`` row and a ``Paired_BattStorage`` row), so summing + ``Capmax MW`` by technology counts each component once and never + double-counts the point of interconnection. + """ + df = pd.read_excel(xlsx_path, sheet_name=CPUC_SHEET, header=CPUC_HEADER_ROW) + + missing = [ + c + for c in (CPUC_CAPACITY_COL, CPUC_REGION_COL, CPUC_TECH_COL, CPUC_INSV_COL, CPUC_RETIRE_COL) + if c not in df.columns + ] + if missing: + raise ValueError( + f"{xlsx_path} sheet '{CPUC_SHEET}' is missing {missing}. The CPUC has changed the " + "workbook layout; update benchmark_cpuc_baseline.py's column constants.", + ) + + df = df.copy() + df[CPUC_REGION_COL] = df[CPUC_REGION_COL].astype("object").where(df[CPUC_REGION_COL].notna()).str.strip() + df[CPUC_TECH_COL] = df[CPUC_TECH_COL].astype("object").where(df[CPUC_TECH_COL].notna()).str.strip() + df[CPUC_CAPACITY_COL] = pd.to_numeric(df[CPUC_CAPACITY_COL], errors="coerce") + for col in (CPUC_INSV_COL, CPUC_RETIRE_COL): + df[col] = pd.to_datetime(df[col], errors="coerce") + + ca = df[df[CPUC_REGION_COL].isin(CA_SERVM_REGIONS)].copy() + if ca.empty: + raise ValueError( + f"No rows in {xlsx_path} carry a California SERVM Region ({', '.join(CA_SERVM_REGIONS)}). " + "Either the wrong sheet was read or the region labels have changed.", + ) + + logger.info( + "Read %d CPUC baseline units, %d of them in California across %d SERVM regions.", + len(df), + len(ca), + ca[CPUC_REGION_COL].nunique(), + ) + return ca + + +def load_tech_map(path: str) -> dict[str, dict[str, str]]: + """ + ``{side: {source_category: compare_category}}`` read from ``servm_tech_map.csv``. + + One file holds both directions, keyed by a ``side`` column of ``cpuc`` or + ``pypsa``, so the two halves of a comparison category can never drift apart + across separate files. + """ + df = pd.read_csv(path, dtype=str) + for col in ("side", "source_category", "compare_category"): + if col not in df.columns: + raise ValueError( + f"{path} must have columns side, source_category, compare_category; got {list(df.columns)}", + ) + df[col] = df[col].str.strip() + + unknown_sides = sorted(set(df.side.unique()) - {"cpuc", "pypsa"}) + if unknown_sides: + raise ValueError(f"{path} has unknown side values {unknown_sides}; expected 'cpuc' or 'pypsa'.") + + out: dict[str, dict[str, str]] = {} + for side, grp in df.groupby("side"): + duplicated = grp.source_category[grp.source_category.duplicated()].unique() + if len(duplicated): + raise ValueError(f"{path} maps {list(duplicated)} more than once on side '{side}'.") + out[side] = dict(zip(grp.source_category, grp.compare_category)) + + for side in ("cpuc", "pypsa"): + out.setdefault(side, {}) + return out + + +def load_benchmark_regions(path: str) -> tuple[pd.Series, pd.Series]: + """ + Return ``(servm_region -> benchmark_region, eia_ba_code -> benchmark_region)``. + + Both lookups come from the same file so the CAISO collapse (PGE/SCE/SDGE and + EIA ``CISO`` landing on one ``CAISO`` row) is stated once. + """ + df = pd.read_csv(path, dtype=str) + for col in ("servm_region", "eia_ba_code", "benchmark_region"): + if col not in df.columns: + raise ValueError( + f"{path} must have columns servm_region, eia_ba_code, benchmark_region; got {list(df.columns)}", + ) + df[col] = df[col].str.strip() + + servm = df.drop_duplicates("servm_region").set_index("servm_region").benchmark_region + dupes = servm.index[servm.index.duplicated()].unique() + if len(dupes): + raise ValueError(f"{path} maps SERVM regions {list(dupes)} to more than one benchmark region.") + + ba = df.drop_duplicates("eia_ba_code") + conflicting = ba.groupby("eia_ba_code").benchmark_region.nunique() + conflicting = conflicting[conflicting > 1] + if len(conflicting): + raise ValueError(f"{path} maps EIA BA codes {list(conflicting.index)} to more than one benchmark region.") + + return servm, ba.set_index("eia_ba_code").benchmark_region + + +# --------------------------------------------------------------------------- # +# shared filtering / mapping +# --------------------------------------------------------------------------- # + + +def filter_active( + df: pd.DataFrame, + horizon: int, + insv_col: str, + retire_col: str, +) -> pd.DataFrame: + """ + Units in service by the end of ``horizon`` and not yet retired. + + A unit counts if it enters service on or before December 31 of the horizon + year and its retirement year is strictly greater than the horizon — the same + inequality ``add_electricity.load_powerplants`` applies to the model fleet, + so the two sides of the benchmark are filtered identically. + + A missing retirement date means "never retires" (the workbook leaves the + field empty for units with no announced retirement; it also uses a + ``2050-12-01`` placeholder for many, which needs no special handling since a + 2050 retirement genuinely keeps the unit alive through any earlier horizon). + A missing in-service date is treated as already in service: the workbook is a + list of the *existing* baseline fleet, so an absent date is a data gap rather + than a future project. + """ + insv = pd.to_datetime(df[insv_col], errors="coerce") + retire = pd.to_datetime(df[retire_col], errors="coerce").fillna(NEVER_RETIRES) + + n_missing_insv = int(insv.isna().sum()) + if n_missing_insv: + logger.warning( + "%d of %d units have no in-service date; treating them as already in service.", + n_missing_insv, + len(df), + ) + + in_service = insv.isna() | (insv <= pd.Timestamp(year=horizon, month=12, day=31)) + not_retired = retire.dt.year > horizon + return df[in_service & not_retired] + + +def map_compare_category(values: pd.Series, mapping: dict[str, str]) -> pd.Series: + """ + Map source technology labels onto comparison categories. + + Anything absent from ``mapping`` becomes ``UNMAPPED: