Problem
The per-rule walltime: block in config.slurm.yaml and the mem_mb expressions in workflow/rules/*.smk date from the v0 pipeline. On the v1 (simplify-early) stack they over-request by one to two orders of magnitude for many rules, which hurts queue priority and wastes allocation on shared clusters.
Evidence
Measured on a California-scoped run (CPUC SERVM config, single weather year, both 4-zone and county resolutions, 33 benchmarked jobs) with benchmark: directives added to every rule. Peak RSS and runtime vs. current requests, selected rules:
| Rule |
Peak RSS |
Peak time |
Currently requests |
| build_renewable_profiles |
2.9 GB |
27 s |
NPROC*5 GB / 4 h |
| build_powerplants |
2.0 GB |
24 s |
30 GB |
| build_fuel_prices |
1.9 GB |
16 s |
30 GB / 20 min |
| add_electricity |
0.7 GB |
24 s |
4 h |
| cluster_network |
0.5 GB |
34 s |
4 h |
| cluster_resources |
0.3 GB |
2 min |
5 h |
| aggregate_to_substations |
0.3 GB |
12 s |
2 h |
Total serial rule time for a full CA build to add_extra_components: under 8 minutes, against day-scale summed walltime requests.
Caveats
- These numbers are from a CA-only network (~1450 generators); western/eastern/usa interconnects scale roughly with bus and generator count and need their own measurements before defaults change.
- Memory overrun is a hard kill while excess walltime only costs priority, so recommended requests should keep asymmetric headroom (e.g. 1.5x on peak RSS, 3x on peak runtime).
Proposal
- Keep
benchmark: directives on every rule (branch feat/benchmark-directives adds the missing ones) so every run produces measurement data.
- Replace flat v0 walltimes/mem with measurement-derived defaults, ideally scaled by an input-size proxy (
input.size) the way some rules already do — but re-fit the coefficients against v1 benchmark TSVs rather than v0 folklore.
- Add a small report script that aggregates
benchmarks/ TSVs into recommended mem_mb/walltime per rule, so per-site tuning is a one-command workflow.
Happy to contribute the CA measurement data and the aggregation script.
Problem
The per-rule
walltime:block inconfig.slurm.yamland themem_mbexpressions inworkflow/rules/*.smkdate from the v0 pipeline. On the v1 (simplify-early) stack they over-request by one to two orders of magnitude for many rules, which hurts queue priority and wastes allocation on shared clusters.Evidence
Measured on a California-scoped run (CPUC SERVM config, single weather year, both 4-zone and county resolutions, 33 benchmarked jobs) with
benchmark:directives added to every rule. Peak RSS and runtime vs. current requests, selected rules:Total serial rule time for a full CA build to
add_extra_components: under 8 minutes, against day-scale summed walltime requests.Caveats
Proposal
benchmark:directives on every rule (branchfeat/benchmark-directivesadds the missing ones) so every run produces measurement data.input.size) the way some rules already do — but re-fit the coefficients against v1 benchmark TSVs rather than v0 folklore.benchmarks/TSVs into recommendedmem_mb/walltimeper rule, so per-site tuning is a one-command workflow.Happy to contribute the CA measurement data and the aggregation script.