Skip to content

Slurm resource requests are v0-stale and heavily over-provisioned for the v1 pipeline #808

Description

@ktehranchi

Problem

The per-rule walltime: block in config.slurm.yaml and the mem_mb expressions in workflow/rules/*.smk date from the v0 pipeline. On the v1 (simplify-early) stack they over-request by one to two orders of magnitude for many rules, which hurts queue priority and wastes allocation on shared clusters.

Evidence

Measured on a California-scoped run (CPUC SERVM config, single weather year, both 4-zone and county resolutions, 33 benchmarked jobs) with benchmark: directives added to every rule. Peak RSS and runtime vs. current requests, selected rules:

Rule Peak RSS Peak time Currently requests
build_renewable_profiles 2.9 GB 27 s NPROC*5 GB / 4 h
build_powerplants 2.0 GB 24 s 30 GB
build_fuel_prices 1.9 GB 16 s 30 GB / 20 min
add_electricity 0.7 GB 24 s 4 h
cluster_network 0.5 GB 34 s 4 h
cluster_resources 0.3 GB 2 min 5 h
aggregate_to_substations 0.3 GB 12 s 2 h

Total serial rule time for a full CA build to add_extra_components: under 8 minutes, against day-scale summed walltime requests.

Caveats

  • These numbers are from a CA-only network (~1450 generators); western/eastern/usa interconnects scale roughly with bus and generator count and need their own measurements before defaults change.
  • Memory overrun is a hard kill while excess walltime only costs priority, so recommended requests should keep asymmetric headroom (e.g. 1.5x on peak RSS, 3x on peak runtime).

Proposal

  1. Keep benchmark: directives on every rule (branch feat/benchmark-directives adds the missing ones) so every run produces measurement data.
  2. Replace flat v0 walltimes/mem with measurement-derived defaults, ideally scaled by an input-size proxy (input.size) the way some rules already do — but re-fit the coefficients against v1 benchmark TSVs rather than v0 folklore.
  3. Add a small report script that aggregates benchmarks/ TSVs into recommended mem_mb/walltime per rule, so per-site tuning is a one-command workflow.

Happy to contribute the CA measurement data and the aggregation script.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions