Benchmark directives on all rules + Slurm resource measurement tooling - #809
Merged
Merged
Conversation
Ten rules in build_electricity.smk and add_sectors in build_sector.smk had no benchmark: directive, so snakemake recorded no wall time or max RSS for them. That included cluster_resources (the largest walltime budget in the config) and add_extra_components (the terminal rule for network-only builds), which are exactly the rules needed to tune Slurm resource requests. Benchmark paths mirror each rule's log path under BENCHMARKS, and carry every wildcard the rule takes so concurrent jobs of the same rule cannot collide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_slurm.sh took a single hard-coded configfile and passed -A {cluster.account},
which Sherlock rejects (it has no accounts). It now takes a list of overlay
configs, runs each as its own `snakemake --cluster` invocation with bounded
concurrency, and reads each build target from config/weather_years/manifest.tsv.
slurm_submit.sh wraps sbatch because several rules compute mem_mb as a float
(e.g. (input.size // 150000) * attempt * 1.5) and `sbatch --mem 4500.0` is
rejected; it rounds up and floors at 2000 MB.
Note: the target must precede --configfile, which is nargs='+' and would
otherwise swallow it as a third config file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The uv-managed CPython ships no CA bundle -- ssl.get_default_verify_paths() returns cafile=None -- so retrieve_cpuc_servm_load died with CERTIFICATE_VERIFY_FAILED. A curl reachability test does not catch this, because curl reads the system trust store that Python never consults. run_slurm.sh now derives SSL_CERT_FILE/REQUESTS_CA_BUNDLE from certifi and relies on sbatch --export=ALL to propagate them to every rule job. Also adds probe_2019.sbatch, the driver job for the resource-tuning probe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sacct splits what we need across two rows: the allocation row has JobName and ReqMem, the .batch row has MaxRSS. `sacct -X` shows MaxRSS blank, which is a quiet trap. This joins them on the base job id and aggregates per rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…dations Complements collect_benchmarks.sh: reads snakemake benchmark TSVs, resolves rule names against rules/*.smk, and prints recommended mem_mb (1.5x peak RSS) and walltime (3x peak runtime) per rule. Offered upstream in issue #808. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #808 (Slurm resource requests are v0-stale).
What this adds
benchmark:directives on the 11 rules that were missing them, so every run emits per-job TSVs (runtime, max RSS/PSS, I/O, CPU load) underbenchmarks/. This is the measurement foundation the issue asks for.workflow/report_benchmarks.py— aggregates benchmark TSVs into per-rule recommendations:mem_mb= 1.5x peak RSS (rounded to 500 MB),walltime= 3x peak runtime (rounded to 5 min). Asymmetric headroom because a memory overrun is a hard kill while excess walltime only costs queue priority. Rule names are resolved againstrules/*.smksince benchmark paths are not uniform.workflow/collect_benchmarks.sh— joinssacctallocation rows with.batchsteps to recover MaxRSS per rule-job for runs submitted via--cluster.workflow/run_slurm.shrewrite): one snakemake driver per scenario overlay, year-grouped serialization to avoid output-file lock collisions on shared artifacts,--rerun-triggers mtimeso mid-campaign code edits don't stampede shared retrieve rules, and a certifi fix for HTTPS retrieve rules under uv-managed Python.p_min_pu,p_nom_extendable,up_time_before).Measurements motivating this (CA-scoped run, 33 jobs)
Selected current-request vs. measured peaks — full table in #808:
Note these are CA-scale numbers; re-fitting the default
mem_mb/walltimeexpressions for the full interconnects should be driven by benchmark data collected with this tooling, and is left for follow-up commits on this PR.More contributions to this PR are incoming from parallel work.
🤖 Generated with Claude Code