Skip to content

fix: make calibration staleness detection content-based - #37

Merged
koen-vg merged 1 commit into
mainfrom
fix-calibration-staleness-check
Jul 28, 2026
Merged

koen-vg merged 1 commit into
mainfrom
fix-calibration-staleness-check

Conversation

@koen-vg

@koen-vg koen-vg commented Jul 28, 2026

Copy link
Copy Markdown
Member

Problem

tools/calibrate --check asked Snakemake whether a dry run would do anything,
which is an mtime question. The five calibration steps write into one directory
that they all read from, so re-running any step marked its successors stale even
when the artefacts came back byte-identical, and a git checkout marked
everything stale. It never converged, and the guidance in the docs had degraded
to "treat [STALE] as worth investigating, don't chase it to all-green".

It also had a false negative: calibrate_deviation_penalty runs its solves
in-process, so it never declared the calibration artefacts those solves read.

Approach

A step is stale when re-running it would change its artefacts, which depends
only on inputs from outside the artefact set. Each set now carries a
fingerprint.yaml hashing exactly those:

  • every external leaf of the step's Snakemake DAG (curated, bundled and
    manually downloaded data, the build_model package),
  • the rule files and scripts that DAG runs,
  • the step config, hashed after a YAML round-trip so comment and formatting
    edits do not register,
  • the artefacts the step wrote, so one edited or checked out behind the chain's
    back is reported rather than silently trusted.

The DAG is rebuilt on each check through the Snakemake Python API, so a
dependency that a rule or code change adds or drops is picked up straight away.
Because the fingerprint ignores the artefact set's own contents, re-running one
step never marks its successors stale.

New tools/calibrate --record re-stamps a set without solving, for code changes
that provably cannot move the artefacts.

Fixes that fell out of building the DAG honestly

  • Solve-time calibration artefacts (feed corrections, exogenous feed and forage,
    food-demand multipliers, the calibrated deviation penalty) were declared as
    inputs of solve_model but not of calibrate_deviation_penalty. Both now
    share calibration_artefact_inputs.

  • health_required built a full deepcopy of the configuration for every
    configured scenario. On gsa.yaml (16384 generated samples) that was ~35 s of
    the ~43 s spent constructing any DAG -- so it slowed down every Snakemake
    invocation against a generated ensemble, not just the check. Reading the
    overrides directly brings Snakefile parsing to ~4 s at an identical 848-job
    DAG.

Verification

Timings on this workstation, gbd-anchored set: --check 3m15s -> 32 s;
default set ~19 s.

Behaviour matrix, all confirmed correct:

change expected observed
touch every artefact + a curated input all up-to-date all up-to-date
hand-edit a feed artefact all 5 stale feed "artefact modified since it was generated", rest "input content changed"
add a file inside a manually-downloaded IHME directory all 5 stale "input content changed"
edit a workflow script or rule file stale stale
cost.yaml slack_marginal_cost 7.5 -> 7.6 only cost only cost
comment-only config edit up-to-date up-to-date
full workflow run that reclaims temp() downloads up-to-date up-to-date

pixi run -e dev pytest 715 passed; test-integration passes; ruff and the
REUSE hook clean. New tests/test_calibration_fingerprint.py covers the
comparison logic and the YAML/directory hashing.

`tools/calibrate --check` asked Snakemake whether a dry run would do
anything, which is an mtime question. The five steps write into one
directory that they all read from, so re-running any step marked its
successors stale even when the artefacts came back byte-identical, and a
`git checkout` marked everything stale. It never converged.

A step is stale when re-running it would change its artefacts, which
depends only on inputs from outside the artefact set. Each set now carries
a `fingerprint.yaml` hashing exactly those: the external leaves of the
step's Snakemake DAG, the rule files and scripts that DAG runs, the step
config (YAML round-tripped, so comments do not count), and the artefacts
the step wrote. The DAG is rebuilt on each check, so dependencies a rule
change adds or drops are picked up immediately.

`--record` re-stamps a set without solving, for code changes that provably
cannot move the artefacts.

Two correctness fixes fall out of building the DAG honestly:

- The solve-time calibration artefacts were declared as inputs of
  `solve_model` but not of `calibrate_deviation_penalty`, whose Broyden
  iteration runs the same solve in-process. Both now share
  `calibration_artefact_inputs`.

- `health_required` built a full deepcopy of the configuration for every
  configured scenario. On `gsa.yaml` (16384 generated samples) that was
  ~35 s of the ~43 s spent constructing any DAG; it is now ~4 s, which
  brings `--check` on the `gbd-anchored` set from 3m15s down to 32 s.
@koen-vg
koen-vg merged commit bb8e5ab into main Jul 28, 2026
4 checks passed
@koen-vg
koen-vg deleted the fix-calibration-staleness-check branch July 28, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant