Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion config/calibration/stability.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# one paired solve (baseline with `enforce_baseline_diet=true` to derive
# consumer values, then main with piecewise utility active).
#
# Warm-starts from the previously calibrated YAML when one exists on disk;
# Warm-starts from an existing calibrated YAML when one exists on disk;
# otherwise uses the seed values in `deviation_penalty.calibration.seeds`.
#
# Usage:
Expand Down
12 changes: 6 additions & 6 deletions config/default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ water:
eta_max: 5.0
# Consumed fraction C/W (consumption over withdrawal). Used to report
# withdrawal in the analysis and to convert the current_use availability
# pool from withdrawal to consumption basis. Global constant for now.
# pool from withdrawal to consumption basis. Global constant.
consumed_fraction: 0.58

# --- section: water_scarcity ---
Expand Down Expand Up @@ -968,10 +968,10 @@ diet:
# residual fix whenever whole_grains is anchored -- see docs/current_diets.rst.
# CALIBRATION COUPLING: this flag changes the baseline diet, which the
# calibration artefacts under data/curated/calibration/ are fit against.
# Changing it (directly or via health.enabled) means the `default` artefact
# set no longer matches; point calibration.source at the `gbd-anchored` set
# (fit with anchoring on) or recalibrate. The provenance check errors on a
# mismatch (see docs/calibration.rst).
# The calibration artefact set must match this choice. Point
# calibration.source at the `gbd-anchored` set when anchoring is enabled,
# or recalibrate. The provenance check errors on a mismatch (see
# docs/calibration.rst).
anchor_groups_to_gbd: match_health
# Foods whose per-country intake is computed directly from FAOSTAT Food
# Balance Sheet supply rather than disaggregated from GDD/GBD group
Expand Down Expand Up @@ -1424,7 +1424,7 @@ health:
risk_cause_map:
fruits: [CHD, Stroke, T2DM]
vegetables: [CHD, Stroke]
nuts_seeds: [CHD] # GBD 2023 BoP no longer links nuts/seeds to T2DM
nuts_seeds: [CHD]
legumes: [CHD]
red_meat: [CHD, Stroke, T2DM, CRC]
whole_grains: [CHD, Stroke, T2DM, CRC]
Expand Down
2 changes: 1 addition & 1 deletion docs/calibration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ Each iteration is one paired solve (baseline with
``enforce_baseline_diet=true`` to derive consumer values, then main
with piecewise utility active). Convergence is typically reached in
3–5 iterations from a cold start and 1–2 from a warm start (the
previously calibrated YAML is auto-detected and used as the seed). The
existing calibrated YAML is auto-detected and used as the seed). The
initial Jacobian is :math:`\mathrm{diag}(-1, -1)`, which is the exact
log-log slope for a relationship of the form
:math:`\text{dev} \propto 1/\ell_1`.
Expand Down
6 changes: 3 additions & 3 deletions docs/crop_production.rst
Original file line number Diff line number Diff line change
Expand Up @@ -509,9 +509,9 @@ numeric codes as:
* 7 – triple cropping (≤2 wetland rice crops)
* 8 – triple rice cropping (up to three wetland rice crops)

Relay cropping opportunities mentioned for the C/F zones are intentionally ignored for now; we
only construct sequential crop chains. This assumption is called out in the configuration and
model framework documentation so users know the limitation.
Relay cropping opportunities mentioned for the C/F zones are not modeled; we only construct
sequential crop chains. This limitation is also noted in the configuration and model framework
documentation.

During ``build_model`` each (combination, region, resource class) creates a single
rainfed or irrigated multi-output link (carrier ``crop_production_multi``) that:
Expand Down
6 changes: 3 additions & 3 deletions docs/current_diets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -705,9 +705,9 @@ multipliers, so the diet mass-balances against the food bus.
total for red meat sat ~24 Mt/yr above what total world supply
(production net of feed/non-food/exports, after post-loss and
consumer waste) can deliver — physically impossible — and
previously inflated the calibrated ``feed`` L1 cost ninefold
because the deviation-penalty calibration was forced to fight
intake-derived consumer values that were structurally above supply.
inflates the calibrated ``feed`` L1 cost because the deviation-penalty
calibration must otherwise accommodate intake-derived consumer values
that exceed supply.

2. **Trade is handled implicitly.** FBS supply per country already
encodes ``production + imports − exports − feed − seed − non-food
Expand Down
4 changes: 2 additions & 2 deletions docs/data_sources.rst
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ FAOSTAT Land Use (RL)

**Retrieval**: Downloaded as bulk CSV (``Inputs_LandUse_E_All_Data_(Normalized).zip``), converted to Parquet, and processed by ``workflow/scripts/prepare_faostat_pasture_area.py``.

**Usage**: Provides per-country permanent pasture area used to scale down satellite grassland area in ``build_model.py``, replacing the previous forage overlap subtraction approach.
**Usage**: Provides per-country permanent pasture area used to scale down satellite grassland area in ``build_model.py``.

FAOSTAT Food Balance Sheets (FBS)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -1140,7 +1140,7 @@ Redistributing datasets via Zenodo
----------------------------------

Some upstream datasets are free to use but sit behind an API key or registration
wall (historically the Copernicus land cover data). Where the licence permits
wall. Where the licence permits
redistribution, GLADE mirrors the exact slice it needs to `Zenodo
<https://zenodo.org/>`__ and downloads it during builds with a plain HTTP
request. This removes the per-user credential, pins an immutable, citable
Expand Down
2 changes: 1 addition & 1 deletion docs/environment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ Manure emissions are calculated in ``workflow/scripts/calculate_manure_emissions
1. **Preprocessing** (``calculate_manure_emissions.py``):

* Calculate VS excretion for each feed category using digestibility and ash content
* Average MCF across climate zones for each management system (temporary simplification)
* Average MCF across climate zones for each management system because model regions do not carry climate-zone detail
* Compute weighted MCF for each animal product using GLEAM MMS distributions
* Calculate CH₄ emissions per kg feed intake: VS × B₀ × MCF\ :sub:`weighted` × 0.67
* Generate ``processing/{name}/manure_ch4_emission_factors.csv`` with emissions by country, product, and feed category
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ The key differences from Part 1:
mutually exclusive — attempting to combine them raises a validation
error.
* The ``ghg_mid`` and ``ghg_high`` scenarios inherit the top-level
``food_utility_piecewise`` settings and no longer set
``food_utility_piecewise`` settings and do not set
``enforce_baseline_diet``, so consumption is free.

The piecewise-utility parameters themselves are worth a brief look:
Expand Down
6 changes: 2 additions & 4 deletions tests/test_compute_gleam3_feed_fractions.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,10 +227,8 @@ def test_mixed_endogenous_exogenous_preserves_exogenous_share(
crop_production: pd.DataFrame,
) -> None:
"""A bucket with both endogenous and exogenous items must emit an
exogenous row alongside the endogenous fractions; previously the
exogenous share was silently absorbed into the endogenous categories
(the real-world failure was GLEAM's 'Grass and leaves' bucket dropping
the LEAVES item)."""
exogenous row alongside the endogenous fractions. The exogenous share
must remain separate from the endogenous categories."""
# Build a Grains bucket with two endogenous items (maize, barley) and
# one unmapped GLEAM code that will be flagged as exogenous.
xlsx_items = pd.DataFrame(
Expand Down
15 changes: 5 additions & 10 deletions tests/test_extract_health_impacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,16 @@

"""Regression tests for ``workflow.scripts.analysis.extract_health_impacts``.

These tests pin down three invariants that previously failed silently:
These tests pin down three analysis invariants:

1. Intake fed into the dose-response chain must come from the post-waste
food-group *store level*, not the pre-waste *food-bus withdrawal*. Earlier
versions of the analysis used ``food_group_consumption.parquet`` for
intake, which is the link-level withdrawal at the food bus -- i.e. the
retail-supply (pre-waste) mass. The LP applies the consumer waste
multiplier on the food_consumption link's group leg, so the store level
is lower. Using the wrong basis silently inflated intake by 10-30%.
food-group *store level*, not the pre-waste *food-bus withdrawal*.
``food_group_consumption.parquet`` contains the link-level withdrawal at
the food bus, while the store level includes the consumer waste multiplier.

2. Dose-response curves must be keyed per ``(health_cluster, risk_factor)``.
``risk_breakpoints.csv`` carries a ``health_cluster`` column because
age-weighted effective RR differs across clusters. A bug pooled curves
across clusters via ``pivot_table(aggfunc='first')``, silently keeping
one cluster's curve everywhere.
age-weighted effective RR differs across clusters.

3. ``RR_d(log_total)`` must be evaluated via the chord PWL of exp() through
``cause_log_breakpoints``, matching ``_add_stage2_lp_tangent``. Using
Expand Down
8 changes: 2 additions & 6 deletions tests/test_sensitivity.py
Original file line number Diff line number Diff line change
Expand Up @@ -722,11 +722,7 @@ def test_full_config(self, mock_network):
np.testing.assert_allclose(result_ch4, original_ch4 * 1.1)

def test_health_rr_config_ignored_at_build_time(self, mock_network):
"""Test that health_relative_risk in config is ignored at build time.

Health RR sensitivity is now applied at solve time via per-risk-factor
quantile interpolation, not at build time.
"""
"""Health relative-risk sensitivity is applied at solve time."""
n = mock_network
original_rr = n.stores.static.loc[
n.stores.static["carrier"].str.startswith("yll_"), "rr_ref"
Expand All @@ -737,7 +733,7 @@ def test_health_rr_config_ignored_at_build_time(self, mock_network):
}
apply_sensitivity_factors(n, cfg)

# rr_ref should be unchanged — health RR is now handled at solve time
# Build-time reference curves remain unchanged.
result_rr = n.stores.static.loc[
n.stores.static["carrier"].str.startswith("yll_"), "rr_ref"
]
Expand Down
11 changes: 3 additions & 8 deletions workflow/rules/common.smk
Original file line number Diff line number Diff line change
Expand Up @@ -76,14 +76,9 @@ def get_effective_config(scenario_name):
# We convert config to dict because it might be a Config object
eff_config = copy.deepcopy(dict(config))

if scenario_name:
if scenario_name not in scenario_defs:
# If scenario is not found, maybe raise warning or error?
# For now, we assume if it's not in cache, no overrides (or invalid scenario handled elsewhere)
pass
else:
overrides = scenario_defs[scenario_name]
_recursive_update(eff_config, overrides)
if scenario_name and scenario_name in scenario_defs:
overrides = scenario_defs[scenario_name]
_recursive_update(eff_config, overrides)

return eff_config

Expand Down
17 changes: 5 additions & 12 deletions workflow/scripts/build_frt_area_attribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,10 @@
"""Per-(country, modelled-fruit) target harvested area for the FRT pool.

The FAO/GAEZ Module-VI ``FRT`` raster bundles ~45 QCL fruit + grape +
tree-nut items at the cell level. Historically the model attributed FRT
cell area to citrus/mango/watermelon via national crop-mix scalars, which
silently placed e.g. citrus area in cells where GAEZ says citrus cannot
grow; those cells were then dropped at build time, losing ~13 Mha of
fruit area globally.

This script replaces the country-share approach with a per-(country,
modelled-fruit) **target area** table that downstream rules use together
with a cell-level yield x suitability weighting to ensure
agroecologically consistent placement.
tree-nut items at the cell level. This script produces a per-(country,
modelled-fruit) **target area** table that downstream rules combine with
cell-level yield x suitability weighting for agroecologically consistent
placement.

For each country and baseline year:

Expand Down Expand Up @@ -90,8 +84,7 @@
234, # Other nuts (excluding wild edible nuts and groundnuts), n.e.c.
)

# All FRT-pool QCL items (fruits + grapes + tree nuts). Mirrors
# build_frt_kept_area_share.FRT_ITEM_CODES at the time this file replaced it.
# All FRT-pool QCL items (fruits + grapes + tree nuts).
FRT_FRUIT_ITEM_CODES: tuple[int, ...] = (
# Melons, tropical and subtropical fruits
567, # Watermelons
Expand Down
6 changes: 2 additions & 4 deletions workflow/scripts/build_model/grassland.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,10 +220,8 @@ def add_grassland_feed_links(

# Defensive guard: only create links whose pasture-pool bus exists. land.py
# builds a pasture pool for every region/class with grassland (cropland or
# grazing-only) supply, so this should retain all rows; but with the
# left-join above we no longer rely on land_rainfed membership to guarantee
# a pool, so drop (and report) any orphan rather than leave an unbalanced
# bus reference.
# grazing-only) supply. Drop and report any orphan rather than leave an
# unbalanced bus reference.
missing_pool = ~work["bus0"].isin(n.buses.static.index)
if missing_pool.any():
logger.warning(
Expand Down
5 changes: 1 addition & 4 deletions workflow/scripts/build_model/land.py
Original file line number Diff line number Diff line change
Expand Up @@ -261,10 +261,7 @@ def add_land_components(
.astype(float)
.rename("area_ha")
)
# Work on a defensive copy: the function used to mutate the caller's
# DataFrame in place via `total_land_area["area_ha"] = ...`. No current
# caller reuses the input after this call, but the side effect is
# invisible from the signature and a latent footgun for future code.
# Work on a defensive copy so callers retain the input DataFrame.
total_land_area = total_land_area.copy()
total_area = total_land_area["area_ha"].astype(float)
# FAOSTAT-baked baseline cropland sometimes exceeds the GAEZ-derived
Expand Down
2 changes: 1 addition & 1 deletion workflow/scripts/build_model/trade.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ def _add_trade_hubs_and_links(
# valid_countries filters out anything outside country_to_hub.index,
# so .map should never introduce NaN and the int dtype is preserved.
# If a NaN slipped through, astype(str) would produce "0.0" strings
# that no longer match the integer-suffix hub bus names.
# that do not match the integer-suffix hub bus names.
assert pairs["hub_idx"].notna().all()
hub_idx_str = pairs["hub_idx"].astype(int).astype(str)

Expand Down
7 changes: 3 additions & 4 deletions workflow/scripts/build_region_watergap.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
- **mined groundwater** = the groundwater-storage decline. The trend reflects
all users, so irrigation's part is attributed by its share of potential
groundwater consumption (``pirrusegw / ptotusegw``, same basis and window);
a basin mined by municipal or industrial pumping then no longer zeroes
a basin mined by municipal or industrial pumping does not zero
irrigation's renewable band.
- **renewable groundwater** = ``max(pirrusegw - mined_irrigation, 0)``: the
recharged part of irrigation groundwater consumption.
Expand Down Expand Up @@ -307,9 +307,8 @@ def monthly_to_regions(path, variable):

# Total irrigation consumption (pirruse, annual): the demand anchor for
# eta_c and the groundwater mining ceiling, on the same basis, simulation
# and reference window as the supply envelope above. Same schema as the
# legacy AWARE anchor (region, agri_consumption_m3) so consumers are
# source-agnostic.
# and reference window as the supply envelope above. The shared schema
# keeps consumers independent of the source dataset.
agri = pd.DataFrame({"region": region_index}).assign(
agri_consumption_m3=lambda d: d["region"].map(irr_total.sum(axis=1)).fillna(0.0)
)
Expand Down
9 changes: 4 additions & 5 deletions workflow/scripts/calculate_manure_emissions.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,8 @@ def average_mcf_over_climate_zones(
) -> pd.DataFrame:
"""Average MCF values across climate zones for each manure management system.

NOTE: This is a temporary simplification. Will be refined when climate zone
data is added to modeling regions.
Climate-zone variation is averaged because model regions do not carry
climate-zone detail.

Parameters
----------
Expand Down Expand Up @@ -518,9 +518,8 @@ def calculate_manure_ch4_for_product(
row["storage_n2o_ef"],
)

# Expand to all countries (same values for now, will be refined later)
# NOTE: Emission factors are currently identical across countries
# Will be differentiated by climate zone and region in future
# Emission factors are shared across countries because the source data is
# not spatially resolved to model regions.
country_emissions = []
for country in countries:
df = emissions.copy()
Expand Down
2 changes: 1 addition & 1 deletion workflow/scripts/calibrate_deviation_penalty.py
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ def main() -> None:
max_iter = int(smk.params.max_iter)
trust_log = float(smk.params.trust_region_log)

# Warm-start from a previously calibrated YAML when available. The path
# Warm-start from an existing calibrated YAML when available. The path
# is passed as a param (not an input) to avoid a Snakemake self-loop on
# calibrated_yaml. The script loads it iff the file exists on disk.
prev_yaml = getattr(smk.params, "previous_yaml", None)
Expand Down
10 changes: 5 additions & 5 deletions workflow/scripts/diet/food_group_projection.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,11 @@
"apple",
)
FRUITS_FRT_POOL_ITEM_CODES: tuple[int, ...] = (
2611, # Oranges, Mandarines (explicit; previously routed to citrus only)
2612, # Lemons, Limes (explicit; previously routed to citrus only)
2613, # Grapefruit (explicit; previously routed to citrus only)
2614, # Citrus, Other (explicit; previously routed to citrus only)
2617, # Apples (explicit; previously routed to apple only)
2611, # Oranges, Mandarines (explicit citrus allocation)
2612, # Lemons, Limes (explicit citrus allocation)
2613, # Grapefruit (explicit citrus allocation)
2614, # Citrus, Other (explicit citrus allocation)
2617, # Apples (explicit apple allocation)
2618, # Pineapples (residual)
2619, # Dates (residual)
2625, # Fruits, Other (residual covering mango, watermelon, etc.)
Expand Down
6 changes: 2 additions & 4 deletions workflow/scripts/estimate_baseline_diet.py
Original file line number Diff line number Diff line change
Expand Up @@ -725,11 +725,9 @@ def build_within_group_shares(
].to_dict()

# Fail fast if any pooled projection references an FBS code that the
# FBS fetch never pulled. This catches silent drift between
# FBS fetch never pulled. This catches drift between
# ``POOL_PROJECTIONS`` here and the fetch list in
# ``prepare_faostat_fbs_items.py`` (which historically defaulted such
# codes to 0 supply and made entire pools — plantain, apples,
# pineapples, dates — silently vanish from the projection).
# ``prepare_faostat_fbs_items.py``.
# Only validate sub-specs whose projection_foods intersect with the
# included foods (the rest are no-ops anyway) so that downstream tests
# can exercise subsets without supplying every code.
Expand Down
3 changes: 1 addition & 2 deletions workflow/scripts/generate_rr_age_attenuation.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@
relative-risk curves. GBD applies age-specific RRs for cardiovascular outcomes
(the proportional effect attenuates with age); diabetes and colorectal cancer
carry no age attenuation. We reconstruct that age structure once and freeze it
into a curated table so the per-build workflow no longer depends on the GBD 2019
relative-risk workbook.
into a curated table used by the per-build workflow.

Method
------
Expand Down
5 changes: 1 addition & 4 deletions workflow/scripts/plotting/plot_food_consumption.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,14 @@
from pathlib import Path

import matplotlib

matplotlib.use("pdf")
import matplotlib.pyplot as plt
import pandas as pd

from workflow.scripts.constants import DAYS_PER_YEAR, GRAMS_PER_MEGATONNE, PJ_TO_KCAL
from workflow.scripts.logging_config import setup_script_logging
from workflow.scripts.plotting.color_utils import categorical_colors

# Alias for backwards compatibility with modules that import from here
KCAL_PER_PJ = PJ_TO_KCAL
matplotlib.use("pdf")


def _load_global_consumption(
Expand Down
5 changes: 2 additions & 3 deletions workflow/scripts/prepare_faostat_animal_production.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,8 @@ def main() -> None:
)

# Egg unit handling. Only hen eggs are mapped to the "eggs" model
# product (config/default.yaml), so the 60 g/egg conversion is a
# reasonable approximation for the legacy "1000 No" vintage. Current
# FAOSTAT QCL reports tonnes ("t") and the legacy branch is dormant.
# product (config/default.yaml). FAOSTAT may report egg production in
# tonnes ("t") or in thousands of eggs ("1000 No").
egg_mask_raw = df["product"] == "eggs"
if egg_mask_raw.any():
if "Unit" not in df.columns:
Expand Down
2 changes: 1 addition & 1 deletion workflow/scripts/prepare_gbd_food_group_intake.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
therefore reconstruct the 25+ exposure ourselves by
population-weighting the adult 5-year buckets (25-29 .. 95+) using
per-country age-bucket population for the reference year.
- Sex: the 2023 bulk files no longer ship an IHME-computed "Both"
- Sex: the 2023 bulk files do not ship an IHME-computed "Both"
aggregate (only Male and Female), and our population table is not
sex-split. We average the two sexes unweighted within each age
bucket before age-weighting. Adult sex ratios are close to parity,
Expand Down
Loading
Loading