Docs/readme and modules

.gitignore

@@ -21,4 +21,7 @@
 .pixi/*
 !.pixi/config.toml
 
-shift_data.zip
+shift_data.zip
+
+# mkdocs build output
+/site

README.md

@@ -1,103 +1,52 @@
 # SHIFT – Steel & Hydrogen Integrated Freight Trade
 
-This repository contains the **SHIFT model**, a spatially resolved techno-economic optimization of global iron and steel supply chains under decarbonization.
-It explores how hydrogen-based direct reduced iron (DRI) production and hot-briquetted iron (HBI) trade can shift value creation to regions with renewable energy and capital availability.
+**SHIFT** is a spatially resolved techno-economic optimization of global iron
+and steel supply chains under decarbonization. It explores how
+hydrogen-based direct reduced iron (DRI) production and hot-briquetted iron
+(HBI) trade can shift value creation to regions with renewable energy and
+capital availability, using a two-stage pipeline (greenfield supply curves,
+then cross-region LP trade) built on [PyPSA](https://pypsa.org/) and
+orchestrated with [Snakemake](https://snakemake.readthedocs.io/).
 
-The model identifies cost-optimal configurations for mining, hydrogen production, DRI processing, and HBI trade, using a two-stage optimization pipeline and open energy system libraries.
-
-## Introduction SHIFT
-
-SHIFT evaluates global supply and trade of low-carbon iron and steel at high spatial resolution.
-The model quantifies where to produce, where to ship, and how to meet demand cost-effectively under decarbonization constraints.
-
-Key features:
-- 🚀 two-stage process: greenfield supply curves + cross-region LP trade
-- 🌍 spatial renewable potentials: PyPSA-Earth wind/solar CF distributions
-- 💰 integrated LCOX: region-level cost curves for H2, DRI, HBI
-- 🛳️ global trade dispatch: route costs, flows, nodal prices, utilization
-- ⚙️ configurable automation: regions, technologies, scenarios via YAML + Snakemake rules
-
-## Quick installation (PIXI)
-
-Clone the repository:
+## Quick start – installation & execution
 
 ```sh
 git clone https://github.com/energyLS/shift.git
 cd shift
-```
-
-Install dependencies with pixi:
-
-```sh
 pixi install
+pixi run snakemake model_trade_all
 ```
 
-Activate the environment:
+Approximate runtimes:
 
-```sh
-pixi shell
-```
+- Installation (`pixi install`): ~5 minutes
+- Full model (`model_trade_all`): ~5 hours
 
-(Optional) Install pre-commit hooks:
+> [!TIP]
+> For a quick, low-resolution run, check out the
+> [`demo` branch](https://github.com/energyLS/shift/tree/demo), which uses
+> fewer regions and quantities and completes in ~1 hour.
 
-```sh
-pre-commit install
-```
+## Key results
 
+`model_trade_all` populates `results/<scenario>/` for each configured trade
+scenario with:
 
-For details, see https://pixi.prefix.dev/latest/ (or your local PIXI docs).
+- `result.csv` – regional production volumes and shipped quantities
+- `map_ironore.pdf`, `map_hbi.pdf`, `map_steel.pdf` – trade-flow maps by product
+- `network.nc` – the full PyPSA network, for further analysis
 
-## Run (core workflow)
+## Documentation
 
-In the workspace root:
+Full documentation - installation details, solver setup, the workflow
+stages, config flags, and a module-by-module reference - lives in `docs/`
+and is built with mkdocs:
 
 ```sh
-pixi run snakemake model_trade_all
+pixi run -e docs mkdocs serve
 ```
 
-To collect all figures (under development):
-
-```sh
-pixi run snakemake collect_figures
-```
-
-
-## Workflow overview
-
-### Configuration & Scenario Setup (implicit)
-
-Before execution, Snakemake reads:
-- **Global settings**: `config/config.yaml` (regions, cost years, solver options, enable flags)
-- **Scenario matrix**: `config/trade_scenarios.csv` (rows = distinct trade scenarios)
-
-These expand into a deterministic wildcard space (cost_year, region, product, scenario) that drives all downstream rule creation. This bootstrap is handled automatically by Snakemake; no user action required.
-
-### Step 0: Renewable potentials (pre-computed inputs)
-
-In this stage we generate the supply-side resource backbone.
-Renewable capacity-factor series and maximum deployable potentials are pre-computed externally using PyPSA-Earth's `build_renewable_profiles` rule and stored in `data/renewable_profiles/`.
-This stage is not part of the current Snakefile. SHIFT consumes pre-computed .nc datasets to avoid the long runtime of full GIS processing.
-
-> **For new users:** No action needed. Renewable data files are provided in the repository.
-
-### Step 1: Greenfield supply curve generation (PyPSA)
-
-With renewable profiles and [techno-economic assumptions](https://github.com/PyPSA/technology-data) in place, SHIFT builds regional PyPSA optimization models to size generation, storage, and process assets.
-It evaluates each candidate plant (H2 electrolyser, DRI furnace, HBI plant, steel mills) across resource quality and cost parameters to produce levelized cost curves (LCOX) as a function of capacity.
-The result is a fleet of supply curve elements (capacity buckets with marginal costs and metadata) for H2, DRI, HBI, and steel by region.
-
-Step 1 is a multi-part stage: load techno-economic data and build regional cost baselines, prepare renewable candidate sets per region, solve optimization problems at discrete demand levels, and consolidate results into piecewise supply curves. Each stage depends on the prior; files are persisted between steps to support reproducibility and debugging.
-The output is a harmonized set of supply curves used as input for the trade stage.
-
-### Step 2: Global trade optimization (LP)
-
-This stage takes regional supply curves and demand obligations, then runs a linear program over the regional network.
-It includes transport cost matrices, ore production constraints, and market compatibility.
-The solver decides how much each region should produce versus import/export, by product and route.
-
-The trade solution yields detailed outputs: regional production volume and shipped quantities.
-It can also be reconciled with scenarios for demand, policy constraints, and infrastructure availability.
-
+then open <http://127.0.0.1:8000>.
 
 ## Acknowledgements
 

config/config.yaml

@@ -1,7 +1,7 @@
 enable:
-  run_supply_chain: False # Enable for first run
-  run_supply_curve: False # Enable for first run
-  cluster_renewables: False # Enable for first run
+  run_supply_chain: True # Set False to skip re-deriving once resources exist on disk
+  run_supply_curve: True # Set False to skip re-deriving once resources exist on disk
+  cluster_renewables: True # Set False to skip re-deriving once resources exist on disk
 
 run:
   disable_progressbar: false

docs/index.md

@@ -0,0 +1,29 @@
+# SHIFT — Steel & Hydrogen Integrated Freight Trade
+
+SHIFT is a spatially resolved techno-economic optimization of global iron and
+steel supply chains under decarbonization. It explores how hydrogen-based
+direct reduced iron (DRI) production and hot-briquetted iron (HBI) trade can
+shift value creation to regions with renewable energy and capital
+availability.
+
+The model identifies cost-optimal configurations for mining, hydrogen
+production, DRI processing, and HBI trade, using a two-stage optimization
+pipeline built on [PyPSA](https://pypsa.org/) and orchestrated with
+[Snakemake](https://snakemake.readthedocs.io/).
+
+Key features:
+
+- **Two-stage process**: greenfield supply curves + cross-region LP trade
+- **Spatial renewable potentials**: PyPSA-Earth wind/solar CF distributions
+- **Integrated LCOX**: region-level cost curves for H2, DRI, HBI
+- **Global trade dispatch**: route costs, flows, nodal prices, utilization
+- **Configurable automation**: regions, technologies, scenarios via YAML + Snakemake rules
+
+## Where to go next
+
+- [Installation & running the pipeline](installation.md) — pixi setup, the
+  core Snakemake commands, and known platform-specific gotchas (notably a
+  Snakemake scheduler issue on Windows).
+- [Workflow overview](workflow.md) — what each pipeline stage does.
+- [Module reference](modules.md) — auto-generated documentation for each
+  script in `workflow/scripts/`, pulled directly from their docstrings.

docs/installation.md

@@ -0,0 +1,61 @@
+# Installation & running the pipeline
+
+## Install dependencies with pixi
+
+```sh
+git clone https://github.com/energyLS/shift.git
+cd shift
+pixi install
+pixi shell
+```
+
+(Optional) install pre-commit hooks:
+
+```sh
+pre-commit install
+```
+
+For details on pixi itself, see <https://pixi.prefix.dev/latest/>.
+
+## Solvers
+
+The pipeline solves PyPSA/linopy optimization problems and supports two
+solver backends, selected via `config/config.yaml`'s `solver.name` (or
+overridden with the `SHIFT_SOLVER` / `SHIFT_SOLVER_OPTIONS` environment
+variables):
+
+- **Gurobi** (`solver.name: gurobi`) — the default. Requires a Gurobi
+  license (e.g. a `gurobi.lic` file); `gurobipy` is installed automatically.
+- **HiGHS** (`solver.name: highs`, or `SHIFT_SOLVER=highs`) — a free
+  fallback that needs no license. This is what CI uses, since CI runners
+  don't have a Gurobi license.
+
+Only these two are actively configured (`config/config.yaml`'s
+`solver_options` only defines `gurobi-default` and `highs-default`
+presets). Other solver names some dependencies could technically support
+are not wired up with tuned options here.
+
+## Run the core workflow
+
+From the repository root:
+
+```sh
+pixi run snakemake model_trade_all
+```
+
+On Windows, Snakemake's default job scheduler (an ILP solver via PuLP +
+CBC) can fail with a `PulpSolverError` caused by a `-threads` flag
+mismatch with the bundled `cbc.exe`. If you hit that, add
+`--scheduler greedy`:
+
+```sh
+pixi run snakemake --scheduler greedy model_trade_all
+```
+
+This only changes how Snakemake orders/parallelizes ready jobs, not what
+gets computed.
+
+`collect_figures` does not currently work - its `input:` block references
+wildcards (`wacc`, `scenario`, `sort`, `demand`) that are never bound to
+concrete values, so Snakemake fails immediately with a `WildcardError`
+before building the DAG.

docs/modules.md

@@ -0,0 +1,39 @@
+# Module reference
+
+Auto-generated from the docstrings in `workflow/scripts/`. If a summary
+here looks out of date, the docstring in the source file is what's wrong -
+fix it there.
+
+## Shared utilities
+
+::: _helpers
+
+::: trade_chain_utils
+
+## Data preparation
+
+::: download_labour_data
+
+::: prepare_labour_cost
+
+::: tech_database
+
+::: cluster_renewables
+
+::: preprocess_consolidate_renewables
+
+::: renewable_profiles
+
+## Supply curve generation (Step 1)
+
+::: build_x_supply_chain
+
+::: prepare_regional_network
+
+::: calculate_lcox
+
+::: create_supply_curve
+
+## Trade optimization (Step 2)
+
+::: model_trade

docs/workflow.md

@@ -0,0 +1,96 @@
+# Workflow overview
+
+### Configuration & scenario setup (implicit)
+
+Before execution, Snakemake reads:
+
+- **Global settings**: `config/config.yaml` (regions, cost years, solver options, enable flags)
+- **Scenario matrix**: `config/trade_scenarios.csv` (rows = distinct trade scenarios)
+
+These expand into a deterministic wildcard space (cost_year, region,
+product, scenario) that drives all downstream rule creation. This
+bootstrap is handled automatically by Snakemake; no user action required.
+
+### The `enable` flags
+
+`config/config.yaml`'s `enable` block gates three of the most expensive
+rules in the pipeline, and ships **`True`** by default so a fresh clone
+runs end-to-end with no manual edits:
+
+```yaml
+enable:
+  run_supply_chain: True   # gates rule calculate_regional_lcox
+  run_supply_curve: True   # gates rule create_supply_curve
+  cluster_renewables: True # gates rule cluster_renewables
+```
+
+These aren't ordinary "skip if already done" toggles - each one wraps an
+entire `rule` definition in an `if config["enable"].get(...)` in the
+`.smk` files, so when a flag is `False`, Snakemake doesn't just skip
+re-running that rule, **the rule doesn't exist in the DAG at all**:
+
+- `cluster_renewables: False` removes the `cluster_renewables` rule, so
+  `resources/renewables_clustered.nc` must already exist on disk.
+- `run_supply_chain: False` removes `calculate_regional_lcox`, so the
+  per-region/product/demand-level LCOX result CSVs it would produce
+  (`resources/lco-{product}/.../results_{demand}.csv`) must already exist.
+- `run_supply_curve: False` removes `create_supply_curve`, so the
+  combined supply-curve CSVs/PDFs
+  (`resources/supply_curves/.../{region}_marginal_cost_{product}.csv`)
+  must already exist.
+
+`resources/` is gitignored - nothing under it ships in the repo - so all
+three flags must be `True` the first time you run against a given config
+(regions, cost year, scenarios). Once the resources exist on disk, you
+can set them back to `False` to skip re-deriving them on subsequent runs
+- useful because renewable clustering alone takes on the order of 10+
+minutes, and LCOX/supply-curve generation scales with the number of
+configured regions.
+
+The trap: if you change something the cached resources depend on (most
+obviously `regions`, but also `cost_year` or clustering parameters) while
+leaving a flag `False`, Snakemake won't silently regenerate the stale
+file for you - since the rule doesn't exist, you'll either get a hard
+`MissingInputException`/`No rule to produce ...` error, or worse, a run
+that silently reuses resources computed for the *previous* region set.
+When in doubt (e.g. after editing `regions:`), set all three back to
+`True` for one run.
+
+### Step 0: Renewable potentials
+
+Renewable capacity-factor series and maximum deployable potentials are
+retrieved from a pre-computed Zenodo archive (see the `retrieve_data`
+rule) rather than generated locally, to avoid the long runtime of full
+GIS processing via PyPSA-Earth's `build_renewable_profiles`.
+
+### Step 1: Greenfield supply curve generation (PyPSA)
+
+With renewable profiles and [techno-economic assumptions](https://github.com/PyPSA/technology-data)
+in place, SHIFT builds regional PyPSA optimization models to size
+generation, storage, and process assets. It evaluates each candidate
+plant (H2 electrolyser, DRI furnace, HBI plant, steel mills) across
+resource quality and cost parameters to produce levelized cost curves
+(LCOX) as a function of capacity. The result is a fleet of supply curve
+elements (capacity buckets with marginal costs and metadata) for H2, DRI,
+HBI, and steel by region.
+
+Step 1 is a multi-part stage: load techno-economic data and build
+regional cost baselines, prepare renewable candidate sets per region,
+solve optimization problems at discrete demand levels, and consolidate
+results into piecewise supply curves. Each stage depends on the prior;
+files are persisted between steps to support reproducibility and
+debugging. See [`calculate_lcox`](modules.md#calculate_lcox) and
+[`create_supply_curve`](modules.md#create_supply_curve).
+
+### Step 2: Global trade optimization (LP)
+
+This stage takes regional supply curves and demand obligations, then runs
+a linear program over the regional network. It includes transport cost
+matrices, ore production constraints, and market compatibility. The
+solver decides how much each region should produce versus import/export,
+by product and route.
+
+The trade solution yields detailed outputs: regional production volume
+and shipped quantities. It can also be reconciled with scenarios for
+demand, policy constraints, and infrastructure availability. See
+[`model_trade`](modules.md#model_trade).