Runner refactoring 6/6: Add multi-seed benchmarking support

runner/README.md

@@ -48,6 +48,7 @@ once per given year.
 - `-a, --append` - Append to the results CSVs instead of overwriting them for the first year
 - `-y, --years YEAR` - Solver-version year to run (repeatable), or `tests` for the shared CI smoke-test env. Defaults to every year with a registered solver version
 - `-s, --solver-configurations CONFIG` - Solver configuration to run (repeatable), e.g. `highs-default` or `highs-hipo`. Defaults to `solver_configurations.yaml`'s `default_configurations`
+- `-n, --num-seeds N` - Number of seeds to try per (problem, solver configuration) pair. When greater than 1, each repetition uses a different seed (1, 2, 3, ...) instead of the configuration's own fixed seed, to gauge the solver's sensitivity to it. Default: 1 (no repetition, the configuration's own fixed seed applies)
 - `-r, --ref-bench-interval SECONDS` - Run a reference benchmark at most once every N seconds. 0 disables it
 - `-u, --run-id RUN_ID` - Identifier shared by every row from this run. Auto-generated if not given
 - `--help` - Show this message and exit
@@ -70,6 +71,11 @@ pixi run -e runner python -m runner.benchmark --solver-configurations highs-defa
 pixi run -e runner python -m runner.benchmark --years 2025 results/metadata.yaml
 ```
 
+4. Run each problem 3 times per solver configuration, under 3 different seeds, to gauge runtime sensitivity to the seed:
+```shell
+pixi run -e runner python -m runner.benchmark --num-seeds 3 --years 2025 benchmarks/sample_run/standard-00.yaml
+```
+
 ## Running with Docker
 
 Docker is optional. On native Linux with systemd, you can run the scripts directly (see above). Memory limit enforcement via `systemd-run` is skipped automatically when systemd is not available.
@@ -124,13 +130,14 @@ docker run --rm \
 Use `runner.utils.solver` to test a single solver on a single problem. This is useful for debugging. Since it's a package module (not a standalone script), run it with `-m` **from the repo root**, not from `runner/`:
 
 ```bash
-python -m runner.utils.solver <solver_configuration> <input_file> <solver_version>
+python -m runner.utils.solver <solver_configuration> <input_file> <solver_version> [--seed N]
 ```
 
 **Arguments:**
 - `solver_configuration` - Solver configuration name (e.g., highs-default, highs-hipo, scip-default)
 - `input_file` - Path to a problem file (.lp or .mps)
 - `solver_version` - Solver version string (e.g., 1.10.0)
+- `--seed N` - Optional. Overrides the configuration's own fixed seed (see `runner/config/solvers.yaml`'s `seed_options`)
 
 **Examples:**
 
@@ -140,6 +147,9 @@ pixi run --manifest-path runner/envs/benchmark-highs-2024 python -m runner.utils
 
 # Test SCIP
 pixi run --manifest-path runner/envs/benchmark-scip-2024 python -m runner.utils.solver scip-default runner/benchmarks/pypsa-eur-elec-op-2-1h.lp 9.2.2
+
+# Test HiGHS with a specific seed instead of highs-default's own fixed one
+pixi run --manifest-path runner/envs/benchmark-highs-2024 python -m runner.utils.solver highs-default runner/benchmarks/pypsa-eur-elec-op-2-1h.lp 1.10.0 --seed 7
 ```
 
 **Output:**

runner/SOLVERS.md

@@ -35,7 +35,7 @@ Onboarding a solver package we don't support at all yet (as opposed to a new ver
 
 1. Confirm [linopy](https://github.com/PyPSA/linopy) already supports it (`linopy.solvers.SolverName` lists every solver it knows how to drive) -- everything below assumes it does. If not, that support has to land in linopy first.
 2. Its per-solver-year pixi manifest(s) under `runner/envs/` (step 1 of [Updating Solver Versions](#updating-solver-versions) above).
-3. A registry entry in `runner/config/solvers.yaml`'s `solvers` map (version/year/env), plus a `packages` entry if its PyPI package name differs from the solver's own name, plus a `license_env_vars` entry if it needs a license env var forwarded under `systemd-run` (see that file's own header comment for the full schema).
+3. A registry entry in `runner/config/solvers.yaml`'s `solvers` map (version/year/env), plus a `packages` entry if its PyPI package name differs from the solver's own name, a `license_env_vars` entry if it needs a license env var forwarded under `systemd-run`, and a `seed_options` entry naming the options key that holds its random seed (used to run a configuration under several different seeds, e.g. via `runner.benchmark`'s `--num-seeds`) -- see that file's own header comment for the full schema.
 4. A solver adapter module at `runner/utils/solvers/<solver_package>.py` exporting `is_mip(model)`, `duality_gap(model)`, and `reported_runtime(model)` -- copy any existing module there as a template. It's auto-discovered by filename; nothing else needs to import or register it (see `runner/utils/solvers/__init__.py`'s own docstring).
 5. At least a `<solver_package>-default` entry in `runner/config/solver_configurations.yaml` (see [Adding a New Solver Configuration](#adding-a-new-solver-configuration) below), and add it to `default_configurations` there if it should run whenever the CLI is given no explicit `--solver-configurations`.
 

runner/benchmark.py

@@ -44,6 +44,17 @@ def run(
         help="Append to the results CSVs instead of overwriting them for "
         "the first year.",
     ),
+    num_seeds: int = typer.Option(
+        1,
+        "--num-seeds",
+        "-n",
+        help="Number of seeds to try per (problem, solver configuration) "
+        "pair. When greater than 1, each repetition uses a different seed "
+        "(1, 2, 3, ...) instead of the configuration's own fixed seed, to "
+        "gauge the solver's sensitivity to it (see "
+        "`runner/config/solvers.yaml`'s `seed_options`). Default: 1 (the "
+        "configuration's own fixed seed, no repetition).",
+    ),
     ref_bench_interval: int = typer.Option(
         0,
         "--ref-bench-interval",
@@ -88,6 +99,7 @@ def run(
                 problems_yaml_path,
                 resolved_solver_configurations,
                 year=year,
+                num_seeds=num_seeds,
                 reference_interval=ref_bench_interval,
                 append=append or index > 0,
                 run_id=resolved_run_id,

runner/config/solver_configurations.yaml

@@ -24,10 +24,13 @@
 #     package's own option-key vocabulary (hence the very different-looking
 #     names for the same underlying setting, e.g. HiGHS's "mip_rel_gap" vs
 #     Gurobi's "MIPGap" vs CPLEX's "mip.tolerances.mipgap" all reference
-#     `shared.mip_gap`). Seed values are NOT unified across solvers --
-#     GLPK/SCIP default to 0, CBC to 1 (0 means "time of day" for CBC
-#     specifically), the rest to 4 -- preserved as-is rather than
-#     homogenized here. A non-default configuration's `options` entirely
+#     `shared.mip_gap`). Seed values default to 0 for every solver except
+#     CBC, whose own seed option (randomCbcSeed) treats 0 as "use the time
+#     of day" rather than an actual fixed seed -- confirmed via each
+#     solver's own option docs that 0 is an ordinary, deterministic value
+#     everywhere else (some solvers have their own *different* "auto"
+#     sentinel instead, e.g. Xpress's is -1, GLPK's is the literal string
+#     "?" -- none of them is 0). A non-default configuration's `options` entirely
 #     replace its solver's default-configuration options (rather than
 #     layering on top), matching how picking a specific algorithm is an
 #     all-or-nothing choice.
@@ -68,7 +71,7 @@ configurations:
   highs-default:
     solver_package: highs
     options:
-      random_seed: 4
+      random_seed: 0
       mip_rel_gap: { shared: mip_gap }
       primal_feasibility_tolerance: { shared: lp_tolerance }
       dual_feasibility_tolerance: { shared: lp_tolerance }
@@ -121,7 +124,7 @@ configurations:
   gurobi-default:
     solver_package: gurobi
     options:
-      seed: 4
+      seed: 0
       MIPGap: { shared: mip_gap }
       FeasibilityTol: { shared: lp_tolerance }
       OptimalityTol: { shared: lp_tolerance }
@@ -131,7 +134,7 @@ configurations:
   cplex-default:
     solver_package: cplex
     options:
-      randomseed: 4
+      randomseed: 0
       mip.tolerances.mipgap: { shared: mip_gap }
       simplex.tolerances.feasibility: { shared: lp_tolerance }
       simplex.tolerances.optimality: { shared: lp_tolerance }
@@ -140,7 +143,7 @@ configurations:
   knitro-default:
     solver_package: knitro
     options:
-      ms_seed: 4
+      ms_seed: 0
       mip_opt_gap_rel: { shared: mip_gap }
       feastol: { shared: lp_tolerance }
       opttol: { shared: lp_tolerance }
@@ -149,7 +152,7 @@ configurations:
   xpress-default:
     solver_package: xpress
     options:
-      randomseed: 4
+      randomseed: 0
       miprelstop: { shared: mip_gap }
       FEASTOL: { shared: lp_tolerance }
       OPTIMALITYTOL: { shared: lp_tolerance }
@@ -158,7 +161,7 @@ configurations:
   mosek-default:
     solver_package: mosek
     options:
-      MSK_IPAR_MIO_SEED: 4
+      MSK_IPAR_MIO_SEED: 0
       MSK_IPAR_INTPNT_BASIS: MSK_BI_NEVER
       MSK_DPAR_MIO_TOL_REL_GAP: { shared: mip_gap }
       MSK_DPAR_INTPNT_TOL_PFEAS: { shared: lp_tolerance }

runner/config/solvers.yaml

@@ -20,6 +20,13 @@
 # needs, forwarded to systemd-run's isolated scope via `--setenv=` (it
 # doesn't inherit the caller's environment by default -- see
 # `execution.run_solver`). Only solvers that need one are listed.
+#
+# `seed_options`: the options key that holds a solver package's random seed,
+# in its own option vocabulary (see solver_configurations.yaml) -- lets a
+# caller override just the seed while keeping a configuration's other tuned
+# options (e.g. for running multiple seeds of the same configuration; see
+# `solver.get_solver`'s `seed` parameter). Every solver package listed in
+# `packages` above has an entry here.
 solvers:
   glpk:
     "5.0":
@@ -134,3 +141,14 @@ license_env_vars:
   knitro: [ARTELYS_LICENSE]
   xpress: [XPRESS, LD_LIBRARY_PATH]
   mosek: [MOSEKLM_LICENSE_FILE]
+
+seed_options:
+  glpk: seed
+  highs: random_seed
+  scip: randomization/randomseedshift
+  gurobi: seed
+  cbc: randomCbcSeed
+  cplex: randomseed
+  knitro: ms_seed
+  xpress: randomseed
+  mosek: MSK_IPAR_MIO_SEED

runner/tests/test_benchmark.py

@@ -88,6 +88,40 @@ def test_single_year_run_writes_results(self, problems_yaml, tmp_path):
         assert list(results["Problem"]) == ["tiny-problem"]
         assert results.iloc[0]["Solver Release Year"] == 2025
 
+    def test_num_seeds_flag_varies_seed_across_repetitions(
+        self, problems_yaml, tmp_path
+    ):
+        result = runner_cli.invoke(
+            benchmark.app,
+            [
+                str(problems_yaml),
+                "--years",
+                "2025",
+                "--solver-configurations",
+                "highs-default",
+                "--num-seeds",
+                "3",
+            ],
+        )
+        assert result.exit_code == 0, result.output
+        results = pd.read_csv(tmp_path / "results" / "benchmark_results.csv")
+        assert sorted(results["Seed"]) == [1, 2, 3]
+
+    def test_default_num_seeds_leaves_seed_column_empty(self, problems_yaml, tmp_path):
+        result = runner_cli.invoke(
+            benchmark.app,
+            [
+                str(problems_yaml),
+                "--years",
+                "2025",
+                "--solver-configurations",
+                "highs-default",
+            ],
+        )
+        assert result.exit_code == 0, result.output
+        results = pd.read_csv(tmp_path / "results" / "benchmark_results.csv")
+        assert results["Seed"].isna().all()
+
     def test_tests_pseudo_year_runs_against_real_solver_registry(
         self, problems_yaml, tmp_path
     ):

runner/tests/test_config.py

@@ -11,6 +11,7 @@
     get_default_configurations,
     get_license_env_vars,
     get_package_name,
+    get_seed_option,
     get_solver_configuration,
     get_solver_options,
     is_solver_eligible,
@@ -131,6 +132,16 @@ def test_solver_with_no_entry_returns_empty_list(self):
         assert get_license_env_vars("highs", config) == []
 
 
+class TestGetSeedOption:
+    def test_looks_up_seed_option(self):
+        config = {"seed_options": {"highs": "random_seed"}}
+        assert get_seed_option("highs", config) == "random_seed"
+
+    def test_solver_with_no_entry_returns_none(self):
+        config = {"seed_options": {}}
+        assert get_seed_option("highs", config) is None
+
+
 class TestConditionMatches:
     def test_empty_condition_matches_anything(self):
         assert _condition_matches({"year": "2025"}, {}) is True

runner/tests/test_execution.py

@@ -117,6 +117,28 @@ def test_command_invokes_solver_module_via_dash_m(self, mocker):
         ]
         assert called_cmd[called_cmd.index("runner.utils.solver") - 1] == "-m"
 
+    def test_seed_is_appended_to_command(self, mocker):
+        cp = subprocess.CompletedProcess(
+            args=[], returncode=124, stdout="", stderr="MaxResidentSetSizeKB=1000"
+        )
+        mocker.patch("runner.utils.execution._systemd_available", return_value=False)
+        run_mock = mocker.patch(
+            "runner.utils.execution.subprocess.run", return_value=cp
+        )
+        run_solver("problem.lp", "highs", timeout=3600, solver_version="1.9.0", seed=42)
+        called_cmd = run_mock.call_args[0][0]
+        assert called_cmd[-2:] == ["--seed", "42"]
+
+    def test_no_seed_omits_seed_flag(self, mocker):
+        # Backward compatibility: omitting `seed` must produce the exact
+        # same command as before this argument existed.
+        cp = subprocess.CompletedProcess(
+            args=[], returncode=124, stdout="", stderr="MaxResidentSetSizeKB=1000"
+        )
+        _, run_mock = self._run(mocker, cp)
+        called_cmd = run_mock.call_args[0][0]
+        assert "--seed" not in called_cmd
+
     def test_env_name_uses_pixi_run(self, mocker):
         cp = subprocess.CompletedProcess(
             args=[], returncode=124, stdout="", stderr="MaxResidentSetSizeKB=1000"

runner/tests/test_orchestrator.py

@@ -129,7 +129,7 @@ def test_append_false_overwrites_existing_results(
         results = pd.read_csv(results_csv)
         assert "Problem" in results.columns
 
-    def test_multiple_iterations_computes_mean_and_stddev(
+    def test_multiple_seeds_computes_mean_and_stddev(
         self, problems_yaml, tmp_path, mocker
     ):
         mocker.patch.object(
@@ -140,16 +140,39 @@ def test_multiple_iterations_computes_mean_and_stddev(
             ["highs-default"],
             year="2025",
             run_id="test-run",
-            iterations=2,
+            num_seeds=2,
         )
         summary = pd.read_csv(
             tmp_path / "results" / "benchmark_results_mean_stddev.csv"
         )
         assert summary.iloc[0]["Runtime StdDev (s)"] == 0.0
 
-    def test_error_status_stops_further_iterations(
-        self, problems_yaml, tmp_path, mocker
-    ):
+    def test_num_seeds_greater_than_one_varies_seed(self, problems_yaml, mocker):
+        run_solver_mock = mocker.patch.object(
+            orchestrator, "run_solver", return_value=dict(_FAKE_METRICS)
+        )
+        orchestrator.run_benchmark(
+            problems_yaml,
+            ["highs-default"],
+            year="2025",
+            run_id="test-run",
+            num_seeds=3,
+        )
+        seeds = [call.kwargs["seed"] for call in run_solver_mock.call_args_list]
+        assert seeds == [1, 2, 3]
+
+    def test_single_seed_passes_no_seed_override(self, problems_yaml, mocker):
+        # Backward compatibility: the default `num_seeds=1` must not
+        # override the configuration's own fixed seed.
+        run_solver_mock = mocker.patch.object(
+            orchestrator, "run_solver", return_value=dict(_FAKE_METRICS)
+        )
+        orchestrator.run_benchmark(
+            problems_yaml, ["highs-default"], year="2025", run_id="test-run"
+        )
+        assert run_solver_mock.call_args.kwargs["seed"] is None
+
+    def test_error_status_stops_further_seeds(self, problems_yaml, tmp_path, mocker):
         error_metrics = {**_FAKE_METRICS, "status": "ER"}
         run_solver_mock = mocker.patch.object(
             orchestrator, "run_solver", return_value=error_metrics
@@ -159,7 +182,7 @@ def test_error_status_stops_further_iterations(
             ["highs-default"],
             year="2025",
             run_id="test-run",
-            iterations=3,
+            num_seeds=3,
         )
         assert run_solver_mock.call_count == 1