diff --git a/docs/PAPER_OCP_VALIDATION.md b/docs/PAPER_OCP_VALIDATION.md index 90deadc..29290ff 100644 --- a/docs/PAPER_OCP_VALIDATION.md +++ b/docs/PAPER_OCP_VALIDATION.md @@ -146,6 +146,50 @@ All rows use `nfe=12`, `ncp=3`, `LAGRANGE-RADAU`, the policy initializer, and a | `n_z=5` | optimal | ~6.19 h | Policy 1 -> Policy 2 | | `n_z=20` | optimal/acceptable | ~6.19 h | Policy 1 -> Policy 2 | +## Paper-vs-LyoPRONTO Baseline Comparison + +Issue #30 compares the paper-reference direct transcription against current +LyoPRONTO baselines before adding any LyoPRONTO-facing policy OCP API. The +closest comparable LyoPRONTO case is the `Tsh` benchmark: shelf temperature is +optimized while chamber pressure is fixed. The committed baseline data were +generated with the current SciPy optimizer and quasi-steady Pyomo FD/collocation +optimizers over the `baseline` 3x3 grid: + +```bash +python benchmarks/grid_cli.py generate \ + --task Tsh --scenario baseline \ + --vary product.A1=16,18,20 \ + --vary ht.KC=2.75e-4,3.3e-4,4.0e-4 \ + --methods scipy,fd,colloc \ + --n-elements 24 --n-collocation 3 \ + --out benchmarks/results/baseline_Tsh_3x3_summary.jsonl +``` + +`Pch` data in `benchmarks/results/baseline_Pch_3x3_summary.jsonl` were also +checked as a non-comparable optimizer sanity run. They use chamber pressure as +the manipulated variable with a fixed shelf profile, so they should not be used +as a paper OCP comparison point. Their drying-time ranges were 18.12-23.67 h +for SciPy, 18.15-23.71 h for Pyomo FD, and 17.90-23.37 h for Pyomo collocation. + +| Model/case | Comparable scope | Drying time | Active constraints/control behavior | Temperature behavior | Interpretation | +| --- | --- | --- | --- | --- | --- | +| Paper Problem 1 direct transcription | Shelf-only OCP, fixed chamber pressure, SI/Kelvin moving-boundary model | ~6.19 h | Policy 1 max heat input, then Policy 2 product-temperature tracking near 2.4 h | Max product temperature tracks 243 K after the switch; shelf temperature backs away from 273 K | Useful validation target for direct transcription and policy classification, but not a drop-in LyoPRONTO baseline | +| LyoPRONTO `Tsh` SciPy baseline | Shelf-only quasi-steady optimizer, fixed 0.1 Torr chamber pressure, cm/Torr/degC model | 12.19-14.47 h across the 3x3 grid; mean 13.33 h | Shelf profile optimizer with product-temperature limit as the active path constraint | Bottom product temperature tracks -25 degC at the constraint | Closest current control-space comparison; drying-time gap is dominated by formulation, units, scenario, and constraint differences | +| LyoPRONTO `Tsh` Pyomo FD | Same quasi-steady benchmark as SciPy, finite-difference transcription, 24 elements | 12.40-14.76 h; mean 13.58 h | Same active product-temperature constraint; no warm start in committed grid | Tracks the same -25 degC limit with 98.9% terminal drying target | Matches the SciPy baseline closely enough for LyoPRONTO regression use, but does not exercise paper Policy 3/Policy 2 structure | +| LyoPRONTO `Tsh` Pyomo collocation | Same quasi-steady benchmark as SciPy, collocation with effective mesh parity | 12.00-14.23 h; mean 13.11 h | Same active product-temperature constraint on a collocation mesh | Tracks the same -25 degC limit with small objective shifts versus SciPy | Confirms transcription effects are modest inside the LyoPRONTO model, separate from paper-vs-LyoPRONTO physics differences | +| Paper Problem 2 direct transcription | Shelf-only OCP with an interface-velocity path constraint | ~8.9 h first-pass coarse validation | Policy 3 interface-velocity tracking, then Policy 1 max heat input, then Policy 2 temperature tracking | Product temperature stays below 240 K after the velocity-limited phase | No current LyoPRONTO optimizer is directly comparable because the public quasi-steady APIs do not expose a flux or interface-velocity cap | + +Recommendation: keep the paper-reference model validation-only and adapt the +policy constraints into LyoPRONTO instead of exposing the SI/Kelvin paper model +as a public optimizer. The paper model should remain in +`lyopronto.pyomo_models.paper_ocp` for upstream validation, convergence checks, +and policy-classification tests. The LyoPRONTO-facing work should proceed +through #31, which already converts the adapter requirements into an +implementation issue: add an optional flux/interface-velocity cap in LyoPRONTO +units, reuse the existing quasi-steady Pyomo optimizer infrastructure, return a +rich result object, classify active policies from LyoPRONTO trajectories, and +avoid changing existing SciPy or Pyomo optimizer behavior. + ## Problem 2 First-Pass Tolerances The Problem 2 validation is intentionally coarse at this stage. The paper diff --git a/tests/test_benchmarks.py b/tests/test_benchmarks.py index 744850f..7e96f8a 100644 --- a/tests/test_benchmarks.py +++ b/tests/test_benchmarks.py @@ -278,6 +278,23 @@ def test_benchmark_results_gitignore_blocks_generated_outputs(repo_root): assert allowed_result.returncode == 1, allowed_result.stdout +def test_paper_ocp_notes_cover_issue30_comparison(repo_root): + notes = (repo_root / "docs" / "PAPER_OCP_VALIDATION.md").read_text(encoding="utf-8") + + assert "## Paper-vs-LyoPRONTO Baseline Comparison" in notes + assert ( + "| Model/case | Comparable scope | Drying time | Active constraints/control behavior | Temperature behavior | Interpretation |" + in notes + ) + assert "benchmarks/results/baseline_Tsh_3x3_summary.jsonl" in notes + assert "12.19-14.47 h" in notes + assert "12.40-14.76 h" in notes + assert "12.00-14.23 h" in notes + assert "Recommendation: keep the paper-reference model validation-only" in notes + assert "#31" in notes + assert "optional flux/interface-velocity cap" in notes + + def test_compute_residuals_uses_native_bools_and_checks_product_temperature(): traj = np.array( [