Feature/parallelize lp problem generation

[gparadis]

🔥 Parallel Constraint Compilation Overhaul (ws3)

Tag: v1.1.0-dev
Date: 2025-08-06
Author: Gregory Paradis

---

✨ Overview

This release introduces a major performance overhaul of the constraint compilation pipeline in ws3, especially for cflw and cgen constraint sets. Improvements include persistent parallel execution, smart batching, and phase-specific tuning — resulting in significantly faster reformulation for large-scale forest estate models.

---

🚀 Highlights

🧠 Adaptive Auto-Batching

• New auto_batch() function:
  • Uses greedy bin-packing and optional size_fn weighting.
  • Supports per-phase max_batch_factor, defaulting to empirically tuned values.

⚙️ Parallel Reformulation Rewrite

• _cmp_cflw_m1 and _cmp_cgen_m1 restructured to:

  • Parallelize tree analysis and constraint generation phases.
  • Use shared PersistentWorkerPool created in add_problem.
  • Minimize executor spawning and memory overhead.

• Constraint generation (Phase 3) now batched by workload size.

🔄 Execution Details

• Uses fork multiprocessing context for better startup cost on Linux.
• Phase-specific max_batch_factor:
  • Phase 1: mbf = 1 (high-variance tasks)
  • Phase 3: mbf = auto (default ~2–4)
• Serial fast path avoids parallel overhead when workers=1.

---

📊 Performance Benchmarks (TSA40)

Configuration	Wall Time (s)	Cores	Notes
Final Parallel Impl.	189.7	16	New best, tuned
Serial Baseline	~330	1	Reference only
Threads (Phase 3)	~215	16	Rejected, slower
Aggressive Parallel	~205	32	Diminishing returns

---

✅ Validation

• All results are bitwise identical to serial baseline.
• add_problem passes a persistent executor to all compilation phases.
• Logs confirm deterministic behavior and stable memory usage.

---

🧹 Cleanup & Future Work

• Finalize removal of legacy commented code.
• Add optional max_batch_factor override for expert users.
• Develop synthetic stress test models with large cgen constraint sets.
• Explore persistent worker pre-warming for multi-problem workflows.

---

📝 API Notes

• No changes to the public API.
• All changes internal to forest.py reformulation methods.
• Behavior remains fully backward compatible with existing model definitions.