feat: PYAUTO_SMALL_DATASETS + canonical auto-sim for the cluster path (un-park cluster smoke)

[Jammy2211]

Overview

Completes PyAutoMind feature/cluster/11_small_datasets_cluster.md (final filed item of the cluster home straight). PYAUTO_SMALL_DATASETS support exists in the libraries (PointSolver short-circuit; PyAutoArray 15×15 grid/mask caps + should_simulate delete-and-regenerate) but never reaches the cluster scripts: PyAutoBuild env_vars.yaml has no cluster entries, the scripts use hand-rolled dataset guards instead of al.util.dataset.should_simulate, and cluster/start_here.py + cluster/modeling.py are parked in no_run.yaml ("test mode breaks it"; control-verified >500 s even on main).

Success criterion: un-park both cluster scripts from no_run.yaml with a timed CI-green fast-mode run.

Autonomy: supervised --auto ("continue --auto" 2026-07-09; heart-ack carried).

Plan

1. Diagnose: time cluster/start_here.py / modeling.py solo under PYAUTO_TEST_MODE=2 PYAUTO_SMALL_DATASETS=1 PYAUTO_FAST_PLOTS=1; identify what stays slow (suspects: simulator auto-regeneration incl. PointSolver JIT compile + full-res imaging leg; per-plane visualization).
2. Fix the small-mode path: ensure the simulator honours small-mode end to end (small imaging grid or skipped imaging leg; solver short-circuit already exists), tracing full call paths (the known decorator-mutation footgun).
3. Canonical auto-sim (user direction): migrate the cluster scripts' hand-rolled existence checks to al.util.dataset.should_simulate (mirroring imaging and the in-flight weak twin, PyAutoLens#581); lenstool/data.py keeps download caching but gates the 96 MB mosaic leg off under small-mode.
4. Wire CI: add cluster entries to PyAutoBuild config/build/env_vars.yaml (overrides in yaml, never os.environ mutation); remove the two no_run.yaml lines; evidence run.
5. Smoke-set promotion of the lenstool scripts only if genuinely fast (small curated subset rule).

Affected Repositories

• autolens_workspace (primary; parallel to weak-strong-lensing + the frozen point-pairing-policies guide claim — scripts/cluster/ disjoint from both; navigator catalogue regen will need a rebase by whichever of the three open branches merges last)
• PyAutoBuild (env_vars.yaml + no_run.yaml; unclaimed)
• PyAutoArray / PyAutoLens only if a small-mode gap requires it (none expected)

Branch: feature/cluster-small-datasets · Worktree: ~/Code/PyAutoLabs-wt/cluster-small-datasets/

Original Prompt

Click to expand

$(cat PyAutoMind/feature/cluster/11_small_datasets_cluster.md)

[Jammy2211]

Shipped to PR-open: #250 + PyAutoLabs/PyAutoHands#123 (merge together; both pending-release ✓).

The success criterion is met — both cluster scripts un-parked from no_run.yaml with timed evidence under the exact sweep env (computed via autobuild.env_config.build_env_for_script), from a clean dataset regeneration:

Script	Time
cluster/start_here.py	24 s
cluster/modeling.py	26 s
cluster/lenstool/data.py	2 s (96 MB mosaic leg skipped under small-mode)
cluster/lenstool/modeling.py	175 s (verification in full — 0.068″ parity intact; the 72-param refit now runs only under LENSTOOL_EXAMPLE_RUN_FIT=1, never TEST_MODE, which had put the factor-graph search into reduced-iterations mode → 600 s+ hang)

The diagnosis was better than the prompt feared: the libraries' small-mode hooks (PointSolver fixed-pair short-circuit, grid caps) already do the heavy lifting — the parking simply predated them. Changes are wiring: canonical should_simulate guards (matching imaging + the weak twin via PyAutoLens#581), the lenstool gates, and the two no_run.yaml lines removed. Smoke-set promotion of the lenstool scripts deliberately not done (175 s is fine for the sweep, too heavy for the curated smoke subset — the small-subset rule).

Merge coordination note: three open branches regen the navigator catalogue (this, #248, the weak thread) — whichever merges last regenerates on main.

[Jammy2211]

PRs #250 + PyAutoBuild#123 merged — cluster scripts un-parked with 24s/26s evidence; should_simulate guards live. Closing.