fix(howtolens): should_simulate every dataset load; run under SMALL_DATASETS#39
Merged
Merged
Conversation
…ATASETS Make every HowToLens tutorial run under PYAUTO_SMALL_DATASETS=1 (16x16) so smoke/CI stays fast, instead of relying on a dead unset-workaround. - Migrate all 22 raw `if not dataset_path.exists():` guards to `al.util.dataset.should_simulate(str(dataset_path))` — the drop-in idiom that ALSO deletes + re-simulates stale data under the cap (so on-disk full-res data can't mismatch the 16x16 masks/grids). - Add missing guards: tutorial_0_visualization (simple__no_lens_light) and tutorial_7_fitting's second load (simple__no_lens_light__mass_sis). - Fix tutorial_7_fitting's first-load guard: it pointed at the wrong simulator (no_lens_light__mass_sis) for the `howtolens` dataset, which is produced by tutorial_6_data.py — a latent bug the migration exposed. - Add guard for tutorial_6_lens_modeling's second load (lens_sersic). - Remove the dead `howtolens/` + `guides/` PYAUTO_SMALL_DATASETS-unset overrides from config/build/env_vars.yaml (they matched zero files here). - Regenerate notebooks + navigator catalogue. Verified: every chapter passes on a FRESH dataset/ dir (cloud shard isolation) under SMALL_DATASETS=1 — ch1 9/9, ch2 8/8, ch3 6/6, ch4 all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #38. Per-repo slice 1/3 of the HowTo
should_simulatemigration (Build#155 Phase 2); moves HowToLens onto "PYAUTO_SMALL_DATASETS just works" so smoke/CI stays fast.Why
Investigation (2026-07-21) showed HowToLens already runs fine at 16×16 — the blockers were missing
should_simulateguards (fresh-clone/shardFileNotFoundError) and rawif not path.exists()guards that don't re-simulate stale data under the cap. Thehowtolens/+guides/unset:[PYAUTO_SMALL_DATASETS]overrides matched zero files (dead).Scripts Changed
if not dataset_path.exists():→al.util.dataset.should_simulate(str(dataset_path))across chapters 1–4 + optional (drop-in that alsormtrees + re-simulates under the cap).chapter_1_introduction/tutorial_0_visualization.py— add guard (simple__no_lens_light).chapter_1_introduction/tutorial_7_fitting.py— guard the 2nd load (simple__no_lens_light__mass_sis) and fix the 1st-load guard to run the correct producer (tutorial_6_data.pymakes thehowtolensdataset; it was wrongly pointed atno_lens_light__mass_sis.py— a latent bug the migration exposed).chapter_4_pixelizations/tutorial_6_lens_modeling.py— guard the 2nd load (lens_sersic).config/build/env_vars.yaml— remove the deadhowtolens/+guides/SMALL_DATASETS-unset overrides.workspace_index.json.Verification
Every chapter passes on a fresh
dataset/(cloud-shard isolation) underPYAUTO_SMALL_DATASETS=1: ch1 9/9, ch2 8/8, ch3 6/6, ch4 all green. (tutorial_5_borders+tutorial_10remain no_run/SLOW-parked, unchanged.)Heart RED at ship; opened under the corrective-PR exception scoped to
workspace validation not passing (10 failed, 2026-07-20T15-09-29Z). Merge deferred to human. Slices 2/3 (HowToGalaxy) + 3/3 (HowToFit) follow.