Skip to content

fix: PYAUTO_TEST_MODE should write to a separate output dir #1291

Description

@Jammy2211

Overview

When a script is smoke-tested under PYAUTO_TEST_MODE and then later run for real, AutoFit silently returns the cached test-mode result ("Fit Already Completed: skipping non-linear search") because the output path is identical between the two runs. The proper run never happens — symptom is every recovered σ on the supposedly-real run being exactly 0.

This was hit on the IC50 EP / graphical scripts. Current workaround is rm -rf output/<path-prefix>/ before the first proper run, which is runtime discipline contributors will forget. The structural fix is to namespace the test-mode output path so smoke and prod outputs are physically co-resident.

Plan

  • Add a small helper in autofit/non_linear/paths/abstract.py that returns "test_mode" when PYAUTO_TEST_MODE is set in the environment, else None.
  • Filter that segment into the path composition at every site that builds an output path from conf.instance.output_path (three sites — see detailed plan).
  • Net effect: prod writes to output/<prefix>/<name>/<id>/, test writes to output/test_mode/<prefix>/<name>/<id>/. Caches can no longer collide.
  • Add a PyAutoFit regression test that runs a search under PYAUTO_TEST_MODE=2, then again with it unset, and asserts the second run re-fits.
  • Validation sweep: /health_check + per-workspace /smoke_test across every workspace, including ic50_workspace where the bug was first hit.
Detailed implementation plan

Affected Repositories

  • PyAutoFit (primary, code change)
  • All workspaces (validation sweep only — no workspace edits): autofit_workspace, autogalaxy_workspace, autolens_workspace, autolens_workspace_test, autogalaxy_workspace_test, euclid_strong_lens_modeling_pipeline, ic50_workspace

Work Classification

Library (PyAutoFit) — single-repo code change with cross-workspace validation. No workspace code edits.

Branch Survey

Repository Current Branch Dirty?
./PyAutoFit main clean
./PyAutoPrompt main clean (new prompt file untracked, expected)

Suggested branch: feature/test-mode-output-path
Worktree root: ~/Code/PyAutoLabs-wt/test-mode-output-path/ (created by /start_library)

No conflicts: worktree_check_conflict test-mode-output-path PyAutoFit exits 0. The smoke-test-optimization entry in active.md touches PyAutoFit test_mode skip-guards but not the paths module, so the surface area doesn't overlap.

Implementation Steps

  1. Add the helper in autofit/non_linear/paths/abstract.py, near the top of the module:
    def _test_mode_segment() -> Optional[str]:
        """Returns 'test_mode' when PYAUTO_TEST_MODE is set, else None."""
        return "test_mode" if os.environ.get("PYAUTO_TEST_MODE") else None
  2. Patch the three composition sites to filter the segment in right after conf.instance.output_path:
    • autofit/non_linear/paths/abstract.py:239output_path property on AbstractPaths.
    • autofit/non_linear/paths/directory.py:533_make_path() on DirectoryPaths. (Note: this site already omits unique_tag that the property includes — pre-existing inconsistency, leave untouched.)
    • autofit/database/__init__.py:60 — read context first; patch only if it represents a search-output dir.
  3. Confirm SubDirectoryPaths is covered for free — its _output_path delegates to self.parent.output_path, so the parent fix propagates. Verify by reading sub_directory_paths.py:31-79,123-128.
  4. Add regression test in test_autofit/non_linear/paths/test_paths.py:
    • Construct DirectoryPaths(name=..., path_prefix=...) with PYAUTO_TEST_MODE=2 set in the env, assert the resolved output_path contains test_mode as a path component.
    • Repeat with the env var unset, assert it does not.
    • Use monkeypatch.setenv / monkeypatch.delenv so the test is isolated.
  5. Document the new path layout in the relevant docstring (the AbstractPaths.__init__ docstring already includes a worked example — add a one-paragraph note that PYAUTO_TEST_MODE prepends test_mode/).
  6. Run PyAutoFit unit suite — must be green.
  7. Cross-workspace validation sweep:
    • /health_check (fast-forward all repos, run library unit tests + workspace smoke).
    • Targeted /smoke_test per workspace: autofit_workspace, autogalaxy_workspace, autolens_workspace, autolens_workspace_test, autogalaxy_workspace_test, euclid_strong_lens_modeling_pipeline, ic50_workspace.
  8. Manual cross-check on one or two scripts that already have output in output/: run with PYAUTO_TEST_MODE=2, confirm the artefact lands under output/test_mode/..., then run unset and confirm the second invocation actually runs the sampler (no "Fit Already Completed").

Key Files

  • PyAutoFit/autofit/non_linear/paths/abstract.py — helper + property patch.
  • PyAutoFit/autofit/non_linear/paths/directory.py_make_path() patch.
  • PyAutoFit/autofit/database/__init__.py — possibly patch, depends on context.
  • PyAutoFit/test_autofit/non_linear/paths/test_paths.py — regression test.

Post-merge follow-up

Update memory feedback_autofit_cache_resume_pyauto_test_mode: the rm -rf workaround is obsolete once this ships.

Original Prompt

Click to expand starting prompt

PYAUTO_TEST_MODE should mutate the AutoFit output path

When a script is smoke-tested under PYAUTO_TEST_MODE and then later run for real, AutoFit silently returns the cached test-mode result ("Fit Already Completed: skipping non-linear search") because the output path is identical between the two runs. The proper run never happens — symptom is every recovered σ on the supposedly-real run being exactly 0.

This was hit on the IC50 EP / graphical scripts and the current workaround is "rm -rf output/<path-prefix>/ before the first proper run", which is runtime discipline that contributors will forget.

What I want

Make PYAUTO_TEST_MODE mutate the output path so smoke and prod outputs are physically co-resident and can't collide. The rm -rf workaround becomes unnecessary.

Where

Inject a test_mode/ segment at the root of the output tree when os.environ.get("PYAUTO_TEST_MODE") is set and truthy:

  • prod: output/<prefix>/<name>/<identifier>/
  • test: output/test_mode/<prefix>/<name>/<identifier>/

I prefer collecting all test-mode outputs under one tree (easy to gitignore or wipe wholesale) rather than splitting at the identifier level.

There are three places that compose paths from conf.instance.output_path — they all need the same treatment, so factor a shared helper rather than patching each site:

  1. PyAutoFit/autofit/non_linear/paths/abstract.py:239output_path property on AbstractPaths.
  2. PyAutoFit/autofit/non_linear/paths/directory.py:533_make_path() on DirectoryPaths, which reconstructs the path manually instead of using the property. (Aside: it omits unique_tag while the property includes it — pre-existing inconsistency, don't fix here.)
  3. PyAutoFit/autofit/database/__init__.py:60 — read the surrounding context before patching; may or may not need the segment depending on whether it represents a search-output dir.

SubDirectoryPaths overrides output_path in sub_directory_paths.py:74,123 but its _output_path delegates to self.parent.output_path, so it inherits the fix for free via the parent.

Suggested helper signature:

def _test_mode_segment() -> Optional[str]:
    return "test_mode" if os.environ.get("PYAUTO_TEST_MODE") else None

Then filter it into the path components at each of the three sites, right after conf.instance.output_path.

Validation

The code change is small; the cost is validation across every workspace, since the risk is a script that secretly relies on smoke and prod sharing the output path.

  • PyAutoFit unit suite green.
  • Add a regression test: run a search under PYAUTO_TEST_MODE=2, then again with PYAUTO_TEST_MODE unset, and assert the second run re-fits (does not hit the is_complete short-circuit).
  • /health_check followed by /smoke_test on every workspace: autofit_workspace, autogalaxy_workspace, autolens_workspace, autolens_workspace_test, autogalaxy_workspace_test, euclid_strong_lens_modeling_pipeline, ic50_workspace. ic50 is where the bug was first hit so it must be in the sweep.
  • Pick one or two scripts that already have output in output/, run them with and without PYAUTO_TEST_MODE, and confirm the two land in distinct directories.

Out of scope

  • Renaming smoke runs script-by-script (the whole point is the per-script rename is unnecessary).
  • Touching the is_complete short-circuit — once paths differ, the existing logic is correct.

Once shipped

Update the feedback_autofit_cache_resume_pyauto_test_mode memory: the rm -rf workaround is obsolete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions