fix: Aggregator.from_directory is not test-mode-aware (interpolate tutorial IndexError)

[Jammy2211]

Overview

Phase 1 of a parked workspace API-drift umbrella (PyAutoMind/draft/bug/workspaces/api_drift_callsite_fixes.md). The autofit_workspace features/interpolate.py tutorial crashes with IndexError: Cannot interpolate: no instances have been added to the interpolator — but only under test mode / CI sweeps, not in real user runs. Reproduced on clean main. Root cause is an asymmetry in Aggregator.from_directory, so the fix lands in the library.

Plan

• interpolate.py:213 (in-memory LinearInterpolator) works fine; the crash is at interpolate.py:265, the aggregator/database section.
• Under PYAUTO_TEST_MODE, searches write results to output/test_mode/<prefix> (via _test_mode_segment() in autofit/non_linear/paths/abstract.py), but Aggregator.from_directory reads the hardcoded non-test path → walks an empty/nonexistent dir → 0 results → empty LinearInterpolator → IndexError: no instances.
• Real (non-test) runs are unaffected — the results are at output/<prefix> where the aggregator looks. This only bites test-mode / full-script sweeps. Aggregator.from_directory is simply not test-mode-aware, unlike the search output paths — that asymmetry is the bug.
• Chosen approach (Option A — library-side, symmetric): make Aggregator.from_directory test-mode-aware. When is_test_mode() is active and the given directory yields no metadata but the test_mode-segment-inserted sibling exists, walk that instead (reusing _test_mode_segment()). Safe for the custom-dir aggregator scripts (result.py, overview_2), which simply won't have a test_mode sibling. Fixes the whole class of aggregator-under-test-mode failures with no workspace-script masking (no injected env-vars / hardcoded test paths in user-facing tutorials).

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary — library fix)
• autofit_workspace (downstream consumer — validation + notebook regen)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autofit_workspace	main	clean

Suggested branch: interpolator-aggregator-test-mode

Implementation Steps

1. PyAutoFit: add test-mode resolution to Aggregator.from_directory (autofit/aggregator/aggregator.py), reusing _test_mode_segment() from autofit/non_linear/paths/abstract.py. Add a unit test under test_autofit/aggregator/ covering the test-mode directory resolution (falls through to the test_mode sibling only when the plain directory has no metadata and the sibling exists). Run pytest test_autofit/aggregator/.
2. autofit_workspace: confirm scripts/features/interpolate.py runs clean under PYAUTO_TEST_MODE=1 with the patched library (no script edits expected under Option A); regenerate the notebook.
3. Ship library-first (ship_library), then the workspace follow-up once API impact is confirmed.

Key Files

• autofit/aggregator/aggregator.py — from_directory (currently os.walks the literal path)
• autofit/non_linear/paths/abstract.py — _test_mode_segment() / is_test_mode() to reuse
• ../autofit_workspace/scripts/features/interpolate.py — the failing consumer (line 265)

Notes / scope corrections from triage

• HowToFit tutorial_5_results_and_samples was paired with this in the original umbrella prompt but does not use the interpolator or Aggregator.from_directory — excluded from this task.
• interpolate.py is not currently in the workspace smoke set; the NEEDS_FIX markers named in the umbrella prompt are no longer literally in the files (curated from an earlier triage sweep).
• Remaining 5 umbrella items (ell_comps/ellipse kwargs, plotter kwarg drift, group/slam PriorException, __hash__ TypeError) stay parked in the umbrella prompt for follow-up tasks.

Original Prompt

Click to expand starting prompt (umbrella — this issue scopes only the interpolator item)

# Workspace API-drift call-site fixes (kwargs / Prior / interpolator / plotter) (parked NEEDS_FIX)

Type: bug
Target: workspaces
Repos: autofit_workspace, autogalaxy_workspace, autolens_workspace, autolens_workspace_test, HowToGalaxy, HowToLens, HowToFit, PyAutoFit
Difficulty: medium | Autonomy: supervised | Priority: normal

A set of independent "script uses stale API after a library change" NEEDS_FIX markers. Each is a
targeted call-site (or small library) fix; some pairs share one bug across repos. Reproduce each on
clean main; remove its NEEDS_FIX marker when green. Split into separate PRs if they touch different libs.

- InstanceInterpolator IndexError (likely a PyAutoFit bug, not just call-site): autofit_workspace features/interpolate + HowToFit chapter_1 tutorial_5 (same family).
- ell_comps kwargs KeyError: autogalaxy_workspace imaging/modeling + HowToGalaxy same.
- ellipse kwargs KeyError 'ellipses.0.centre_0': HowToGalaxy ellipse/modeling.
- plotter kwarg drift plot_grid() unexpected 'plot_grid_lines': HowToGalaxy guides/advanced/over_sampling.
- group/slam PriorException (upper<=lower): autolens_workspace group/slam + HowToLens group/slam.
- __hash__ returns non-int TypeError in linear_light_profile_intensity_dict: autolens_workspace_test database/scrape/general.

First step: triage the interpolator one first (library, 2 repos) — likely the highest-value single fix.

[Jammy2211]

Library PR Created — awaiting human merge

PR: #1402 (feature/interpolator-aggregator-test-mode, label pending-release)

Made Aggregator.from_directory test-mode-aware: when is_test_mode() is active and the first scan finds no search-outputs, it retries once against the test_mode sibling (<dir>.parent/"test_mode"/<dir>.name). Reproduced the IndexError on clean main first — the crash was at interpolate.py:265 (aggregator/DB section), not the in-memory interpolation, and only under test mode.

Autonomous-ship gate

• tests: 58/58 test_autofit/aggregator/ (93 with interpolator/), incl. 2 new from_directory test-mode tests
• smoke: n/a — interpolate.py is not in the smoke set; validated directly under PYAUTO_TEST_MODE=1 (exit 0, 3 outputs loaded, interpolation succeeds)
• review faculty: CLEAN (adversarial pass on the "no-op outside test mode / real-run unchanged" claim — gated by is_test_mode() + .exists(), proven by the off-switch test)
• Heart: RED, human-waived — sole RED reason was an unrelated dirty file (PyAutoLens/paper_jax/paper.md WIP); all YELLOW reasons pre-existing/unrelated

Workspace impact — option (iii), no migration

Behaviour-compatible: all 23 from_directory call-sites across the workspaces keep working unchanged. interpolate.py is fixed by the library alone (no script edit → notebook stays in sync), so no ship_workspace follow-up is needed for this item.

Scope

Phase 1 of the workspaces API-drift umbrella. HowToFit tutorial_5 was paired in the umbrella but does not use the interpolator/aggregator — excluded. The other 5 umbrella items remain parked in PyAutoMind/draft/bug/workspaces/api_drift_callsite_fixes.md for follow-up tasks.

Run ended at PR-open (--auto); merge stays human.

[Jammy2211]

Shipped ✅

#1402 merged 2026-07-21. Aggregator.from_directory is now test-mode-aware; features/interpolate.py runs clean under PYAUTO_TEST_MODE=1. Completion record: PyAutoMind/complete/2026/07/interpolator-aggregator-test-mode.md.

Closing — this was Phase 1 (interpolator) of the workspaces API-drift umbrella; the remaining 5 items are tracked in draft/bug/workspaces/api_drift_callsite_fixes.md and will open as their own follow-up tasks.