fix(interferometer): resample non-PD inversion instead of crashing the search

[Jammy2211]

Overview

The interferometer analysis crashes the whole non-linear search when a sampled model produces a non-positive-definite inversion matrix. AnalysisInterferometer.log_likelihood_function returns the fit's figure_of_merit unwrapped, so a numpy.linalg.LinAlgError: Matrix is not positive definite from the Cholesky log-det term (PyAutoArray/autoarray/inversion/inversion/abstract.py:743) propagates through Nautilus's multiprocessing pool and kills the search (~38 s). The imaging analysis already guards this; interferometer does not. Surfaced on the 2026-07-13 release-validation tail (resolves item G, PyAutoHeart#72).

This is NOT jax-0.10.2 numerical drift (the original hypothesis). It is a NumPy/JAX backend divergence: the release profile runs interferometer/model_fit.py with PYAUTO_DISABLE_JAX=1 (NumPy path), where np.linalg.cholesky raises on a non-PD matrix; the JAX path returns NaN, which the fitness resamples — so JAX silently rejects the same pathological point that NumPy crashes on. Reproduced locally on jax 0.10.2: crashes with PYAUTO_DISABLE_JAX=1, completes cleanly without it.

Plan

• Mirror the imaging analysis's NumPy-path guard in the interferometer analysis so a non-PD inversion at a sampled model resamples instead of crashing.
• Split log_likelihood_function on self._use_jax: JAX path returns unwrapped (must stay exception-free for tracing); NumPy path wraps the fit in try/except Exception → raise af.exc.FitException (caught by autofit/non_linear/fitness.py → resample_figure_of_merit).
• Verify whether the point-source analysis has the same missing guard; align only if it shares the unwrapped figure_of_merit path.
• Validate: NumPy-path interferometer/model_fit.py completes (bad point resampled) instead of LinAlgError; JAX path unchanged; PyAutoLens unit suite green.

Detailed implementation plan

Affected Repositories

• PyAutoLens (primary — the fix)
• autolens_workspace_test (repro / validation only; no edit expected)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoLens	main	clean
./autolens_workspace_test	main	clean

Suggested branch: feature/interferometer-analysis-fitexception

Implementation Steps

1. autolens/interferometer/model/analysis.py — log_likelihood_function (~L166–173): replace the direct return (fit… - penalty) with the imaging pattern from autolens/imaging/model/analysis.py:132-144:

   log_likelihood_penalty = self.log_likelihood_penalty_from(instance=instance)
   if self._use_jax:
       return (self.fit_from(instance=instance, preloads=shared).figure_of_merit
               - log_likelihood_penalty)
   try:
       return (self.fit_from(instance=instance, preloads=shared).figure_of_merit
               - log_likelihood_penalty)
   except Exception:
       raise af.exc.FitException

   (af is already imported at analysis.py:22.)
2. autolens/point/model/analysis.py — inspect log_likelihood_function (L91–160); align to the same guard only if it returns an unwrapped NumPy-path figure_of_merit.
3. Add/mirror a unit test asserting a non-PD interferometer inversion raises FitException on the NumPy path, if an imaging analogue exists to mirror.

Testing

• Repro (must now pass): wipe output/build/model_fit/interferometer, then
  PYAUTO_TEST_MODE=0 PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 PYAUTO_DISABLE_JAX=1 JAX_ENABLE_X64=True python scripts/interferometer/model_fit.py → completes instead of LinAlgError.
• JAX path unchanged: same script with no env → still completes.
• Unit: pytest test_autolens/interferometer/ (NumPy-only).
• Trade-off: catching broad Exception → FitException is the accepted sibling (imaging) pattern; real bugs still surface on the unwrapped JAX path.

Key Files

• autolens/interferometer/model/analysis.py — the fix (missing _use_jax split + try/except).
• autolens/imaging/model/analysis.py:132-144 — the reference pattern to mirror.
• PyAutoArray/autoarray/inversion/inversion/abstract.py:743 — where the raw LinAlgError originates (Cholesky log-det of the regularization matrix).
• autofit/non_linear/fitness.py:235,239 — where FitException / NaN become resample_figure_of_merit.

Original Prompt

Click to expand starting prompt

interferometer/model_fit crashes with numpy.linalg.LinAlgError: Matrix is not positive definite on the 2026-07-13 release-validation tail (PyAutoHeart#72), under the release profile (PYAUTO_TEST_MODE=0). Confirmed diagnosis (2026-07-14): the release profile runs the script with PYAUTO_DISABLE_JAX=1 (NumPy path); np.linalg.cholesky raises on a non-PD regularization matrix at a sampled model, whereas the JAX path returns NaN and resamples. The interferometer analysis lacks the NumPy-path try/except → FitException guard that the imaging analysis has (imaging/model/analysis.py:132-144). Fix: mirror the imaging guard in interferometer/model/analysis.py; verify point_source. NOT jax-0.10.2 drift.

[Jammy2211]

⏸ Parked at ship sign-off — --auto supervised (bug cap)

The fix is implemented and validated in the worktree but not committed/pushed. Under the autonomy contract, the bug work-type caps at supervised, so the ship step (commit / push / PR) is a human sign-off gate — this run stops here rather than opening the PR. Answer below to ship (or adjust scope) and I'll run ship_library from the recorded state.

What was done

Diagnosis (overturns the original hypothesis): not jax-0.10.2 drift. The release profile runs interferometer/model_fit.py with PYAUTO_DISABLE_JAX=1 (NumPy path). A non-PD regularization matrix at a sampled model makes np.linalg.cholesky (PyAutoArray/.../abstract.py:743, log-det evidence term) raise LinAlgError, which propagates through Nautilus and kills the search. The JAX path returns NaN → resampled, so it silently rejects the same point. The interferometer analysis simply lacked the NumPy-path guard the imaging analysis already has.

Fix — autolens/interferometer/model/analysis.py, mirroring imaging/model/analysis.py:132-144:

-        return (
-            self.fit_from(instance=instance, preloads=shared).figure_of_merit
-            - log_likelihood_penalty
-        )
+        if self._use_jax:
+            return (
+                self.fit_from(instance=instance, preloads=shared).figure_of_merit
+                - log_likelihood_penalty
+            )
+
+        try:
+            return (
+                self.fit_from(instance=instance, preloads=shared).figure_of_merit
+                - log_likelihood_penalty
+            )
+        except Exception as e:
+            raise af.exc.FitException

The JAX branch is byte-for-byte unchanged (it must stay exception-free for tracing; NaN → resample already handles it). The NumPy branch now raises af.exc.FitException, which autofit/non_linear/fitness.py:235 catches → resample_figure_of_merit. The method docstring already promised this behaviour ("the model is discarded and its likelihood value is passed to the non-linear search in a way that it ignores it").

Validation (worktree, jax 0.10.2)

Check	Before fix	After fix
interferometer/model_fit.py, PYAUTO_DISABLE_JAX=1 (release config)	crash at first sampled point — LinAlgError (exit 1)	completes — sampler resamples the bad point, Search complete, exit 0
interferometer/model_fit.py, JAX path (no disable)	completes	completes (unchanged branch)
pytest test_autolens/interferometer/	—	22 passed

Self-review (ship-gate leg 3)

CLEAN — single causal file, 13/4 lines, no unrelated scope; mirrors the established sibling (imaging) pattern; the accepted broad except Exception matches imaging and is guarded to the NumPy path only (real bugs still surface on the unwrapped JAX path).

Sign-off question — scope

autolens/point/model/analysis.py has the same latent gap: log_likelihood_function returns self.fit_from(instance=instance).log_likelihood unwrapped, no _use_jax split, no try/except — and its docstring makes the same discard-on-exception promise. It is not the reported/reproduced bug and uses a different return path (.log_likelihood, no penalty/shared), so I kept this PR narrow.

How should I ship?

• A (recommended) — ship the interferometer fix as-is (narrow, causal, matches how sibling tail bugs stayed tight); file point_source as its own follow-up bug prompt.
• B — expand this PR to also guard point/model/analysis.py in the same style now.
• C — hold; you want to review the diff first.

On A or B I run ship_library (commit → push → open the pending-release PR against PyAutoLens; merge stays yours). This resolves release-validation tail item G (PyAutoHeart#72).

[Jammy2211]

✅ Shipped under the Heart-RED corrective-PR exception — PR #607 (open, not merged)

Human authorization (verbatim, contemporaneous, 2026-07-14):

Authorize the Heart-RED corrective-PR exception for RED reason release validation FAILED (stage integrate) — commit, push, and open one pending-release PR for PyAutoLens#606 whose diff is exactly the two-file analysis guard (interferometer/model/analysis.py + point/model/analysis.py), repairing the interferometer model_fit.py stage-integrate crash (tail item G).

RED reason cleared (contributor): release validation FAILED (stage integrate) — re-read from pyauto-heart readiness at ship time, unchanged from the authorized string.

Causal map: Heart RED release validation FAILED (stage integrate) → #606 diagnosis (NumPy-path unguarded Cholesky log-det crash on a non-PD inversion) → NumPy-path try/except → af.exc.FitException guard mirroring imaging/model/analysis.py:132-144 → commit 904cd64e3 → PR #607.

Diff (exactly as authorized — no mixed scope):

• autolens/interferometer/model/analysis.py (+13/−4)
• autolens/point/model/analysis.py (+7/−1)

Validation:

• NumPy-path interferometer/model_fit.py (PYAUTO_DISABLE_JAX=1, real search) — completes (bad point resampled); was a LinAlgError crash at the first sampled point.
• JAX path unchanged (branch byte-for-byte identical).
• pytest test_autolens/ — 381 passed (incl. 22 interferometer + 58 point).

Scope & limits (AUTONOMY.md): this permitted commit/push/open-one-PR only. No merge, no issue close, no release. This PR clears only the interferometer contributor to the stage-integrate failure; the sibling RED reasons remain and Heart stays RED / release stays blocked until each clears by its own path:

• workspace validation not passing (3 failed, 2026-07-09)
• 58 stale parked script(s)
• PyAutoMind: open PR 14d old
• install verification not run

Recovery sequence (resumes only along this path): a human merges #607 (separate human act) → fresh post-merge wheels are built → release-integration validation is re-run on main → Heart emits a new verdict. Release stays human-required throughout.

[Jammy2211]

🔀 PR #607 merged (squash 5250d80a5) — issue kept open

Merged by explicit human direction (2026-07-14). Per the corrective-red recovery sequence, this issue stays open and release stays blocked until the corrective validation completes:

1. ✅ Human merged the corrective PR (this step).
2. ⏳ Build fresh post-merge wheels on main.
3. ⏳ Re-run release-integration validation.
4. ⏳ Heart emits a new verdict over that fresh evidence.

The RED reason release validation FAILED (stage integrate) is only partially cleared — this fix removes the interferometer model_fit.py contributor (tail item G); the sibling RED reasons remain (workspace validation 3 failed, 58 stale parked scripts, PyAutoMind open PR 14d old, install verification not run), so Heart stays RED. Every release remains human-required.

[Jammy2211]

Corrective validation GREEN — closing

The fresh mode=release validation run on the current main HEADs passed end-to-end:

• Stage 2 TestPyPI rehearsal v2026.7.14.1.dev66001 — all 5 wheels built + uploaded.
• Stage 3 wheel-based real-search integration — 542 passed, 0 failed, 0 timeout, 87 skipped.
• Ingested → release_ready: true; Heart is now YELLOW (no RED reasons remain).

This fix is on the validated main HEADs (commit_shas matched), so its corrective validation is satisfied. Closing.