research: realistic-uv iterative potential-correction recovery campaign

[Jammy2211]

Overview

Certify iterative δκ recovery (not just mechanics) for the visibility-space potential-correction engine al.pc.IterFitDpsiSrcInterferometer on a realistic uv configuration — the validation tier smoke-scale synthetics structurally cannot provide (their source meshes cannot reach χ²/dof ≈ 1, so corrections absorb source-model error). Bridges toward the Powell et al. 2025 / Vegetti et al. 2026 sensitivity regime. Follows the merged engineering of #623 (parent epic #618); cites Cao et al. 2025 throughout.

Plan

• Phase A — campaign harness (autolens_workspace_developer): potential_correction_campaign/campaign_sdp81_uv.py — simulate an SIE + NFW-subhalo lens through the real SDP.81 ALMA uv coverage already on RAL (/mnt/ral/jnightin/autolens_jax_joss/dataset/interferometer/sdp81*, three visibility tiers), source pixelization sized so the smooth model reaches χ²/dof ≈ 1; run one-shot FitDpsiSrcInterferometer and iterative IterFitDpsiSrcInterferometer (sparse route, arc dpsi_mask, gauge constraints, xp=jnp on GPU); metrics JSON (global + local δκ correlation, peak distance, evidence) + figures. Local validation on a uv subsample before dispatch.
• Phase B — RAL/A100 runs: refresh the RAL PyAuto mirror (today's merges), dispatch detached (nohup+setsid+sentinel per RAL conventions; precision-operator cached to disk), sweep subhalo mass toward the ~10⁶–10⁸ M☉ regime as tiers allow.
• Phase C — results + docs: results writeup on this issue; add the interferometer section to guides/advanced/potential_correction.py (autolens_workspace) using the certified configuration as exemplar; promote thresholds into a wst regression if a CI-affordable config emerges.

Detail

Affected Repositories

• autolens_workspace_developer (primary, phase A/B)
• autolens_workspace (phase C guide section)

Work Classification

Workspace (research campaign)

Suggested branch: feature/potential-correction-uv-campaign
Worktree root: ~/Code/PyAutoLabs-wt/potential-correction-uv-campaign/

Known traps (from #623)

WSL OOM ≥4k random vis at 64×64 — heavy configs on RAL only; uv beyond real-space Nyquist explodes χ²; global δκ correlation is sidelobe-limited — report local-window + peak metrics too; precision-operator computation scales with n_vis × mask (cache to disk); LAPACK-vs-XLA slogdet agreement is rtol≈1e-8 at these condition numbers.

Original Prompt

Prompt file: PyAutoMind/active/potential_correction_realistic_uv_campaign.md (from the user instruction: "do draft/research/autolens/potential_correction_realistic_uv_campaign.md — iterative δκ-recovery certification on a realistic uv configuration (B1938-like, or reusing the SDP.81 coverage already on RAL)... It's a compute campaign... likely with RAL/A100 in the loop.")

[Jammy2211]

Campaign started — configuration pivot (2026-07-17)

SDP.81 unavailable: /mnt/ral/jnightin/autolens_jax_joss/dataset/interferometer/ does not exist on RAL — prep job 330605's install step exited 1 although its done-sentinel fired (a script error-handling flaw; recovery belongs to the jax-joss task's resume notes, autolens_workspace#281). The campaign pivots to ALMA-like earth-rotation-synthesis uv coverage (Gaussian antenna configuration × hour-angle track → dense elliptical-ring sampling with honest sidelobes), which is fully configurable in scale and reproducible — arguably better for a controlled certification than a fixed observed coverage. The local sma.fits real coverage remains available as a cross-check, and SDP.81 can be swapped in later once the jax-joss prep is repaired.

Harness landed (autolens_workspace_developer/potential_correction_campaign/campaign_uv_recovery.py): three tiers (local ~1.1k vis / mid ~10k / full ~70k, RAL A100 with --use-jax), precision-operator disk caching, arc-restricted dpsi_mask, and — the key addition over the smoke tests — an explicit regime gate: the smooth-model source fit's χ²/dof is measured first and must sit near ~1–3 before recovery metrics mean anything (the smoke-scale failure mode was χ²/dof ≈ 4×10³). Metrics: global + local(1″) δκ correlation, peak distance, peak significance, evidence, for both the one-shot and iterative engines. Local-tier validation run in progress.

[Jammy2211]

Local-tier results — the campaign's first science finding (2026-07-17)

Harness validated end-to-end on the local tier (1,224 synthesis visibilities, 72×72 @ 0.08″, regime gate χ²/dof = 3.56) with a δψ-regularization mini-sweep (Matérn52, c ∈ {2e3, 2e4, 2e5, 2e6}, s = 4):

c	one-shot evidence	one-shot corr / local / peak	iterative evidence	iterative corr / local / peak
2e3	−6033	0.83 / 0.72 / 0.14″ (6.2σ)	−6649	0.13 / −0.14 / 1.20″
2e4	−6081	0.76 / 0.80 / 0.09″	−8247	0.04 / −0.16 / 1.38″
2e5	−6346	0.71 / 0.34 / 0.38″	−9541	0.01 / −0.16 / 1.38″
2e6	−7462	0.49 / 0.06 / 1.00″	−9780	0.01 / −0.16 / 1.38″

Findings:

1. One-shot recovery is certified on realistic coverage — best of the entire campaign (corr 0.83, peak 0.14″, 6.2σ) — and its evidence maximum coincides with the best recovery (c=2e3): evidence-driven hyper-parameter selection works as designed.
2. The iterative engine's failure mode is now precisely characterized: it converges fast and cleanly (Marquardt damping — 6 accepted steps, χ²/dof → 0.59) but lands on the same recovery-free solution across a 1000× δψ-regularization range → the over-fitting lives in the source block (the freely re-fit pixelized source absorbs the subhalo perturbation; δψ stays noise-shaped, and correlation metrics are scale-free). This is the classic source/potential degeneracy the Vegetti-school methodology controls by jointly evidence-optimizing both regularizations per iteration.
3. The evidence self-protects in every configuration: the iterative solution is ranked below the one-shot in all four runs — an evidence-driven pipeline would never select it.

Phase B design (RAL + local follow-up): joint (source-coefficient × δψ-coefficient/scale) evidence-sampled optimization around the iterative engine (IterDpsiSrcInvInterferometerAnalysis with a search, or per-iteration re-optimization à la Koopmans/Vegetti — an engine enhancement worth considering), at the mid/full tiers where the added visibilities constrain the source more tightly. Harness pushed to feature/potential-correction-uv-campaign (autolens_workspace_developer).

[Jammy2211]

Engine enhancement validated — per-iteration evidence re-optimization (2026-07-17)

PyAutoLens#628 (pending-release): IterFitDpsiSrcInterferometer(reg_optimize_every=N) — every N accepted LM iterations the regularization strength multipliers (a_src, a_δψ) are re-optimized by maximising the Laplace evidence at the fixed normal equations (one Cholesky per grid candidate ≈ free). This is the Koopmans 2005 / Vegetti & Koopmans 2009 objective control, targeting exactly the characterized failure mode.

Local-tier validation:

iterative configuration	corr	corr_local	peak dist
fixed coefficients (best of 1000× sweep)	0.13	−0.14	1.20″
reg_optimize_every=1	0.41	0.27	0.44″

The source-absorption degeneracy is broken by the evidence control, as the published methodology predicts. Remaining gap to the one-shot (0.83 / 0.14″) is expected to close with (a) more LM iterations + a finer re-optimization grid, and (b) the mid/full visibility tiers — both the RAL leg (--use-jax; the sparse-operator FFTs are already JAX in every run). 439 tests pass; default engine behaviour unchanged.

[Jammy2211]

Mid-tier results + two harness lessons (2026-07-17, session close)

Harness lessons (both fixed + pushed): (1) the regime gate demonstrably predicts iterative failure — at χ²/dof 29 the iterative failed exactly as forecast; noise retuned to 4.0 brings the gate to 3.19. (2) The precision-operator cache must be noise-keyed (TᴴC⁻¹T is noise-dependent); a stale cache silently poisons every evidence in the run.

Mid tier (9,920 vis, gate 3.19), clean run:

engine	corr / local / peak	significance
one-shot	0.84 / 0.68 / 0.34″	9.3σ — campaign's strongest detection; significance scales with data density (6.2σ local → 9.3σ mid)
iterative, reg_optimize_every=2	0.14 / 0.13 / 1.56″	2.6σ — did not transfer

Contrast with local tier where reg_optimize_every=1 lifted the iterative from 0.13 → 0.41 / 0.44″: the re-optimization schedule is the identified sensitivity (every-step control vs every-2-steps; the mid run also used n_iter=10 with re-opt resets consuming budget). Open questions for the next leg: ro=1 at mid tier, larger n_iter, finer re-opt grid, and whether the evidence-optimal per-step trajectory needs damping of the scale jumps (currently ±2 decades per re-opt).

Session state: one-shot certification now stands at two tiers (local + mid, both in-regime, 6.2σ and 9.3σ); iterative certification stands at the local tier only. Next: the ro-schedule study at mid tier, then the full tier + mass sweep on RAL A100 (--use-jax), then Phase C (workspace guide interferometer section from the certified config). Harness improvements all pushed to the campaign branch.

[Jammy2211]

ro-schedule hypothesis refuted at mid tier (2026-07-17)

reg_optimize_every=1 at the mid tier reproduces the ro=2 result exactly (corr 0.136 / peak 1.56″ / evidence −1.336e5 vs −1.335e5): the iterative converges deterministically to the same poor basin regardless of the evidence-control schedule. The local-tier certification (ro=1, corr 0.41) therefore reflects the basin structure at that configuration, not the schedule per se.

Sharpened diagnosis: basin selection from the x = 0 cold start. At mid-tier resolution/noise the LM's early steps commit to a basin where δψ stays small before the corrections can engage; the evidence again self-protects (iterative −1.34e5 ≪ one-shot −5.74e4) but the optimizer never visits the recovering basin.

Queued next experiment — warm start: initialize the iterative state from the certified one-shot solution [s★ | δψ★] rather than zero. The one-shot is certified at both tiers (mid: 0.84 / 0.34″ / 9.3σ), so the LM then starts inside the right basin and its re-trace feedback refines rather than searches — this also mirrors the published pipelines, which linearize around progressively corrected models rather than cold-starting. One-line engine addition (x0= parameter); high probability of closing the mid-tier iterative certification. Then: full tier + mass sweep (RAL A100 --use-jax), Phase C guide section.

[Jammy2211]

MID-TIER ITERATIVE CERTIFICATION ACHIEVED — via a formulation bug the warm start exposed (2026-07-17)

The warm-start experiment did more than close the certification — it exposed and fixed a real formulation bug in the interferometer LM objective (PyAutoLens#629, pending-release):

The bug: the normal-equation objective counted the state's δψ twice — once through the re-traced lens, once as the linear G·δψ model term in χ² and the gradient. From cold starts this is second-order small (why all previous runs converged plausibly); from a warm start at the certified one-shot solution the optimizer's best move under the inconsistent objective is to cancel the correction — the smoking gun was δκ corr −0.83 (strong anti-correlation). Fix: χ² + Gauss–Newton gradient use the source block only (residual d − T·f(ψ₀+δψ)·s, JᵀC⁻¹r = D − F·[s|0]), and the re-optimization's linear δψ output applies as an increment on the lens state. The imaging engine never had this bug (its kernels are residual-based).

Certification (mid tier, 9,920 vis, gate 3.19):

engine	corr / local / peak	significance
one-shot	0.841 / 0.682 / 0.34″	9.3σ
iterative, warm-started, evidence-controlled, fixed objective	0.839 / 0.679 / 0.34″	9.2σ

Statistically identical recovery — the visibility-space iterative engine is now certified at the mid tier. The earlier "source-absorption" and "ro-schedule" observations are re-explained: they were downstream symptoms of the inconsistent objective.

Remaining: full tier (~70k vis) + subhalo-mass sweep on RAL A100 (--use-jax) as the scale/sensitivity leg, and Phase C (workspace guide interferometer section from this certified configuration: warm-start from the one-shot + evidence control is the documented recipe, exactly as published pipelines linearize around progressively corrected models).

[Jammy2211]

Phase C shipped + engine parity (2026-07-17)

• PyAutoLens#630 (merged): x0= warm start added to the imaging iterative engine for parity with the certified interferometer recipe. (To answer the natural question: the double-counting bug was interferometer-only — the imaging engine's kernels are residual-based, so its objective was always consistent; upstream Cao's iterative.py likewise.)
• autolens_workspace#285 (merged): the interferometer section of guides/advanced/potential_correction.py — the certified recipe (sparse-operator route, arc dpsi_mask, one-shot inversion, iterative warm-started from the one-shot with reg_optimize_every=1, regime-gate guidance).

Remaining on this issue: the full tier (~70k vis) + subhalo-mass sweep on RAL A100 (--use-jax) — the scale/sensitivity leg toward the ~10⁶ M☉ regime — and PR-ing the campaign branch (autolens_workspace_developer) once that leg's results land.

[Jammy2211]

Full-tier A100 results — subhalo-mass sweep (RAL job 330633)

Full tier: 70,200 visibilities (earth-rotation synthesis, 40 antennas x 90 hour-angle steps), 128x128 @ 0.05", noise 0.5, certified recipe (--use-jax --reg-optimize 1 --warm-start), sparse (w-tilde) route throughout, JAX x64 on one A100.

subhalo mass	gate chi2/dof	corr	corr (r<1")	peak offset	significance	verdict
1e10 Msun	203.2*	0.66	0.43	0.46"	5.8 sigma	detected
1e9 Msun	1.37	0.52	0.35	0.51"	4.1 sigma	detected
1e8 Msun	0.99	-0.02	-0.35	2.47"	3.0 sigma	non-detection

* The gate fits the smooth model, so its residuals include the subhalo's own imprint — at this depth a 1e10 perturber alone drives chi2/dof to ~200. At 1e9/1e8 the gate sits at ~1, confirming the source model is adequate and the recovery metrics are clean.

Findings

• Sensitivity floor of this configuration is between 1e8 and 1e9 Msun: 1e9 is a clean 4-sigma localization (peak 0.5" from truth), 1e8 yields an uncorrelated 3-sigma field peak 2.5" away — honest noise, not a false positive at the true position.
• Iterative == one-shot at every mass under the certified recipe (warm start lands at the optimum; evidence control keeps it there) — consistent with the mid-tier certification.
• Wall time ~2.5-3.5 min per mass end-to-end (simulate + precision operator + one-shot + 15 LM iterations with per-step evidence re-optimization) at 70k visibilities. The sparse route's mask-pixel scaling means real ALMA-scale visibility counts change this budget only through the one-off precision-operator build.
• One infrastructure fix en route: above the 10k-visibility guard the campaign now selects TransformerNUFFT (job 330629 failed on the DFT guard; fixed in the campaign branch).

Results JSON + dkappa maps under euclid_jump:/mnt/ral/jnightin/pc_campaign/results/full_m*/.