design: k×s evaluation/convolution coupling — note + adaptive ground truth

[Jammy2211]

Phase 1 of the k×s coupling series (PyAutoMind/feature/autoarray/oversampling_kxs_coupling.md, split per Feature Agent score 7; mechanism: adaptive evaluation → partial bin to uniform s → convolve at s → bin to image resolution). Design-only: settle the pre-bin placement fork (Convolver-internal vs caller-side util) grounded against the refactored helpers, and extend the brute-force ground truth with an adaptive-k_i reference whose k=1 limit reproduces the existing s=2 numbers exactly. Supervised --auto; parks at design sign-off like the parent series' phase 1.

🤖 Generated with Claude Code

[Jammy2211]

Phase 1 complete — design + adaptive ground truth ready for sign-off

Both deliverables pushed to PyAutoMind (kxs_design.md, kxs_ground_truth.py). No source edits.

Mechanism confirmed numerically: adaptive evaluation (per-pixel k_i·s) → partial bin to a uniform s-resolution fine image → PSF convolve at s → bin s→1. The k=1 limit reduces exactly to the parent s=2 reference values; the adaptive k∈{2,1} reference numbers are pinned (sum 2.785595827470031, ~0.4% from uniform — the finer central integration is doing real work).

Fork decision: (b) caller-side util, segment map cached on OverSampler (mirroring its existing segment_ids machinery), because the refactored Convolver engines take uniform-s sub-block input validated against a cached permutation — feeding them mixed-resolution input would mean dual-length detection and coupling the state to evaluation sizes, against the series' loud-single-contract style. The equality rule relaxes to divisibility (every entry % s == 0, adaptive Array2D now allowed when divisible). Blurring grids stay at s. Mapper rows pre-bin with the same util (k_i² build-cost profiling flagged, as the parent series profiled the ×s² cube).

⏸ Design sign-off (supervised --auto)

• "Approved" → phase 2 (core library) starts from this note verbatim — after PyAutoArray#361 + PyAutoGalaxy#484 merge (it builds on the refactored helpers).
• Amendments → numbered points, note updated first.

🤖 Generated with Claude Code

[Jammy2211]

✅ Design approved ("go --auto"). Phase 2 (core library) starting — the refactor PRs it builds on are merged.

🤖 Generated with Claude Code

[Jammy2211]

Phase 2 (core library) implemented — at the ship sign-off checkpoint

Two local commits (not pushed): PyAutoArray +251/−29, PyAutoGalaxy +70/−6.

Your pixelization question — settled, in code

k×s for pixelizations is exact, by linearity: a fine cell's value is the mean of its k_i² sub-samples, each sub-sample is a weighted sum over source pixels, so the fine-cell mapping-matrix row is exactly the mean of its sub-sample weight rows — no approximation beyond the sampling itself, same status as the existing sub_fraction fold. Better: the k_i·s intermediate is never built — mapping_matrix_from already scatters onto parent rows, so fine-cell segment ids as parents with 1/k_i² fractions produce the k×s matrix directly (Mapper.mapping_matrix_over_sampled_for(s)). Memory is n·s²×n_src, independent of k; the feared k² build-cost risk is struck from the design note (§4b added).

Verification

Leg	Result
Ground truth	library reproduces the phase-1 adaptive k∈{2,1} pinned numbers exactly through the public API; k=1 fast path is an identity object
Mapper	mapping_matrix_over_sampled_for: k=1 == 2b property bit-for-bit; full s²→1 fold == mapping_matrix to 1e-14
End-to-end	FitImaging round trip with adaptive [8,4,2] radial evaluation at s=2: chi² = 0; pixelized fit with adaptive pixelization sizes at s=2: runs, finite evidence
Guards	divisibility raises (int 3 with s=2; adaptive containing 3); the old equality-guard tests updated to the relaxed rule (the only intentional behaviour change)
Suites	PyAutoArray 863 (updated guard test + 4 new) / PyAutoGalaxy 946 / PyAutoLens 336
Sites	three blurred variants, linear-lp override, and convolved_padded_image_2d_from (simulation now inherits adaptive sizes on the padded frame — the simulator adoption in phase 4 needs no further library work)

Ship? On approval: PyAutoArray PR then PyAutoGalaxy PR (pending-release), then phases 3 (workspace tests) and 4 (simulator adoption + docs) continue per the prompt, refactor exercise last.

🤖 Generated with Claude Code

[Jammy2211]

🚢 Phase 2 shipped + phases 3/4 status ("ship it and do phase 3 and 4 --auto"):

Shipped: PyAutoArray#363 → PyAutoGalaxy#486 → autolens_workspace#236 (docs leg: guide + simulator prose updated from uniform-equal to the divisibility rule — my own phase-4 prose from the parent series was about to go stale). All pending-release, merge in that order.

Phase 3 (workspace tests) — queued: autolens_workspace_test is claimed by the live task dpie-lenstool-param; the adaptive-k×s test legs for convolution_over_sampled.py are specified in the prompt and start when that ships.

⏸ Phase 4 fork — executed-simulator adoption needs your call

The survey (the prompt's re-baselining risk, quantified): 76 scripts auto-load the start-here dataset; 127 call Imaging.from_fits. Flipping the executed simulators to s=2 changes the saved dataset, and the fork is what psf.fits becomes:

• (a) Save the fine-resolution psf.fits and update loaders to pass convolve_over_sample_size_lp/_pixelization + psf_pixel_scales — fully consistent examples, but touches up to ~127 call sites (or ~76 scoped to the start-here dataset chain).
• (b) Simulate at s=2 but save the psf binned to image resolution — loaders unchanged, data fidelity improves, but every example then fits s=2 data with s=1 convolution: a real systematic (~1–2% of peak) in 76 examples' residuals. Physically honest (mirrors real undersampled data) but pedagogically noisy.
• (c) Keep executed simulators at s=1 (prose-only adoption, the current state after this docs PR) and revisit adoption when a start-here convolve_over_sample_size modeling example exists to pair with it.

Recommendation: (c) now, (a) as a dedicated follow-up prompt scoped to the start-here chain with its own loader-update budget — (a) done casually inside phase 4 would be the largest-ripple change of the whole series, and (b) quietly degrades 76 examples.

🤖 Generated with Claude Code

[Jammy2211]

✅ Fork resolved: (c) — executed simulators stay at s=1; (a) is filed as a dedicated follow-up prompt scoped to the start-here dataset chain with its own loader-update budget. Merging the phase-2 chain.

🤖 Generated with Claude Code

[Jammy2211]

➕ Follow-up shipped: PyAutoArray#364 — the segment-id cache the design promised (phase 2 rebuilt it per likelihood evaluation; now ×750 on repeat calls, 863→867 suite green incl. 4 k×s tests picked up). Behaviour-preserving.

Series board: option (a) filed (feature/autolens_workspace/oversampled_psf_dataset_adoption.md); phase 3 (workspace tests) queued behind dpie-lenstool-param's claims on autolens_workspace_test and now PyAutoGalaxy; the k×s refactor exercise waits for phase 3's test surface per the prompt's own sequencing. Nothing else is unblocked.

🤖 Generated with Claude Code

[Jammy2211]

✅ #364 merged (main green). Series queue state: phase 3 (workspace tests) blocked by dpie-lenstool-param (autolens_workspace_test); the visible-input pass blocked by cluster-scaling-lenstool (autolens_workspace) — both registered in planned.md and will show ready in /health when those ship. Refactor exercise follows phase 3; full adoption (a) follows the visible-input pass. Nothing in the k×s series is currently unblocked — run ends cleanly at the queue.

🤖 Generated with Claude Code

[Jammy2211]

🚢 Phase 3 shipped: autolens_workspace_test#154 (pending-release) — the claim on autolens_workspace_test freed this morning and the adaptive legs went straight in against the merged mains. Full script output: every parent-series section + adaptive lp round trip (chi²=0; adaptive-vs-uniform evaluation differs by 4.3e-2), adaptive pixelized fit (6.6e-5, finite evidence), divisibility guard, adaptive simulate→fit (1.1e-29). One stale parent guard updated (equality → divisibility).

On merge, the k×s refactor exercise unblocks (needs PyAutoGalaxy free — currently claimed by dpie-lenstool-param). The visible-input pass remains behind weak-modeling's autolens_workspace claim.

🤖 Generated with Claude Code