test mode: size-realistic fake samples — phase 1: design (D1–D4)

[Jammy2211]

Overview

Design phase (1 of 4) for extending PYAUTO_TEST_MODE bypass runs to produce size-representative outputs: a sample-count knob so samples.csv matches a production Nautilus SLaM stage (N ~ 10k–100k rows) instead of today's 4 fake samples. This makes cold SLaM runs instant while keeping resume-overhead profiling honest (born from the slam-resume-profiling task, autolens_profiling#70). This phase edits no source — the deliverable is four locked design decisions (D1–D4) posted on this issue, which phase 2 implements verbatim.

Plan

• Ground the design against the live bypass path: PYAUTO_TEST_MODE levels in autoconf, _fit_bypass_test_mode → _build_fake_samples in PyAutoFit.
• Fix the parity target: measure a production Nautilus SLaM stage's samples.csv (rows, columns, byte size) against today's 4-sample bypass output.
• Lock and post the four decisions: D1 knob name + accessor placement, D2 vectorized synthesis recipe, D3 downstream contract + the concrete phase-2 test list, D4 representativeness limits (recorded, not solved).
• Exit: D1–D4 on this issue; phase 2 unblocked with zero open design questions.

Detailed implementation plan

Affected Repositories

• PyAutoFit (primary — design target; no edits this phase)
• PyAutoConf (only if D1 places the accessor next to test_mode_level())

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean (claimed by aggregator-sqlite, PR#1376 awaiting merge)
./PyAutoConf	main	clean

Suggested branch: feature/test-mode-representative-samples (family branch, created in phase 2; phase 1 needs none). Phase 2 is blocked until the aggregator-sqlite PyAutoFit claim frees.

Implementation Steps

1. Read autoconf/test_mode.py (test_mode_level() accessor pattern) — decide D1: env var PYAUTO_TEST_MODE_SAMPLES (default 4 = today's behaviour, byte-identical when unset); accessor in PyAutoConf next to test_mode_level() vs a local read in PyAutoFit (the latter drops PyAutoConf from phase 2 entirely).
2. Read autofit/non_linear/search/abstract_search.py — _fit_bypass_test_mode, _build_fake_samples (prior-median best + 3 perturbed), save_samples / save_samples_summary — and the samples CSV writer + Samples.summary() quantile/weight machinery. Design D2: vectorized numpy synthesis at N (deterministic seed and where it lives; parameter scatter around the prior median; monotone plausible log-likelihoods; valid normalized weights; exact samples.csv column layout/dtypes so row count and byte size match production).
3. Measure a production reference from the slam-resume-profiling outputs (autolens_profiling#70): rows, columns, bytes of a real Nautilus SLaM stage samples.csv; same for a bypass-completed stage. Sets the N target range.
4. Enumerate D3 — everything that must keep working on the synthetic set, with the concrete phase-2 test for each (numpy-only unit tests; no JAX): samples.summary() quantiles, search chaining (result.model / result.instance), aggregator scrape, database load, existing 4-sample multi-batch structural guarantees at large N, default-unset back-compat, and a generation-speed check (N=50k in seconds, vectorized).
5. Record D4 — known representativeness limits, recorded not solved: single bypass likelihood call leaves no realistically-sized search-internal state; adapt-image FITS values come from the prior-median model (size right, values wrong); test mode auto-skips latents.
6. Post D1–D4 as issue comments; mark phase 2 unblocked (design-wise).

Key Files

• PyAutoConf/autoconf/test_mode.py — test_mode_level(); candidate home for the new accessor (D1)
• PyAutoFit/autofit/non_linear/search/abstract_search.py — _fit_bypass_test_mode, _build_fake_samples (D2)
• PyAutoFit/autofit/non_linear/samples/ — CSV writer, Samples.summary(), weight handling (D2/D3)
• PyAutoFit/autofit/aggregator/ — scrape path exercised by the D3 contract

Original Prompt

Click to expand starting prompt

Test-mode representative outputs — Phase 1: design

Type: feature
Target: autofit
Repos:

• PyAutoFit
• PyAutoConf
  Difficulty: small
  Autonomy: supervised
  Priority: normal
  Status: formalised

Phase 1 of 4 of draft/feature/autofit/test_mode_representative_outputs_size_realistic.md
(the parent umbrella — read it first; its scoping notes were verified against source
2026-07-16). Design only — no source edits; the deliverable is a set of locked
decisions recorded on this phase's GitHub issue, which phase 2 implements verbatim.

Decisions to lock

• D1 — knob. Confirm PYAUTO_TEST_MODE_SAMPLES=N (env var, default 4 = today's
  behaviour, so nothing changes for existing users/tests) and where it is read:
  accessor next to test_mode_level() in @PyAutoConf autoconf/test_mode.py vs
  reading it locally in @PyAutoFit. Decide whether PyAutoConf is touched at all.
• D2 — synthesis scheme. Exact vectorized recipe _build_fake_samples uses at
  large N: parameter scatter around the prior median (deterministic seed — where does
  the seed live?), monotone plausible log-likelihoods, valid normalized weights, and
  the samples.csv column layout/dtypes so row count and byte size match a production
  Nautilus SLaM stage (N ~ 10k–100k).
• D3 — downstream contract. Enumerate what must keep working on the synthetic set
  and how phase 2 tests each: samples.summary() quantiles, search chaining
  (result.model / result.instance), aggregator scrape, database load, and the
  existing 4-sample multi-batch structural guarantees at large N.
• D4 — representativeness limits. Record (not solve) the known deltas from a
  production resume: single bypass likelihood call leaves no realistically-sized
  search-internal state; adapt-image FITS values come from the prior-median model
  (size right, values wrong); test mode auto-skips latents.

Exit criteria

D1–D4 posted on the issue as locked decisions; phase 2 unblocked with no open design
questions.

[Jammy2211]

Locked design decisions (phase 1 deliverable — phase 2 implements these verbatim)

Grounded against source on main 2026-07-16 and a production reference: slam-resume-profiling hst_fast/source_lp[1] samples.csv = 10,187 rows × 21 cols, 9.07 MB (~890 B/row, fixed-width fields from write_table).

D1 — knob: PYAUTO_TEST_MODE_SAMPLES, accessor in autoconf

• Env var PYAUTO_TEST_MODE_SAMPLES=N, default 4 — unset behaviour is byte-identical to today.
• Accessor test_mode_samples() lives in autoconf/test_mode.py next to test_mode_level(), re-exported through the existing shim autofit/non_linear/test_mode.py (added to its __all__). This follows the repo rule "shared utilities (e.g. test_mode) belong in autoconf" — PyAutoConf IS touched; phase 2 ships the PyAutoConf PR first (dependency direction: autofit → autoconf only).
• N < 4 raises ValueError loudly (no silent clamp): 4 is the floor that preserves the existing multi-batch structural guarantee.

D2 — synthesis: vectorized numpy → the production Sample.from_lists path

• N == 4 (default): the existing literal branch runs unchanged (scales 1.001/0.999/1.002, weights 1/.5/.25/.125) — exact back-compat, no behavioural risk to every existing test-mode user.
• N > 4: vectorized synthesis, then materialization through the identical machinery a real Nautilus run uses (nautilus/search.py:549-562: numpy arrays → .tolist() → Sample.from_lists → SamplesPDF → write_table). Structure, CSV layout (headers = model.joined_paths + log_likelihood, log_prior, log_posterior, weight) and byte-size are then representative by construction; only the parameter/logL/weight generation is new, and it is pure numpy:
  • rng = np.random.default_rng(0) — fixed constant seed; identical outputs across runs/machines, no coupling to the search identifier.
  • Parameters: (N, D) array p * (1 + 1e-3 * G) with G ~ N(0,1), prior-median vector p as row 0 unperturbed (today's zero-handling kept: p == 0 → the scatter term itself).
  • Log-likelihoods: today's rule generalized — logL_i = logL_best − i (best-first, monotone), so max_log_likelihood_sample stays the prior-median sample in both branches (mode 3's logL_best = -1e99 works unchanged).
  • Weights: w_i ∝ exp(−i / (N/10)), normalized to sum 1 — strictly positive, healthy effective sample size (~N/10), and w_min ≈ (10/N)·e⁻¹⁰ ≥ 4.5e-10 for the supported profiling range N ≤ 1e5, safely above the default samples_weight_threshold: 1.0e-10. (Note: the bypass write path applies no pruning anyway — samples_above_weight_threshold_from is only called from updater.py; the bound matters for later loads that re-apply the threshold, cf. PyAutoFit#1375's 2-sample pruning incident.)
  • log_prior = 0 for all samples, as today.
• Side benefit: save_covariance_matrix gets a non-degenerate covariance (today's 4 near-collinear samples are borderline singular).
• No JAX anywhere in this path; no per-sample python loops in the synthesis (the from_lists materialization cost is exactly what production pays at the same N — cost parity, not overhead).

D3 — downstream contract + phase-2 test list (numpy-only unit tests)

1. Back-compat: env unset → _build_fake_samples output equals today's 4 samples exactly (literal expected values).
2. Summary: synthetic set at small N (e.g. 50): samples.summary() finite; median_pdf ≈ prior median within scatter tolerance; errors at σ=1 and σ=3 finite and positive.
3. Chaining: result.model / result.instance from the bypass result reconstruct the prior-median instance (search-chaining contract).
4. Persistence round-trip: save_samples → reload samples.csv: row count N, header layout joined_paths + 4 tail cols, weights normalized/positive/above the D2 floor.
5. Aggregator + database: scrape a bypass output dir at N=1000 → one result, 1000 samples, summary loads (in-library test; it reads autofit's own output only).
6. Scale: functional test at N=50_000 (generation + summary + write completes; no hard timing assert in CI — vectorization is enforced by construction/review, generous timeout only).
7. Validation error: PYAUTO_TEST_MODE_SAMPLES=3 → ValueError.

D4 — representativeness limits (recorded, NOT solved — timing-honest, not science-honest)

• Search-internal state: 1 (mode 2) or 0 (mode 3) likelihood calls leave no realistically-sized search_internal/ checkpoint — resume timings measured this way exclude search-internal restore cost.
• Adapt-image FITS: production size, prior-median values (not a converged fit).
• Latents: auto-skipped whenever test mode is active (autoconf.test_mode.skip_latents) — a production resume includes latent load; the phase-3 recipe must state this delta.
• log_evidence is stubbed to the single logL; evidence arithmetic is size-only, not value-representative.

Phase-2 consequences

• Two PRs, ordered: PyAutoConf (accessor) → PyAutoFit (_build_fake_samples + tests). Family branch feature/test-mode-representative-samples.
• Phase 2 remains blocked until the aggregator-sqlite PyAutoFit claim frees (perf: aggregator result-loading speedups (single-scan listing, cached summaries, faster samples parse) #1376 awaiting merge).

Exit criteria met: no open design questions. Amendments to these decisions before phase 2 starts are welcome on this thread.

[Jammy2211]

Phase 1 complete — D1–D4 locked in the comment above (design validated; no source edits in this phase). Phase 2 (core API: PyAutoConf accessor → PyAutoFit vectorized builder, branch feature/test-mode-representative-samples) will be issued next; it starts once the aggregator-sqlite PyAutoFit claim (#1377) frees. Completion record: PyAutoMind complete/2026/07/test-mode-representative-samples-phase-1-design.md.