Skip to content

feat: size-realistic test-mode bypass samples (PYAUTO_TEST_MODE_SAMPLES)#1381

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/test-mode-representative-samples
Jul 17, 2026
Merged

feat: size-realistic test-mode bypass samples (PYAUTO_TEST_MODE_SAMPLES)#1381
Jammy2211 merged 1 commit into
mainfrom
feature/test-mode-representative-samples

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

PYAUTO_TEST_MODE=2/3 bypass runs can now write a samples.csv whose row count and byte size match a production sampler stage: set PYAUTO_TEST_MODE_SAMPLES=N (N ~ 10k–100k). Cold pipeline runs stay seconds-fast while resume/load timings measured against the completed outputs are honest — born from the SLaM resume-profiling task (PyAutoLabs/autolens_profiling#70). Implements the D1–D4 design locked on #1378, verbatim (#1379).

Merge order: requires the paired PyAutoConf PR (new test_mode_samples() accessor) — merge that first.

API Changes

Added-only and default-off: new env var PYAUTO_TEST_MODE_SAMPLES (default 4 — behaviour is byte-identical when unset) with the autoconf accessor re-exported as autofit.non_linear.test_mode.test_mode_samples. The private _build_fake_samples gains a vectorized N > 4 branch; the N == 4 literal branch is untouched. No existing public symbol changed.
See full details below.

Test Plan

  • pytest test_autofit/ — 1493 passed, 1 skipped (full suite, worktree)
  • 7 new unit tests: legacy 4-sample output byte-identical when env unset; ValueError at N=3; structure + fixed-seed determinism at N=50; zero-valued-parameter perturbation; summary()/median_pdf on the synthetic set (weighted-quantile path, pdf_converged is True); write_table round-trip at N=100 with every weight above the 1e-10 samples_weight_threshold; functional N=50,000
  • End-to-end: PYAUTO_TEST_MODE=2 PYAUTO_TEST_MODE_SAMPLES=10000 DynestyStatic fit of af.ex.Gaussian → 1.7 s wall, samples.csv = 10,000 rows / 1.31 MB, min weight 4.5e-8, weights sum to 1.0, result.instance/result.model intact, normal .completed + zip cycle
Full API Changes (for automation & release notes)

Added

  • Env var PYAUTO_TEST_MODE_SAMPLES=N — number of fake samples the test-mode bypass writes (default 4 = historical behaviour; N < 4 raises ValueError).
  • autofit.non_linear.test_mode.test_mode_samples — re-export of the new autoconf accessor (shim __all__ updated).

Changed Behaviour

  • AbstractSearch._build_fake_samples (private): at N > 4 synthesizes samples vectorized (numpy, default_rng(0); prior-median best sample first; monotone logL_i = best − i; normalized exp(−i/(N/10)) weights) and materialises them through the same Sample.from_lists path a production sampler run uses. At N > ~11 the max weight is ~10/N < 0.99, so SamplesPDF.pdf_converged is True and the real weighted-quantile median_pdf path runs (the 4-sample branch keeps its max-likelihood fallback). The bypass write path applies no weight-threshold pruning (samples_above_weight_threshold_from is updater-only); the weight floor ~(10/N)e⁻¹⁰ stays above the 1e-10 threshold for N ≤ 1e5 so threshold-applying loads keep the full set.

Migration

  • None — default-off; unset env is byte-identical to current behaviour.

Generated by the PyAutoLabs agent workflow.

_build_fake_samples now reads test_mode_samples(): unset/4 keeps today's
literal 4-sample branch byte-identical; N > 4 synthesizes vectorized numpy
samples (seed 0, prior-median best first, monotone logL, normalized
exp-decay weights) through the same Sample.from_lists path a production
sampler run uses, so samples.csv row count and byte size are
representative. Design D1-D4 locked on #1378; task #1379.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant