Aggregator profiling harness + result-loading speedups

[Jammy2211]

Overview

The PyAutoFit Aggregator loads modeling results at scale (3000+ strong lenses in Euclid work, building csv/fits/png catalogues) and is too slow, with unclear scaling drivers — is it the number of results, samples per result, or model complexity? This task builds a fast profiling harness in autofit_workspace_test that mocks realistic output folders without running samplers, then uses it to measure and fix the low-hanging performance fruit in autofit/aggregator/. The long-unused sqlite database build path is explicitly a follow-up task, not part of these PRs.

Plan

• Build a synthetic-results generator in autofit_workspace_test that fabricates realistic aggregator output directories (metadata, model, samples csv, summaries, optional images/zips) parameterised by number of results, samples per result, and model dimensionality — thousands of mock results in seconds.
• Add a profiling grid runner that times each loading pathway separately (from_directory scan, iteration, values("samples_summary"), values("samples"), query, a csv_make-style catalogue loop) and emits a table + JSON, so scaling behaviour is explicit.
• Use the harness to identify and fix low-hanging fruit in PyAutoFit's aggregator (candidates already identified — see detailed plan), each change behaviour-preserving and measured before/after.
• Later phase (blocked): extend autolens_workspace_test with lens-specific profiling once viz-render-gallery frees that repo.
• Follow-up task (separate prompt/issue): exercise and assess the sqlite database build/write path (autofit/database/).

Detailed implementation plan

Work Classification

Both (library + workspace).

Worktree root

~/Code/PyAutoLabs-wt/aggregator-profiling/ (created by start_library)

Affected Repositories

• PyAutoFit (primary)
• autofit_workspace_test
• autolens_workspace_test (later phase — currently claimed by viz-render-gallery, not claimed now)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autofit_workspace_test	main	clean
./autolens_workspace_test	main (repo claimed by viz-render-gallery)	clean

Suggested branch: feature/aggregator-profiling

Implementation Steps

Phase A — profiling harness (autofit_workspace_test, new scripts/profiling/aggregator/):

1. mock_results.py — writes synthetic search-output directories the real Aggregator accepts: metadata, .completed, files/model.json, files/samples.csv + files/samples_info.json, files/samples_summary.json, files/search.json, optional latent/, optional dummy image/*.png + files/*.fits, optional .zip packaging. Parameterised by n_results, n_samples, model dimensionality (nested af.Model/af.Collection for realistic path lengths). Build one template result via real af serialization (formats never drift), then stamp N copies with per-result tweaks.
2. profile_aggregator.py — grid runner over (n_results × n_samples × model_dims) timing each stage with time.perf_counter: Aggregator.from_directory (cold + completed_only), iteration/len, values("samples_summary"), values("samples"), values("model"), a query() predicate, and a csv_make-style summary loop (mirroring autolens_workspace/scripts/guides/results/workflow/csv_make.py). Emits printed table + JSON.
3. Ensure mock-output and profiling-result dirs are gitignored; do not add to smoke_tests.txt.

Phase B — speedups (PyAutoFit, each measured before/after on grids like N∈{10,100,1000,3000}, samples∈{100,1k,10k}, dims∈{5,20,50}):

1. autofit/aggregator/aggregator.py::unzip_directory — currently an unconditional full os.walk + zip handling on every from_directory, followed by a second full walk for the scan. Skip zip handling when there are no zips; merge into a single traversal.
2. autofit/aggregator/search_output.py::AbstractSearchOutput.value() — currently materialises all four suffix lists (jsons/pickles/arrays/fits), each doing rglob over files/ and image/ → 8 sweeps per SearchOutput to fetch one file. Resolve the requested name from one cached directory listing.
3. SearchOutput.__init__ — eager metadata open/parse per result at scan time; make lazy or fold into the scan walk.
4. samples_summary path — avoid forcing full model/samples.csv loads where a summary-only read suffices.
5. Unit tests in test_autofit/aggregator/ (numpy-only); full pytest test_autofit/ before ship.

Phase C — later, blocked: lens-level profiling in autolens_workspace_test (ImagingAgg/FitImagingAgg, workflow csv/fits/png makers) once viz-render-gallery ships.

Phase D — follow-up (separate issue): exercise autofit/database/ build/scrape (database/aggregator/scrape.py, session writes) via the harness's mock outputs; fix cheap bugs; written assessment if deep.

Key Files

• PyAutoFit/autofit/aggregator/aggregator.py — from_directory, unzip_directory
• PyAutoFit/autofit/aggregator/search_output.py — AbstractSearchOutput.value, SearchOutput.__init__, _load_samples
• PyAutoFit/autofit/aggregator/file_output.py — FileOutput/JSONOutput load costs
• autofit_workspace_test/scripts/profiling/aggregator/ — new harness package
• autolens_workspace/scripts/guides/results/workflow/csv_make.py — reference for the catalogue-loop profile (read-only)

Verification

• Harness: 100 results × 1k samples grid completes in seconds–minutes; real Aggregator loads mock dirs identically to real output (spot-check values("samples_summary"), samples, query).
• Speedups: pytest test_autofit/ green; before/after harness numbers quoted in the PR; the 3000-result case shows the from_directory + values wins explicitly.

Original Prompt

Click to expand starting prompt

The PyAutoFit Aggregator is used to load and inspection modeling results, including workflow variants
for .fits, .csv and .png files.

It is used heavily in /mnt/c/Users/Jammy/Science/euclid, where it was often loading results for 3000+ strong lenses
and making the catalogue of csv, fits, png files.

There are many variants of how to load results, aggregators may be skipped for small numbers of lenses, whereas
an sqlite database can be built for large sets (albeit I havent done this in a while and it may be buggy).

A good run through of loadin gresults in all the different ways is at autolens_workspace/scripts/guides/results

The goals of this task are the following:

1. Extend autofit_workspace_test and autolens_workspace_test to have the tools required to profile the speed of the aggregator
   in different models, for differnt models and really work out "where does it scale poorly", "when is it too
   slow to do sciecne", "is it model complexity, number of sampels or something else whjich sl;ows it down?". Ideally,
   This will create output folders that mock real results, but it will do so in a fast way which does not run actual
   samplers or do anything that is to slow.

2. Use these profiling tools to begin speeding up the existing Aggregators and other ways of loading reults, basically
   target the low hanging fruit for now.

3. dont work on sqlite database builds on 1 or 2, but then do a follow upw here you use this functionality,
   inevitably encounter and fix bugs for its use, and assess the code which writes directly to the sqlite database,
   which has not been used in a long time and I'm sure is buggy. If its hard to fix dont bother, but its good to work out where its at.

[Jammy2211]

Phases A + B shipped — PRs open

• Library PR (merge first): perf: aggregator result-loading speedups (single-scan listing, cached summaries, faster samples parse) #1376 — aggregator result-loading speedups
• Workspace PR: feat: aggregator profiling harness (mock results + loading-pathway grid) autofit_workspace_test#48 — the profiling harness that measured them

What the profiling found (baseline, mock results, idle machine)

Per-result timings; grid varies one axis at a time around (100 results × 1000 samples × 5-component model):

axis	from_directory	values("samples_summary")	values("samples")	AggregateCSV
3000 results	0.26 ms	11.4 ms	56.8 ms	26.4 ms
10k samples	0.28 ms	11.7 ms	611 ms	25.2 ms
15-component model	0.22 ms	19.9 ms	128 ms	110 ms

• The dominant axis is samples-per-result, through the full samples.csv parse — not the directory scan, which is cheap (~0.2 ms/result even at 3000 results).
• Model complexity multiplies both the samples parse and the AggregateCSV catalogue build.
• Memory scaling limit: values("samples") across 3000 × 1k-sample results was OOM-killed (~6.6 GB RSS) — every SearchOutput caches its Samples, so even generator-style iteration accumulates. The profiler caps that stage at 500 results. Follow-up candidate: an opt-out or weak cache for bulk iteration.
• Summary-only workflows (samples_summary) cost ~10 ms/result, dominated by JSON→object deserialization (from_dict), not file listing — a deeper follow-up candidate, not low-hanging.

What the speedups bought (before → after, per-result)

stage	before	after	Δ
values("samples") @10k samples	611 ms	334 ms	−45%
values("samples") @1000 results	68 ms	45 ms	−34%
values("samples") @15 components	128 ms	93 ms	−28%
AggregateCSV @15 components	110 ms	83 ms	−25%
AggregateCSV @1000 results	29 ms	22 ms	−24%

Plus: repeat Aggregator.from_directory over zipped result trees no longer re-extracts every zip (extraction is skipped when the extracted directory exists), and per-SearchOutput file listing dropped from 8 directory sweeps to 1.

At the Euclid working point (3000 lenses, ~10k samples, complex models) the samples-loading pass drops from roughly half an hour to ~17 minutes single-threaded, and summary-based catalogue workflows (csv/fits/png makers) get the AggregateCSV win.

Remaining scope

• Phase C (blocked): autolens_workspace_test lens-level profiling — repo currently claimed by viz-render-gallery; starts when free.
• Phase D (follow-up, per the original prompt's goal 3): sqlite database build path — will be filed as its own prompt/issue when these PRs merge.
• Heart was RED on 6 pre-existing, unrelated reasons at ship time; acknowledged for PR-open only (recorded in Mind active.md).

[Jammy2211]

Shipped and merged (2026-07-16):

• perf: aggregator result-loading speedups (single-scan listing, cached summaries, faster samples parse) #1376 — aggregator result-loading speedups (merged)
• feat: aggregator profiling harness (mock results + loading-pathway grid) autofit_workspace_test#48 — profiling harness (merged)

Findings, before/after tables and the OOM scaling note are in the comment above. Post-merge cleanup done (worktree removed, branches deleted, Mind completion record written).

Follow-ups get their own prompts/issues:

• Phase C — lens-level profiling in autolens_workspace_test (was blocked by another task's repo claim at ship time): PyAutoMind/draft/feature/autolens_workspace_test/
• Phase D — sqlite database build path (goal 3 of the original prompt): filed via intake next.

Closing.