perf: aggregator result-loading speedups (single-scan listing, cached summaries, faster samples parse)#1376
Merged
Merged
Conversation
- from_directory: fold zip extraction into the single scan walk; skip
zips whose extracted directory already exists (repeat opens no longer
re-extract every zip)
- SearchOutput: one directory sweep builds the json/pickle/csv/fits
listings (was 8 rglobs per result via value()); cache
samples_summary/latent_summary/id per instance
- AggregateCSV: cache per-row kwargs lookups; fix Row.dict() evaluating
each column twice
- samples_from_iterator: classify + split csv headers once, not per row
(values("samples") ~30-45% faster)
- tests: from_directory zip semantics, suffix listing, summary caching
Profiled with the new autofit_workspace_test aggregator harness (#1375).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
Collaborator
Author
|
Workspace harness PR: PyAutoLabs/autofit_workspace_test#48 (merge this library PR first). |
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Low-hanging performance fruit in aggregator result loading, identified with a new profiling harness (autofit_workspace_test PR to follow) that fabricates mock result directories at scale without running samplers (#1375). At catalogue scale (3000+ results), loading was dominated by redundant directory sweeps, repeated JSON loads and per-row CSV header parsing.
Measured per-result on mock results (idle machine, before/after JSON artefacts in the harness output):
values("samples"): 611 → 334 ms at 10k samples (−45%); 68 → 45 ms at 1k samples × 1000 results (−34%); 128 → 93 ms on 15-component models (−28%)AggregateCSV: 110 → 83 ms on 15-component models (−25%); 29 → 22 ms at 1000 results (−24%)from_directoryover zipped result trees no longer re-extracts every zipsamples_summary/modelloads unchanged (~10 ms/result — dominated by JSON→object deserialization, a candidate for follow-up work)Also documented while profiling:
values("samples")across 3000 × 1k-sample results OOMs (~6.6 GB RSS) because everySearchOutputcaches itsSamplesinstance — the profiler caps that stage; the scaling limit is left as a known issue for follow-up.API Changes
None — internal changes only. One behaviour note:
Aggregator.from_directoryno longer re-extracts a zip whose extracted directory already exists (delete the directory to force re-extraction), and extraction now happens in the same traversal as the metadata scan.See full details below.
Test Plan
pytest test_autofit/— 1485 passed, 1 skippedtest_autofit/aggregator/test_from_directory.py): zip extract + skip semantics, single-scan suffix listing,samples_summarycachingprofile_*_before_clean.json/profile_*_after_clean.json)Full API Changes (for automation & release notes)
Changed Behaviour
Aggregator.from_directory— zip extraction folded into the scan walk; a zip whose extracted directory already exists is skipped instead of re-extracted every call.SearchOutput.samples_summary/latent_summary/id— now cached per instance (matching the existingmodel/samplescaching).AggregateCSVRow— median-pdf / max-likelihood / latent kwargs cached per row;Row.dict()no longer evaluates each column twice.samples_from_iterator— CSV headers classified and split once instead of per row.Removed
AbstractSearchOutput._outputs/._outputs_in_directory(private helpers) — replaced by the single-scan_outputs_by_suffixcached property; the publicjsons/pickles/arrays/fitsproperties are unchanged.Generated by the PyAutoLabs agent workflow.