feat: aggregator profiling harness (mock results + loading-pathway grid)#48
Merged
Conversation
- mock_results.py: fabricate mock search-output directories via the real DirectoryPaths machinery (no sampler runs; thousands of results in seconds; optional zip/latent variants; manifest regen skip) - profile_aggregator.py: time each loading pathway over a one-axis grid (n_results x n_samples x model size); table + JSON output; tiny single cell under PYAUTO_TEST_MODE for validation sweeps - measured the before/after for PyAutoFit#1376; baseline findings (samples parse dominates; 3000x1k full-samples load OOMs) on #1375 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New aggregator profiling harness (PyAutoLabs/PyAutoFit#1375): fabricates mock search-output directories through the real
DirectoryPathsmachinery — no sampler runs, thousands of results in seconds — and times each aggregator loading pathway over a one-axis-at-a-time grid (number of results × samples per result × model size), so it is explicit which axis drives poor scaling.Findings from the first baseline (recorded on the issue):
values("samples")dominates and scales linearly with samples-per-result and model size;AggregateCSVscales with model complexity; thefrom_directoryscan itself is cheap; and full-samples loading across 3000 × 1k-sample results OOMs (~6.6 GB RSS) because everySearchOutputcaches itsSamples(the profiler caps that stage at 500 results).These tools produced the before/after numbers for the PyAutoFit speedups PR.
Upstream PR
Scripts Changed
scripts/profiling/__init__.py— new (package marker)scripts/profiling/aggregator/__init__.py— new (package marker)scripts/profiling/aggregator/mock_results.py— new: mock result-set generator (--n-results/--n-samples/--n-gaussians/--zip/--latent); template written via the library's own paths machinery, stamped N times via copytree; manifest-based regeneration skipscripts/profiling/aggregator/profile_aggregator.py— new: grid runner timingfrom_directory, iteration,values("samples_summary"/"model"/"samples"), a metadata query, and anAggregateCSVcatalogue build; prints a table and writes JSON underoutput/profiling_aggregator/results/; shrinks to a single tiny cell underPYAUTO_TEST_MODEso validation sweeps only smoke-check itscripts/profiling/aggregator/README.md— new: one-paragraph usageAll outputs land under
output/(gitignored). Not added tosmoke_tests.txt(profiling tools, not smoke tests).Test Plan
PYAUTO_TEST_MODE=1 python scripts/profiling/aggregator/profile_aggregator.py— tiny cell completes in seconds (the run_all_scripts.sh path)Aggregatorloads the mock sets identically to real output (summary/samples/query spot-checks), including the--zipvariantGenerated by the PyAutoLabs agent workflow.