Skip to content

Add multiprocessing capability to independent sampler#771

Merged
TomWagg merged 6 commits into
COSMIC-PopSynth:developfrom
TomWagg:fast-sampling
Apr 7, 2026
Merged

Add multiprocessing capability to independent sampler#771
TomWagg merged 6 commits into
COSMIC-PopSynth:developfrom
TomWagg:fast-sampling

Conversation

@TomWagg

@TomWagg TomWagg commented Mar 26, 2026

Copy link
Copy Markdown
Collaborator

I noticed that sampling was getting a bit slow once the number of binaries exceeded 10,000. I realised that COSMIC was doing sampling single-threaded. I don't think we need to do this?

So I've added options for pool and nproc (similar to Evolve) that can be used to parallelise the sampling. For a sample of 1 million binaries I can get a ~14x speedup when using 16 cores on my laptop (2.2s instead of 32s), I think this should approximately scale with the number of cores for large enough samples.

For now I set the minimum chunk size to 1000 binaries so that people don't overdo it on splitting up the sampling.

(Schwimmbad confuses me so feel free to swap that in for the basic multiprocesing pool I used @katiebreivik)

@TomWagg TomWagg self-assigned this Mar 26, 2026
@codecov

codecov Bot commented Mar 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.65%. Comparing base (8772c07) to head (bbb763d).
⚠️ Report is 143 commits behind head on develop.

Files with missing lines Patch % Lines
src/cosmic/sample/sampler/independent.py 0.00% 27 Missing ⚠️
src/cosmic/_version.py 0.00% 1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (0.00%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #771      +/-   ##
===========================================
- Coverage    86.91%   77.65%   -9.26%     
===========================================
  Files           40       50      +10     
  Lines        25542    27579    +2037     
  Branches         0      875     +875     
===========================================
- Hits         22198    21414     -784     
- Misses        3344     5878    +2534     
- Partials         0      287     +287     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TomWagg

TomWagg commented Mar 26, 2026

Copy link
Copy Markdown
Collaborator Author

You can test the speedups with this code :)

from cosmic.sample.initialbinarytable import InitialBinaryTable
from time import time
import numpy as np

final_kstar = [13, 14]

sampling_params = {
    'qmin': 0.0,
    'm2_min': 0.08,
    "binfrac_model": 1.0,
    "SF_start": 12000,
    "SF_duration": 0.0,
    "met": 0.02,
    "sampling_target": "size",
    "trim_extra_samples": False,
    "q_power_law": 0.0,
    "primary_model": "kroupa01",
    "ecc_model": "sana12",
    "porb_model": "sana12"
}

print("Sampling speed test time!!")
print("--------------------------\n")

for size in [10_000, 100_000, 1_000_000]:
    for nproc in [1, 4, 16, 32]:
        print(f"Sampling {size} binaries with nproc={nproc}...")

        start = time()
        ibt, mass_singles, mass_binaries, n_singles_req, n_bin_req = InitialBinaryTable.sampler(
            'independent', final_kstar, final_kstar, size=size, **sampling_params, nproc=nproc
        )
        end = time()
        print(f"    Took {end - start:.2f} seconds")
        print("    End results:")
        print(f"        Number of binaries sampled: {len(ibt)}")
        print(f"        Mass in singles: {mass_singles:.2f} Msun")
        print(f"        Mass in binaries: {mass_binaries:.2f} Msun")
        print(f"        Number of singles required: {n_singles_req}")
        print(f"        Number of binaries required: {n_bin_req}")
        print(f"        Primary mass percentiles: {np.percentile(ibt['mass_1'], [2.5, 50, 97.5])}")
    print()

@katiebreivik katiebreivik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's totally fine to just use the regular multiprocessing here. Thanks @TomWagg !

@TomWagg
TomWagg merged commit f92c855 into COSMIC-PopSynth:develop Apr 7, 2026
4 of 6 checks passed
@TomWagg
TomWagg deleted the fast-sampling branch April 7, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants