Distributed fails when nSets > cpu_count() in docker #66
Description
The latest version available fails when run in distributed mode in docker if:
- 1. nSets >= mp.cpu_count()-1
- 2. nSets = 1
How to reproduce 1
- run the latest image with 4cpus:
docker run -it --cpus=4 --entrypoint /bin/bash ghcr.io/fertiglab/pycogaps - validate that the Standard config works:
echo "if __name__ == '__main__':
from PyCoGAPS.parameters import *
from PyCoGAPS.pycogaps_main import CoGAPS
import scanpy as sc
modsimpath = 'data/ModSimData.txt'
modsim = sc.read_text(modsimpath)
params = CoParams(path=modsimpath)
params.printParams()
setParams(params, {
'nIterations':10000,
'seed': 42,
'nPatterns': 3
})
params.printParams()
start = time.time()
result = CoGAPS(modsimpath, params)
end = time.time()
print('TIME:', end - start)
result.write('data/dist_modsim.h5ad')" > test.py
python3 test.py
output:
...
GapsResult result object with 25 features and 20 samples
3 patterns were learned
TIME: 0.9117803573608398
- Validate that distributed config works with
nSets < mp.cpu_count():
#see how many cores we have
root@7f3405e40da1:/pycogaps# python3
Python 3.8.18 (default, Sep 20 2023, 11:41:31)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing as mp
>>> mp.cpu_count()
4
run distributed with 2 cores:
echo "if __name__ == '__main__':
from PyCoGAPS.parameters import *
from PyCoGAPS.pycogaps_main import CoGAPS
import scanpy as sc
modsimpath = 'data/ModSimData.txt'
modsim = sc.read_text(modsimpath)
params = CoParams(path=modsimpath)
params.printParams()
setParams(params, {
'nIterations':10000,
'seed': 42,
'nPatterns': 3,
'useSparseOptimization': True,
'distributed': 'genome-wide'
})
params.setDistributedParams(nSets=2)
params.printParams()
start = time.time()
result = CoGAPS(modsimpath, params)
end = time.time()
print('TIME:', end - start)
result.write('data/dist_modsim.h5ad')">test_2nsets.py
python3 test_2nsets.py
output:
...
GapsResult result object with 13 features and 20 samples
2 patterns were learned
Stitching results together...
TIME: 2.979750871658325