pyCoGAPS gets stuck on Sampling Phase

[pikapika505]

I've been trying to run pyCoGAPS on a dataset with ~10K cells and ~16K genes. I am running it on an hpc server where 12 CPUs are allocated for this job. CoGAPS is distributed, so nSets = 8. Here is the script I use:

if __name__ == "__main__":
    from PyCoGAPS.parameters import *
    from PyCoGAPS.pycogaps_main import CoGAPS
    import scanpy as sc

    dog_adata = sc.read('dog_adataCoGAPS.h5ad')

    params = CoParams(adata=dog_adata) 
    setParams(params, {
        'nIterations': 10000,
        'seed': 42,
        'nPatterns': 20,
        'useSparseOptimization': True,
        'distributed': "genome-wide"
    })

    params.setDistributedParams(nSets=8)

    start = time.time()
    result = CoGAPS(dog_adata, params)
    end = time.time()

    print("TIME:", end - start)
    result.write("dogOS1_cogaps_result20.h5ad")
    #To save as a .csv file, use the following line:
    result.write_csvs(dirname='./cogaps20', skip_data=True, sep=',')

It runs for approximately 10 hours and then doesn't produce anything. The last messages on the Sampling phase look like this:

10000 of 10000, Atoms: 22414(A), 132246(P), ChiSq: 110973088, Time: 09:30:20 / 09:30:21
worker 1 is finished! Time: 09:30:22
10000 of 10000, Atoms: 22292(A), 133690(P), ChiSq: 109608360, Time: 09:34:45 / 09:34:46
worker 1 is finished! Time: 09:34:48
8000 of 10000, Atoms: 23256(A), 133080(P), ChiSq: 115623760, Time: 09:37:37 / 10:49:31
10000 of 10000, Atoms: 22121(A), 132537(P), ChiSq: 109911624, Time: 09:46:17 / 09:46:18
worker 1 is finished! Time: 09:46:20
10000 of 10000, Atoms: 22939(A), 134133(P), ChiSq: 112030752, Time: 09:49:59 / 09:50:00
worker 1 is finished! Time: 09:50:02
9000 of 10000, Atoms: 22402(A), 136417(P), ChiSq: 109871584, Time: 09:50:50 / 10:25:34
9000 of 10000, Atoms: 23230(A), 133072(P), ChiSq: 115623528, Time: 10:09:16 / 10:45:05
10000 of 10000, Atoms: 22432(A), 136955(P), ChiSq: 109873688, Time: 10:21:06 / 10:21:08
worker 1 is finished! Time: 10:21:09
10000 of 10000, Atoms: 23300(A), 133025(P), ChiSq: 115627600, Time: 10:40:29 / 10:40:31
worker 1 is finished! Time: 10:40:31

I ran it twice and it gets stuck exactly there