Problem with Parallelization, Jackknife, and High Performance Computing Clusters

[konratp]

Hello,

I am having trouble running the conText() function with a large dataset on the high performance computing cluster at my institution with parallel processing turned on. When I try to run the function, whether on one core or up to sixteen cores, conText() starts, but after the jackknife gets initialized, it does not seem to progress further, and no results are produced even after running the model for multiple days, both with and without parallelizing the process.

When I try running various combinations (jackknife on, parallel processing on; jackknife off, parallel processing on; jackknife on, parallel processing off) on a small (1000 documents) subset of my data, the function does produce results when I run it locally on my machine. When running the same combinations on the HPC cluster, the jackknife on and parallel processing on combination sometimes produces results, but sometimes the function gets stuck. Since this only happens when running the code on the cluster, I suspect that the issue may lie with the way parallelization works on HPC clusters.

I appreciate your support! Please let me know if there's any other information I can provide.

Best,
Konrat

[ArthurSpirling]

@konratp -- I'll keep this one open, but will also drop you a line off repo.

[sofiaaj]

@konratp My suspicion is that this is related to fork-based parallelism on some HPC systems. One workaround that seems to behave more consistently is to use an explicit PSOCK backend instead of registerDoParallel(n):

cl <- parallel::makePSOCKcluster(n)
doParallel::registerDoParallel(cl)

I’ve updated the quick start guide to include this suggestion.