Batch processing multi-core question

[gcmestre]

Hey,

Thank you for the work done on this lib.

I was testing the batch processing, in the same computer with the same dataset and got the following results when testing different number of cpu settings on windows and on Linux.

Everything is the same only the number of cores in dataset.process_all(settings, verbose=True, cores=1) changes.
My example has 3 chromatograms.

On windows:

• 1 core - 589 s
• 2 cores - 291 s
• 8 cores - 428 s

On Linux:

• 1 core - 146 s
• 2 cores - 135 s
• 8 cores -170 s

I was not expecting the performance to not increase when setting more cores. Moreover when running mocca2 batch processing on Linux and settings CPU=1 I verify that it actually uses more then 1 core always. It splits the work into multiple cores.

So my question is if anyone has any ideia why I'm experiencing this results and why if I set to run in 1 cores the work is actually done is multiple cores.

Thank you for the support.

[ctapobep]

I was not expecting the performance to not increase when setting more cores.

This is a typical picture when doing parallel computing - in many scenarios the compute power doesn't scale with the number of cores e.g. because there are shared parts where all threads need to synchronize on. All in all:

1. Python doesn't fully support threads yet, so MOCCA spawns processes instead. It's a quite heavy operation, and it's repeated multiple times for different steps in the processing. Each time it incurs extra cost.
2. The processes need to communicate the data and the results with the main thread - this means that the data is copied back and forth many times.
3. After every processing steps, the main thread waits for the ALL children to finish. So on each step the performance depends on the slowest step.

So all in all the work isn't as parallelizable as one would hope. Probably would've been to split the work differently give the whole sample to each process. Then there wouldn't be a need for the processes to communicate and block each other much..

Moreover when running mocca2 batch processing on Linux and settings CPU=1 I verify that it actually uses more then 1 core always. It splits the work into multiple cores.

What makes you think that? How do you know it takes more than 1 core? And how many cores do you have?

[gcmestre]

Thanks for the quick response and explanation @ctapobep.

I have 20 cores on my computer. If I monitor the cpu usage with HTOP I notice that the work. I noticed that serveral cpus start working intensively .

On hidle:

While running:

This with the settings as 1 CPU. If I set multiple cpu I have a simillar picture.
Maybe I'm doing something wrong.