Fragile parallel processing

[ezwelty]

Many methods in glimpse benefit from placing large arrays into shared memory to share across parallel processes. We currently accomplish this task using sharedmem, which has worked brilliantly but has some downsides:

• Ease of use over multiprocessing.Pool comes at the cost of not supporting Windows. However, this comment by sharedmem's author suggests that this can be addressed by installing Windows Subsystem for Linux (WSL, https://docs.microsoft.com/en-us/windows/wsl/install-win10).
• Not actively maintained, which could lead to trouble down the road. For example, MapReduce no longer seems to work on Python 3.8 (see MapReduce no longer working on Python 3.8 rainwoodman/sharedmem#24), requiring the slower MapReduceByThread to be used (glimpse.config.backend = sharedmem.MapReduceByThread).

Possible alternatives to consider:

• multiprocessing, which as of Python 3.8 implements SharedMemory (https://docs.python.org/3/library/multiprocessing.shared_memory.html).
• joblib, especially once/if SharedMemory is supported (Feature: Multiprocessing-based Backend using SharedMemory + Pickle 5 for 2-3x Faster IPC joblib/joblib#1094, Python 3.8 SharedMemory as alternative to memmapping during multiprocessing joblib/joblib#915).
• ray, based on the tutorial here: https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1.