MPI unpickling error in surrDAMH lib.

[Paulie14]

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 198, in main
    run_command_line(args)
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "surrDAMH/surrDAMH/process_SAMPLER.py", line 20, in <module>
    no_samplers, problem_path = comm_world.recv(source=MPI.ANY_SOURCE)
  File "mpi4py/MPI/Comm.pyx", line 1438, in mpi4py.MPI.Comm.recv
  File "mpi4py/MPI/msgpickle.pxi", line 341, in mpi4py.MPI.PyMPI_recv
  File "mpi4py/MPI/msgpickle.pxi", line 306, in mpi4py.MPI.PyMPI_recv_match
  File "mpi4py/MPI/msgpickle.pxi", line 152, in mpi4py.MPI.pickle_load
  File "mpi4py/MPI/msgpickle.pxi", line 141, in mpi4py.MPI.cloads
_pickle.UnpicklingError: invalid load key, '\x00'.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 198, in main
    run_command_line(args)
  File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "surrDAMH/surrDAMH/process_SAMPLER.py", line 20, in <module>
    no_samplers, problem_path = comm_world.recv(source=MPI.ANY_SOURCE)
  File "mpi4py/MPI/Comm.pyx", line 1438, in mpi4py.MPI.Comm.recv
  File "mpi4py/MPI/msgpickle.pxi", line 341, in mpi4py.MPI.PyMPI_recv
  File "mpi4py/MPI/msgpickle.pxi", line 306, in mpi4py.MPI.PyMPI_recv_match
  File "mpi4py/MPI/msgpickle.pxi", line 152, in mpi4py.MPI.pickle_load
  File "mpi4py/MPI/msgpickle.pxi", line 141, in mpi4py.MPI.cloads
_pickle.UnpicklingError: invalid load key, '\x00'.
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

[Paulie14]

Observations:

• never happened to SB while running locally
• it seems it never happens while running on a single cluster node
• while being logged in frontend, it can pass several times and then it repeats this error 20x

[Paulie14]

receiving (error):

process_SAMPLER.py:20 >>
no_samplers, problem_path = comm_world.recv(source=MPI.ANY_SOURCE)

sending:

process_SOLVER.py:15 >>
comm_world = MPI.COMM_WORLD
rank_world = comm_world.Get_rank()
size_world = comm_world.Get_size()

no_samplers = rank_world

assert(len(sys.argv) == 3)
problem_path = sys.argv[1]
output_dir = sys.argv[2]

for i in range(size_world):
    if i != rank_world:
        comm_world.send([no_samplers,problem_path],dest=i)

[Paulie14]

https://stackoverflow.com/questions/57936912/how-to-fix-pickle-unpickling-error-caused-by-calls-to-subprocess-popen-in-parall