bug using MPI - Seg Fault

[othman-bouizi]

Hi,

I want to analyze an application written in C or C++ and running in parallel with MPI.
It is complied with GNU 14. Malt is compiled with GNU 14 as well.
The binary is executed on 112 cores, and generates this error (of course the application does not crash without MALT)

MALT: Start memory instrumentation of pluto_gcc - 338820 by library override.
MALT: Resolving symbols with addr2line...
[cw37053:338799:0:339779] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
BFD: BFD version 2.35.2-54.el9 internal error, aborting at ../../bfd/elf.c:9149 in _bfd_elf_get_lineno

BFD: Please report this bug.

However MALT does not crash with 1 MPI and 2 MPI ranks. It crashes with 4 MPI ranks.

Is there a way to intercept the signal 11 and to report it in the gathered data ?

Othman.

[svalat]

Hi Othman,
thanks for reporting.

Question

What version of MALT are you using ?

Question

Did you followed (and didn't changed MPI / compilers after the --prep-mpi phase, otherwise need to redo):

malt --prep-mpi
mpirun malt --mpi ./pluto_gcc

Get backtrace

To get a backtrace, in non MPI mode there is --gdb on malt, but that does not work with MPI, maybe make a wrapper script setting ulimit :

Create a script ulimit-wrapper.sh :

#!/bin/bash
ulimit -c $((1024*1024*10))
"$@"
exit $?

The run mpirun -np 112 ./ulimit-wrapper.sh malt --mpi ./pluto_gcc

Try to skip global variables

It looks to fail in libefl at the exit when dumping the profile, probably to get the virtual memory of the global variables.

Maybe you can also try to run malt with the option :

mpirun -np 112 malt --mpi -o tools:nm=false ./pluto_gcc

So it will probably skip the problematic part.

Maybe try branch NEXT

In case, can you also try the next branch, there was many fixes which will be merged in the coming week but it is still in stabilization of its build system due to fresh new web interface.

[svalat]

In case you can also get stack trace in a more simple way (but probably mixed in MPI outputs) :

Patch manually {PREFIX}/bin/malt at then end of the file by replacing :

-gdb -iex "set environment LD_PRELOAD=${ld_preload}" --args "$@"
+gdb -iex "set environment LD_PRELOAD=${ld_preload}" -ex "run" -ex "bt" -ex "set confirm off" -ex "exit" --args "$@"

Then just add option --gdb to your malt call, I need to make it via an option.

Thanks for reporting your issue.

[svalat]

In case, I make it as an option in to branch next which will be released soon.

mpirun -np 112 malt --gdb-bt --mpi ./pluto_gcc

[othman-bouizi]

Hi Othman, thanks for reporting.

Question

What version of MALT are you using ?

1.4.1

Question

Did you followed (and didn't changed MPI / compilers after the --prep-mpi phase, otherwise need to redo):

Yes, I did it. And I executed malt by invoking ${PREFIX_MALT}/bin/malt, otherwise I get the error: error while loading shared libraries: libunwind.so.8: cannot open shared object file: No such file or directory

Get backtrace

To get a backtrace, in non MPI mode there is --gdb on malt, but that does not work with MPI, maybe make a wrapper script setting ulimit :

Create a script ulimit-wrapper.sh :

#!/bin/bash
ulimit -c $((1024102410))
"$@"
exit $?
The run mpirun -np 112 ./ulimit-wrapper.sh malt --mpi ./pluto_gcc

No coredump file is created. I went further by setting ulimit -c unlimited but the result is the same: no coredump file.

Try to skip global variables

So it will probably skip the problematic part.

Nope :-(

Maybe try branch NEXT

In case, can you also try the next branch, there was many fixes which will be merged in the coming week but it is still in stabilization of its build system due to fresh new web interface.

I used:
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home_nfs/MY_PATH/opt/malt/next/x86_64/gcc/14.3.0 -DLIBUNWIND_PREFIX=/home_nfs/MY_PATH/opt/libunwind/1.8.3/x86_64/gcc/14.3.0 -DLIBUNWIND_INCLUDE_DIR=/home_nfs/MY_PATH/opt/libunwind/1.8.3/x86_64/gcc/14.3.0/include -DLIBUNWIND_LIBRARY=/home_nfs/MY_PATH/opt/libunwind/1.8.3/x86_64/gcc/14.3.0/lib

There is an issue compiling MALT (commit 251402c):

+ export MALT_VITE_ORIG_DIR=/home_nfs/MY_PATH/opt/src/malt/src/webview/client-files
+ MALT_VITE_ORIG_DIR=/home_nfs/MY_PATH/opt/src/malt/src/webview/client-files
+ mkdir -p /home_nfs/MY_PATH/opt/src/malt/build/src/webview/client-files
+ for tmp in .* *.*
+ cp /home_nfs/MY_PATH/opt/src/malt/src/webview/client-files/. /home_nfs/MY_PATH/opt/src/malt/build/src/webview/client-files/.
cp: -r not specified; omitting directory '/home_nfs/MY_PATH/opt/src/malt/src/webview/client-files/.'
make[2]: *** [src/webview/client-files/CMakeFiles/webview-client-all.dir/build.make:74: src/webview/client-files/dist/index.html] Error 1
make[1]: *** [CMakeFiles/Makefile2:1210: src/webview/client-files/CMakeFiles/webview-client-all.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

[svalat]

cp: -r not specified; omitting directory '/home_nfs/MY_PATH/opt/src/malt/src/webview/client-files/.'

Thanks for pointing, this is a mistake. Don't know why I didn't saw it yet ???? Fixed.

[othman-bouizi]

In case you can also get stack trace in a more simple way (but probably mixed in MPI outputs) :

Patch manually {PREFIX}/bin/malt at then end of the file by replacing :

-gdb -iex "set environment LD_PRELOAD=${ld_preload}" --args "$@"
+gdb -iex "set environment LD_PRELOAD=${ld_preload}" -ex "run" -ex "bt" -ex "set confirm off" -ex "exit" --args "$@"
Then just add option --gdb to your malt call, I need to make it via an option.

Thanks for reporting your issue.

It generates json files for the mpi run on 112 ranks, but the output says:

...
MALT: profile dump done : malt-pluto_gcc-2279663.json ...
MALT: profile dump done : malt-pluto_gcc-2279585.json ...
MALT: profile dump done : malt-pluto_gcc-2279666.json ...
MALT: profile dump done : malt-pluto_gcc-2279642.json ...
MALT: profile dump done : malt-pluto_gcc-2279568.json ...
[Inferior 1 (process 2279627) exited normally]
No stack.
[Inferior 1 (process 2279587) exited normally]
No stack.
[Inferior 1 (process 2279571) exited normally]
No stack.
[Inferior 1 (process 2279586) exited normally]
No stack.
[Inferior 1 (process 2279569) exited normally]
No stack.
[Inferior 1 (process 2279595) exited normally]
No stack.
[Inferior 1 (process 2279663) exited normally]
No stack.
...

[svalat]

Hum, OK, no bug when in GDB.....
I think this is probably because there might be some non ELF file mapped which I try to interpret.

Can you try to uncomment in src/libinstrum/profiler/AllocStackProfiler.cpp line 647 (in master or 1.4.1) and line 746 in next.

-			//fprintf(stderr,"MALT: Need to load global variables from %s\n",it->file.c_str());
+			fprintf(stderr,"MALT: Need to load global variables from %s\n",it->file.c_str());

[othman-bouizi]

cp: -r not specified; omitting directory '/home_nfs/MY_PATH/opt/src/malt/src/webview/client-files/.'

Thanks for pointing, this is a mistake. Don't know why I didn't saw it yet ???? Fixed.

With the next version of MALT (commit 27c8396):

Error : At /home_nfs/MY_PATH/opt/src/malt/src/libinstrum/tools/Addr2Line.cpp:96 :
Error : Get error while using addr2line on /SYSV78cd381a to load symbols : Success.
MALT: profile dump done : malt-pluto_gcc-87.json ...
MALT: profile dump done : malt-pluto_gcc-78.json ...

I didn't had a look at the results yet.