When running the converter in memory prediction mode (-m), it was suggested I use ~136 GB (see below), which is for a spectral-line cube of shape (1, 1010, 4096, 4096). However, I allocated 150 GB (see below) and the job crashed with an OOM error. I then ran it using 232 GB and the SLURM reported 190.27 GB MaxRSS, ~40% higher than predicted.
jcollier@slurm-login:~$ /carta_share/hdf_convert/run_hdf_converter -m blah.fits
APPROXIMATE MEMORY REQUIREMENTS:
Z stats: 0.536871 GB
XYZ stats: 0.0331285 GB
Rotation: 67.78 GB
XY stats: 0.033128 GB
Main dataset: 67.78 GB
Mipmaps: 67.7138 GB
TOTAL: 136.163GB (Rotated dataset and mipmaps are not allocated at the same time.)
srun --mem=150GB --time=30 --cpus-per-task=30 /carta_share/hdf_convert/run_hdf_converter -p -o /carta_share/current/users/jcollier/blah.hdf5 blah.fits
When running the converter in memory prediction mode (
-m), it was suggested I use ~136 GB (see below), which is for a spectral-line cube of shape (1, 1010, 4096, 4096). However, I allocated 150 GB (see below) and the job crashed with an OOM error. I then ran it using 232 GB and the SLURM reported 190.27 GB MaxRSS, ~40% higher than predicted.