Memory issues when running event coincidence on merged LST-1 DL1 files

[aleberti]

Last week Axel reported that he had some jobs for the event coincidence failing with a memory issue (too much memory requested). We found out the problem happened because the event coincidence was run over LST-1 merged DL1 files. In such a case, the number of events is too large when performing the coincidence, and it causes the script to exit with an error similar to:

Checking the event coincidence...
Traceback (most recent call last):
  File "/fefs/aswg/workspace/alessio.berti/miniconda3/envs/magic-lst1/bin/lst1_magic_event_coincidence", line 8, in <module>
    sys.exit(main())
  File "/fefs/aswg/workspace/alessio.berti/miniconda3/envs/magic-lst1/lib/python3.8/site-packages/magicctapipe/scripts/lst1_magic/lst1_magic_event_coincidence.py", line 452, in main
    event_coincidence(
  File "/fefs/aswg/workspace/alessio.berti/miniconda3/envs/magic-lst1/lib/python3.8/site-packages/magicctapipe/scripts/lst1_magic/lst1_magic_event_coincidence.py", line 226, in event_coincidence
    cond_lolim = timestamps_magic.value >= times_lolim[:, np.newaxis].value
numpy.core._exceptions.MemoryError: Unable to allocate 586. GiB for an array with shape (4724405, 133284) and data type bool

Therefore, it would be ideal to check the input file and exit if it is a merged DL1 file.

[jsitarek]

while I guess you can distinguish the merged file by having multiple subrun numbers, I am not sure if this is a proper approach. Maybe it would be better to check the number of events in the file because the actual memory limit should be directly related to that one. The error message might however include the information not to run over merged files

[gabemery]

I have found that it is possible to adapt the code to use only a limited number of lst events at a time. Looping over all events by slice of a pre-defined step.
This could be implemented to limit the maximum memory usage while allowing for full run processing.