Create list of modules which were terminated from Nextflow

[aramp10]

Generate list of ghost modules (module tests which were terminated from Nextflow)

This list contains the modules which were "missed" or absent from the PkgAuto test summary file

This list originates from the .nextflow.log file, these modules have "Process .* terminated with an error exit status"

[milechin]

I tested the script and it seems to work. This is how I completed the test.

1. In file scripts/pkgauto_email.qsub, I commented out the last line (call to email_notif.pl), so no emails will be sent.
2. I created a test modules_cronjob.csv file to be used as the input for the next flow job. For the first 4 jobs, I manually updated the path to be purposely wrong to cause an error. The contents of the test file are below:

module_name,version,module_name_version,module_pkg_dir,module_installer,module_install_date,module_category,module_prereqs,test_path,qsub_options
R,4.3.1,R/4.3.1,/share/pkg.8,ktrn,06/18/23,statistics,,/share/pkg.8/r/4.3.1/BADPATH/test.qsub,
R,4.4.0,R/4.4.0,/share/pkg.8,rcsr,04/30/24,statistics,,/share/pkg.8/r/4.4.0/BADPATH/test.qsub,
R,4.4.3,R/4.4.3,/share/pkg.8,rcsr,02/28/25,statistics,,/share/pkg.8/r/4.4.3/BADPATH/test.qsub,
R,4.5.2,R/4.5.2,/share/pkg.8,Katia (rcsr),11/17/25,statistics,,/share/pkg.8/r/4.5.2/BADPATH/test.qsub,
cellpose,4.0.4,cellpose/4.0.4,/share/pkg.8,bgregor,05/20/25,imaging,,/share/pkg.8/cellpose/4.0.4/test/test.qsub,-pe omp 8
cellprofiler,4.2.6,cellprofiler/4.2.6,/share/pkg.8,mhorn,10/11/23,imaging,,/share/pkg.8/cellprofiler/4.2.6/test/test.qsub,
cellranger,7.2.0,cellranger/7.2.0,/share/pkg.8,aramp10,10/20/23,bioinformatics,bcl2fastq/2.20,/share/pkg.8/cellranger/7.2.0/test/test.qsub,
cellranger,8.0.1,cellranger/8.0.1,/share/pkg.8,milechin,11/25/24,bioinformatics,bcl2fastq/2.20,/share/pkg.8/cellranger/8.0.1/test/test.qsub,
cellranger,9.0.1,cellranger/9.0.1,/share/pkg.8,aramp10,03/06/25,bioinformatics,bcl2fastq/2.20,/share/pkg.8/cellranger/9.0.1/test/test.qsub,

3. In the file scripts/pkgauto_email.qsub, I commented out the find_qsub.py line, so the script will use my modules_cronjob.csv file.
4. Then I ran the script using bash scripts/pkgauto_email.qsub

The script produced a file called /Nextflow_terminated_taskList.txt, and the following was the content:

R/4.3.1
R/4.4.0
R/4.4.3
R/4.5.2

This matches the expected failure of the first 4 entries.

I did find one bug. I reran the nextflow pipeline with a modified input file with different BADPATHS set, the tasks that failed in the previous run also appeared in the output file /Nextflow_terminated_taskList.txt. So the current code does not search the logs for a specific run, but all the runs that are in the .nextflow.log files.

This might not impact the cronjobs, as each run has its own directory, but could cause confusion if the nextflow pipeline was reran after certain tests were fixed, as they may still be listed as terminated, while the nextflow output indicates success.