Check copy success and report in CSV report for failures.

[milechin]

Add a check of the copy command exit code. If the exit code is not zero, then it is assumed the copy of the test directory had failed and so the test fails. This failure is now recorded in the CSV report. This was not reported before.

Context
Looking at the November nextflow job log file pkgautotest_cronjob.o1168887, it indicates 7 tests failed for the following modules:

grass/7.8.3
hail/0.2.132
snakemake/8.11.3
seadas/8.4.1
uv/0.7.3
rmats2sashimiplot/3.0.0
nodejs/20.12.2

Majority of the failures was because of "permission denied" errors when copying the "test" directory. Since this failure was at the Nextflow level, this failure was not recorded in the CSV output.

The updated Nextflow script now checks the exit code for the copy command. If the exit code is not 0, then the test is marked as "failed" in the CSV report.

[yshen16]

Thank you for the good fix and the good writing of what has been fixed. One question I couldn't help to bring up is that - is this 'permission denied' behavior when copying test/ folder over happened to the previous month batch tests as well, or it is new? in either case, we may ask further why it wasn't detected or why it only happened in this month's run?

[aramp10]

I tested this code with this:

#Step 1 - run find_qsub.py
./scripts/find_qsub.py --dir /share/module.8/ --mod mmsplice,gatk input.csv
# Loading module test info.
# Writing input.csv
#Step 2 - Nextflow pipeline
module load nextflow
# Run it in local mode (not submit jobs):
nextflow ./nextflow/pkgtest.nf --csv_input ./input.csv  --keep_passed false --executor local

But all tasks failed:

executor >  local (4)
[0b/d8630a] runTests (mmsplice/2.4.0) [100%] 4 of 4, failed: 4 �
[26/b3e5e3] NOTE: Process `runTests (gatk/4.6.2.0)` terminated with an error exit status (2) -- Error is ignored
[b8/2da5a9] NOTE: Process `runTests (gatk/4.5.0.0)` terminated with an error exit status (2) -- Error is ignored
[0f/b560e1] NOTE: Process `runTests (gatk/4.4.0.0)` terminated with an error exit status (2) -- Error is ignored
[0b/d8630a] NOTE: Process `runTests (mmsplice/2.4.0)` terminated with an error exit status (2) -- Error is ignored

Example .command.log file:

/projectnb/rcstest/aramp10/PkgAutoTest/work/39/7ae735414bc56a2aca81a778ad9a21/.command.sh: line 89: warning: here-document at line 75 delimited by end-of-file (wanted `EOF')
/projectnb/rcstest/aramp10/PkgAutoTest/work/39/7ae735414bc56a2aca81a778ad9a21/.command.sh: line 90: syntax error: unexpected end of file