Skip SGE submission for tests with invalid qsub options, and add a self-test suite

[milechin]

Skip SGE submission for tests with invalid qsub options, and add a self-test suite

Summary

Builds on the existing qsub_valid column (from #47) to make the pipeline act on it. Tests whose qsub options are rejected by qsub -w p are no longer submitted to SGE — they are marked FAILED locally instead of being silently dropped when the scheduler rejects the job at submission. Also adds the repo's first automated test suite, covering this end to end.

What changed

Execution — nextflow/pkgtest.nf

• The workflow now branches input rows on qsub_valid:
  • True → runTests (submitted to SGE, unchanged behavior).
  • not True → a new reportInvalid process pinned to executor 'local', so no job is submitted to the scheduler. It records the test as FAILED (exit_code=1, tests_failed=1, job_number=NA) with an explanatory results.txt.
• Both paths emit the same 13-column test_metrics.csv, so the report layout is unchanged and email_notif.pl / the rshiny app are unaffected.
• Rationale: the CSV qsub_options are injected into each job's clusterOptions, so invalid options make SGE reject the job at submission — before any in-process guard inside runTests could run. Handling it outside runTests is the only place that works.

Tests — new tests/ directory (pytest)

• First self-test suite: validates find_qsub.py + pkgtest.nf against a checked-in fixture (a fake module pkgselfcheck/1.0 with one valid and one invalid test.qsub).
• test_invalid_qsub.py asserts that find_qsub.py sets qsub_valid correctly, and that the pipeline PASSES the valid test while FAILING the invalid one via reportInvalid (job_number=NA), never submitting it.
• Two tiers via a --mode option:
  • local (default) — stubs replace module/qsub and the pipeline runs --executor local: no Lmod, no SGE, no module load. Runs in the conda env (tests/environment.yml).
  • integration — real Lmod module, real qsub, and real SGE on an SCC node.
• find_qsub.py is not modified for testing; it runs as a subprocess with a scrubbed env (Lmod's exported module function removed, stubs prepended to PATH).
• Extensible: add a future test as a new tests/test_<feature>.py reusing the run_env fixture in tests/conftest.py.
• tests/environment.yml pins nextflow>=24 and openjdk=17; leaving them unpinned lets conda install a JDK newer than Nextflow supports (Java 11–21), which fails with "cannot find Java or it's a wrong version".

Docs

• README.md gains a Testing section; CLAUDE.md added with architecture notes and the branching behavior; .gitignore excludes the test runtime artifacts.

Testing

• Local tier green: bash tests/run_local.sh -v → 2 passed (Nextflow --executor local, no SGE).
• Integration behavior verified manually on the SCC with gdal2.csv: the invalid gdal/3.1.2 row is reported FAILED/NA with no SGE job submitted, and the valid rows submit and PASS.

How to run the tests

conda env create -f tests/environment.yml   # on the SCC: module load miniconda first
conda activate pkgautotest-selftest
bash tests/run_local.sh -v