Strip inline comments from #$ directives so later qsub options are not dropped

[eastagiletracker]

This PR proposes stripping inline comments from #$ directives in find_qsub.py so that a commented directive no longer silences the resource requests that follow it (issue #49). We include this PR work along with a full history of your repo at https://eastagiletracker.com/projects/316. You can sign in with your GitHub ID to claim ownership of the project.

What changed and why

SccModule.extract_qsub_opts() reads each #$ line from a test.qsub and joins them all into the single qsub_options CSV column, which pkgtest.nf then hands to SGE as one clusterOptions line. As issue #49 records, SGE ignores everything after a # on that line — so once the directives are joined, a comment on an early directive comments out every option after it. In the shapeit/5.1.1 case from that issue, -l avx512 was dropped and the test ran on a node without those instructions.

The fix removes each directive's comment while the directives are still on separate lines, which is the same per-line reading SGE applies to the file itself. A directive line that is entirely a comment now contributes nothing instead of an empty entry. Directives without comments produce byte-identical output, and the existing -j / -P / -N filtering is unchanged, so nothing already working moves.

This is worth doing in the tooling as well as in the guidance already noted on the issue, because the failure is silent and the existing qsub -w p validation cannot catch it: SGE reads test.qsub line by line, where the comment is harmless, so the file validates as good and the corruption appears only after the join. There is a second effect the same # causes — gpu_filter_tests() matches -l gpus anywhere in that joined string, so comment text mentioning GPUs can misroute a test under --gpus-only / --no-gpus.

Reproduction on main at c5a5f00, with a test.qsub holding the two directives quoted in the issue:

$ python3 -c "
import importlib.util
s = importlib.util.spec_from_file_location('fq','scripts/find_qsub.py')
m = importlib.util.module_from_spec(s); s.loader.exec_module(m)
print(repr(m.SccModule.extract_qsub_opts(None,'/tmp/test.qsub')))"
'-pe omp 4 # Use 4 CPUs -l avx512 # Request a node that supports AVX512 and newer CPU instructions'

Feeding that column through the real to_csv_rows() / save_csv() writer and reading it back the way pkgtest.nf does gives the clusterOptions line -P rcstest -N nf_shapeit_5.1.1 -pe omp 4 # Use 4 CPUs -l avx512 # ..., of which SGE parses only -P rcstest -N nf_shapeit_5.1.1 -pe omp 4. With this change the same end-to-end run yields -P rcstest -N nf_shapeit_5.1.1 -pe omp 4 -l avx512.

Verification. tests/test_extract_qsub_opts.py covers the commented case, the uncommented case, the ignored-flag filtering, and a directive commented out in place. Two of the four fail on the current tree (assert '-pe omp 4 # Use 4 CPUs -l avx512 # ...' == '-pe omp 4 -l avx512') and all four pass with the change; the other two are controls that pass in both states, which is what shows existing behavior is untouched. There is no test harness or CI on main today, so the before/after comparison is that file plus python3 -m py_compile scripts/find_qsub.py, both clean before and after — no new failures. The tests are plain pytest functions in a tests/ directory and add no dependency beyond what find_qsub.py already imports, so they sit alongside the suite proposed in #53 rather than competing with it.

#$ directives are left in test.qsub files untouched; only the extracted option string changes.

How this was managed

We imported your issues and pull requests onto a live agile board — 56 stories — and worked this change on the story for issue #49, https://eastagiletracker.com/projects/316/stories/204444, with the rest of the history on the board at https://eastagiletracker.com/projects/316.

If you'd rather not receive contributions like this, reply no-more-prs on this pull request and we won't open any further ones on your repositories.

---

Lawrence W. Sinclair
CEO / East Agile
linkedin.com/in/lwsinclair/
eastagile.com