Update runtime for JAWS efficiency

[kaijli]

This was branched from v1.0.21 but has been merged with main branch as well as v1.0.14-alpha.1. Most of the testing was done on another branch, but it has been merged in.

Changes made:

• simplified docker image usage from bfoster/img-omics and microbiomedata/jq to microbiomedata/workflowmeta. Only thing of note is that I don't know where the Dockerfile for the workflowmeta image is stored...
• updated bbtools image for short reads from 38.96 to 39.63 using bryce911/bbtools:39.65 container
• separated stats json generation from rqcfilter call because the new image does not contain json tools but the workflowmeta one does.
• added variable for all runtime calls used by JAWS and made callable from short reads WDL or interleave WDL, but not the main rqcfilter.wdl
• added time calls to every task (was used for testing and metrics calculations, don't see a reason to remove)
• for short reads, matched threads and cpu values in rqcfilter calls. bumped value to 32.
• matched interleave WDL and short reads WDL so that the switch over is easier whenever that happens.
• checked and changed order of the qa stats json used in wf automation
• checked through tabbing

Test runs:

• 165414 small test for long reads (just to check json generation)
• 165423 small test for short reads
  • with new bbtools and json update
• 165532 for short reads interleaved (running)
  • just to test the interleaving with larger file, because small file somehow failed?
• 165533 for interleaved WDL (running)
  • 164452 without json updates, just bbtools and cpu bump

[kaijli]

Waiting on JAWS jobs 166719, 166720, and 166721 to test some empty file checking and tahoma usage.

update: tahoma tests failed, not sure where because perlmutter is down and logging into tahoma is a pain

[jasmynp]

@kaijli , we're waiting to pull this request until we hear back re the tahoma failure modes once perlmutter is back up. Thanks. cc @samobermiller

[aclum]

these are java errors, the dtns are up which can be used to see the results. I'll ping the slack channel but please resubmit this test to perlmutter when it is back up.

[samobermiller]

Resubmitted the test files to perlmutter 2/27 11am. Jaws ids: 168329 (rqcfilter wdl - rqcfilter int json),168330 (rqcfilter wdl - rqcfilter paired json), 168332 (interleaved rqcfilter wdl - interleave json). All appear queued.

Update 2/27 3:30pm: tests failed due to issue #65 , switched input files to Neon (non-interleaved) and blanchard (interleaved). Now re-queued.

Update 3/6:

• test 168383 succeeded (interleave_rqcfilter wdl - interleave json)
• test 168381 (rcqfilter wdl - rqcfilter_int json) ran out of memory due to bbtools issue per nmdc-jaws slack message. will rerun once bbtools releases fix in version 39.8
  • new job id: WAITING on release fix
• test 169038 (rqcfilter wdl - rqcfilter_paired json) failed apparently because input files empty (despite same inputs being used for 168383). will try rerunning with different input (already have --no-cache parameter to force rerun)
  • new job id: 169374 submitted 3/6 1pm

[aclum]

After testing we needed the input fastq files to a String variable type because cromwell has issues staging the NEON urls, we believe because of url redirects.

[aclum]

submitted manually
non-interleaved neon url test: 170897
interleaved test: 170895

[samobermiller]

Manually submitted interleaved tests with bbtools upgrade: 170952, 170953

[samobermiller]

tests continuing to fail with the same java memory error from bbtools. reached out to bbtools in the same slack thread, they have the input files and are trying to replicate the issue