Hi Steffen,
I just observed a GLASSgo job, which produced a clustal distmat tmp file of 4G. Given a network file system, this dramatically slows down GLASSgo.. The problematic code is this:
https://github.com/lotts/GLASSgo_ACC-Version/blob/2a1ac3068958c44c73ac46926411f9eb62081631/GLASSgo.py#L383
that is the clustal distmat is written to file. Please check if you can provide the fake filename "stdout" instead (hope it works, it is used that way for outfile and is one way in python to write to STDOUT) and set the normal output of the clustal call to nirvana (outfile=/dev/null), since you are not accessing it anyway (stdout of the following line is not used/red anywhere)
https://github.com/lotts/GLASSgo_ACC-Version/blob/2a1ac3068958c44c73ac46926411f9eb62081631/GLASSgo.py#L390
that way, you might boost GLASSgo since you reduce a looot of IO...
if you want to avoid having both clustal and the parsed distmat in memory at the same time (which requires the tmp-file), you might want to skip the convenience wrapper of clustal and call it directly. Given the distmat can be written to stdout, you can generate a call yourself (nothing else is done by the wrapper), which pipes the clustal (distmat) output through gzip. that way, you reduce IO by a factor of 10-20. the gzipped file can be read conveniently in python.
this would reduce IO not as much as via direct clustal2glassgo pipe but would keep the memory consumption lower..
but only a suggestion... 😄
hope that helps,
best,
Martin
( ping @JensGeorg )
Hi Steffen,
I just observed a GLASSgo job, which produced a clustal distmat tmp file of 4G. Given a network file system, this dramatically slows down GLASSgo.. The problematic code is this:
https://github.com/lotts/GLASSgo_ACC-Version/blob/2a1ac3068958c44c73ac46926411f9eb62081631/GLASSgo.py#L383
that is the clustal distmat is written to file. Please check if you can provide the fake filename "stdout" instead (hope it works, it is used that way for outfile and is one way in python to write to STDOUT) and set the normal output of the clustal call to nirvana (
outfile=/dev/null), since you are not accessing it anyway (stdoutof the following line is not used/red anywhere)https://github.com/lotts/GLASSgo_ACC-Version/blob/2a1ac3068958c44c73ac46926411f9eb62081631/GLASSgo.py#L390
that way, you might boost GLASSgo since you reduce a looot of IO...
if you want to avoid having both clustal and the parsed distmat in memory at the same time (which requires the tmp-file), you might want to skip the convenience wrapper of clustal and call it directly. Given the distmat can be written to stdout, you can generate a call yourself (nothing else is done by the wrapper), which pipes the clustal (distmat) output through gzip. that way, you reduce IO by a factor of 10-20. the gzipped file can be read conveniently in python.
this would reduce IO not as much as via direct clustal2glassgo pipe but would keep the memory consumption lower..
but only a suggestion... 😄
hope that helps,
best,
Martin
( ping @JensGeorg )