pairtools parse dedup cpu issues

[jiangshan529]

Hi, dear all,

I am using pairtools to process some hic experiemental files. I want to set the cores to be used to 96 for parse and dedup command, but actually only 1-2 cores are really been used! Is there a way to really accelarate the processing time? Thanks!

samtools view -@ 96 -h 150.bam | pairtools parse -c hg38.chrXYM.chrom.sizes --add-columns mapq --drop-sam --drop-seq --walks-policy all --nproc-in 96 --nproc-out 96  | pairtools sort --nproc 96 --memory 128G --tmpdir ./ --output 150.pairs.gz

[Phlya]

In your command there is no dedup... But anyway, basically no... The cores you set to 96 here are just for the reading/writing de/compression processes. They don't need so many to be very fast.
For the actual algortithms, they aren't really parallelized. Dedup technically is, but in practice I haven't noticed an increase in performance with multiple cores with the scipy or sklearn backends (and with cython there is no parallelization implemented).

With very large datasets the typical approach if you have access to a large server or a cluster is to split the fastq files into chunk from the beginning and do all steps up until dedup with chunks, this can speed it up a lot. For dedup you have to merge, otherwise duplicates will be missed, and then it just takes time.

You can check out our pipeline that implements the chunking and all other steps: https://github.com/open2c/distiller-nf

[jiangshan529]

In your command there is no dedup... But anyway, basically no... The cores you set to 96 here are just for the reading/writing de/compression processes. They don't need so many to be very fast. For the actual algortithms, they aren't really parallelized. Dedup technically is, but in practice I haven't noticed an increase in performance with multiple cores with the scipy or sklearn backends (and with cython there is no parallelization implemented).

With very large datasets the typical approach if you have access to a large server or a cluster is to split the fastq files into chunk from the beginning and do all steps up until dedup with chunks, this can speed it up a lot. For dedup you have to merge, otherwise duplicates will be missed, and then it just takes time.

You can check out our pipeline that implements the chunking and all other steps: https://github.com/open2c/distiller-nf

Hi, Phyla, I have another problem. After mapping and use pairtools to select UU reads, I got this error:(the bam file is 132Gb and certainly I can see different reads in the file, not fully duplicated)
WARNING:pairtools:Empty of fully duplicated library, can't estimate complexity

[Phlya]

Sorry for the late reply, forgot to get back to it after the winter holidays.

Is the output also empty, or do you just get this warning, but the output is fine? What were the actual commands you used?