xTea gets randomly stuck but doesn't crash

[hcastel]

Hi,

I have a weird issue with xTea, which is it gets stuck and seems to basically stop moving on at one point.
The logs just don't print anything anymore and htop shows that it's barely even running anymore, but still the process stays there for as long as i let it run, without ever properly crashing:

It seems somewhat similar to #106 but I'm using the classic short-reads version of xTea instead.
And, as shown in htop output, it usually stops at the transduction step (even though I believe there were times where it stopped at other steps).
More precisely, seemingly after the bwa index ...all_with_polymerphic_flanks.fa command, as reported by the logs:
HG00268.L1.err.txt and HG00268.L1.out.txt.

The main issue is that this problem, while happening very often, doesn't even seem to be 100% reproductible, as running many times the same BAM files led to some of them working at times ; and on the flip side, BAM files that seemed to work sometimes could not work again at later times (HG00268 is one of those examples).

Steps to maybe reproduce:

• I installed xTea=0.1.9 through conda in its own environment in the same way as the README
• and I'm running it on BAM files from the 1KGP "1000 Genomes 30x on GRCh38" collection downloaded from e.g. ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR324/ERR3240225/HG00268.final.cram
• running xTea's main script in the following way (after conda env activation):

XTEA_DIR="$HOME/.conda/pkgs/xtea-0.1.9-hdfd78af_0/lib"
PROJECT_FOLDER="$HOME/projects/training"

REFERENCE="$PROJECT_FOLDER/data/reference/grch38/GRCh38_full_analysis_set_plus_decoy_hla.fa"
GENE_ANNOTATIONS="$PROJECT_FOLDER/supporting/gencode.v33.chr_patch_hapl_scaff.annotation.gff3"
TE_LIB="$PROJECT_FOLDER/supporting/te_lib/"
SAMPLE_IDS="$PROJECT_FOLDER/supporting/sample_ids.txt"
CRAM_PATHS="$PROJECT_FOLDER/supporting/cram_paths.txt"

WORKING_DIR="$PROJECT_FOLDER/xtea_wd/"
OUTPUT_SCRIPT="submit_jobs.sh"

~/.conda/envs/xtea/bin/xtea \
    --xtea "$XTEA_DIR" \
    -r "$REFERENCE" \
    -g "$GENE_ANNOTATIONS" \
    -l "$TE_LIB" \
    -i "$SAMPLE_IDS" \
    -b "$CRAM_PATHS" -x null \
    -p "$WORKING_DIR" -o "$OUTPUT_SCRIPT" \
    -f 5907 -y 15 -n 16 -m 64 \
    > "logs/xtea/prep.out" \
    2> "logs/xtea/prep.err"

• and then running them individually in each subfolder with a simple sh, e.g. sh xtea.wd/HG00268/L1/run_xTEA_pipeline.sh > logs/xtea/HG00268.L1.out 2> logs/xtea/HG00268.L1.err

[simoncchu]

Hi, thank you for reporting this issue.
I occasionally observed this issue before when running on large cohorts, but not frequently see this. One potential reason could be the "multi-threads" mode within python (it doesn't support multi-threads but mimic). When run in parallel, sometimes the program needs to call "external" methods, like "bwa" to index and re-align the reads. The main thread will distribute jobs and wait until all "threads" return, but if one "thread" hang there, the program will keep on waiting there. I don't know exactly why some thread doesn't return. For your case, could you try with: 1. Instead of using the conda version, could you try the github version? 2. change -n 16 -m 64 to -n 8 -m 32. Please let me know, if this still frequently occur in your case.

[grennfp]

I had a similar issue with getting xTea to finish running consistently on my university's cluster. What seemed to work was changing line 368 of x_clip_disc_filter.py to this: pool = Pool(self.n_jobs,maxtasksperchild=1). It seemed my jobs were getting stuck around that step. Adding that change made it go from ~20% success rate to ~80%. Hope that helps!

[simoncchu]

Thank you @grennfp for sharing this. My understanding is when created workers run external 3rd tools, the "thread" cannot communicate with the external 3rd tools, and if the 3rd tool has large outputs which block the pipe, it could cause the thread hang. I optimized this in the latest versions to limit the 3rd tool outputs which "solved" the issue when I test. Tested on two large HPC clusters and also cloud settings on large cohorts, and all work well in my case. Not sure why add the maxtasksperchild=1 could improve the finish rate in your case. Will think more on this.

[hcastel]

Hi,

Apologize for the late reply, I wanted to try out different things before answering (and I was also working on other projects).

So I tried both of your solutions on one genome (the same throughout all trials):

• varying multithreading and memory parameters, with different values, especially including the ones you gave me
  • that I tried on two different servers on different locations and different architectures ; also through SLURM on one of them
• installing "manually", more or less without conda: because I'm trying to use it on a server I am not a superuser on, I actually still used conda for dependencies (most notably python libs, as done in the README)

and it always stops at the exact same step as before.

Of course, one easy answer would be that I only tried one input CRAM and that this specific file maybe simply doesn't work for some reason (which would still be an issue). But this CRAM (HG00358 from the 1KGP) is actually part of one of the few that I managed to get to work, simply by retrying the same calling many times, so there is at least one time when this file has led to sensible results.

At this point the only thing I can think of that I did not try would be to install everything directly on my computer, as I have full admin rights on it. But I would prefer not to mess with dependency installations outside conda or venv environments, especially regarding python libraries.

Would you have any other suggestions? Let me know if you want more info.

[simoncchu]

@hcastel , for the "installing manually" tests, are you using the latest code from "git clone" (conda installed environment is fine)?

[hcastel]

@simoncchu Yes, I cloned the last version of xTea.
Basically I more or less did for installation:

# For BWA
git clone https://github.com/lh3/bwa.git
cd bwa; make
cd ..

# For samtools
wget https://github.com/samtools/samtools/releases/download/1.23.1/samtools-1.23.1.tar.bz2
tar -xjf samtools-1.23.1.tar.bz2
cd samtools-1.23.1
./configure --prefix=/home/hcastel/projects/training/8-xtea_manual/samtools_install
make
make install
cd ..

# For xTea
git clone https://github.com/parklab/xTea.git
conda create -n xtea-manual python
conda activate xtea-manual
mamba install pysam sortedcontainers scikit-learn numpy pandas
pip install deep-forest

And for running xTea:

conda activate xtea-manual

# Adding BWA and samtools to the PATH
export PATH=/home/hcastel/projects/training/8-xtea_manual/samtools_install/bin:$PATH
export PATH=/home/hcastel/projects/training/8-xtea_manual/bwa:$PATH

sh scripts/xtea_prep.sh

with xTea being run in xtea_prep.sh in this way:

/home/hcastel/projects/training/8-xtea_manual/xTea/bin/xtea \
    --xtea /home/hcastel/projects/training/8-xtea_manual/xTea/xtea \
    [...other params...]