Advice - issue taxonomic assignment

[BelenPradoVivar]

Hi MaestSi,

I have been reading all the issues regarding taxonomic assignment errors No. 137. I have been trying to follow some of the suggestions to try to solve them.

1. As a background I have 320 samples to analyze, most of them have around 100 000 to 500 000 reads. I would like to run all the samples in one go together, but It crash due the memory that they need to perform taxonomic assignment. These are the parameters that I have been using in that part (I increased the RAM and cores available in the HPC that where I am running the analysis).

withName:assignTaxonomy{
container = 'maestsi/metontiime:latest'
cpus = { params.assignTaxonomy ? 10 : 1 }
memory = { params.assignTaxonomy ? 200.GB + (1.GB * (task.attempt-1)) : 1.GB }
errorStrategy = { task.exitStatus == 130 ? 'retry' : 'terminate' }
maxRetries = 3
}

2. Also I tried to ran in batch of 20 samples, and set as maxNumReads=50 000, and also 20 000, to check which one is faster. I didn't have luck with 50 000 reads.

3. My concern comes when: if I run in batches, and then I want to merge the OTU tables. I was reading, this could not be possible because will cause a lot of singletons in final table, this is related because the cluster-features-de-novo, will generate a OTU ID, different per each batch of samples. Only I would be able to merge taxonomy, but I want to use the OTU table to do extra analysis.

Please could you give me your advice, what is your recommendation that could solve this issue.

Thanks a lot,
Belen

[MaestSi]

Hi,
I ran something similar to this to merge multiple count and taxonomy tables.

#!/bin/bash

TABLES=$1
TAXONOMIES=$2

TABLES_LIST=$(echo $TABLES | sed 's/,/ --i-tables /g')
TAXONOMIES_LIST=$(echo $TAXONOMIES | sed 's/,/ --i-data /g')

qiime feature-table merge \
--i-tables $TABLES_LIST \
--o-merged-table Merged_counts.qza

qiime feature-table merge-taxa \
--i-data $TAXONOMIES_LIST \
--o-merged-data Merged_taxonomy.qza

qiime taxa barplot \
--i-table Merged_counts.qza  \
--i-taxonomy Merged_taxonomy.qza  \
--o-visualization Merged_counts_taxa-bar-plots.qzv

qiime taxa filter-table \
  --i-table Merged_counts.qza \
  --i-taxonomy Merged_taxonomy.qza \
  --p-exclude Unassigned \
  --o-filtered-table Merged_counts-no-Unassigned.qza

qiime taxa barplot \
--i-table Merged_counts-no-Unassigned.qza  \
--i-taxonomy Merged_taxonomy.qza  \
--o-visualization Merged_counts-no-Unassigned_taxa-bar-plots.qzv

If your final goal is to do extra analyses with taxonomy, then this should work. While, if you need to perform other analyses not related to taxonomy (for example, build a phylogenetic tree), then you should probably consider all your samples at once, and probably also reduce the value for the --clusteringIdentity, but this is not thoroughly tested. In this case, reducing the number of reads per sample may be a good solution, if you don't have more RAM available.

I hope this helps,
SM

[BelenPradoVivar]

Thanks @MaestSi Simone for your reply, and sharing your script to merge taxonomic tables.
One last question please, If reducing the number of maxNumReads= to 10000 or 5000, which implications will bring of setting this values low? I know that will speed the analysis, but are we going to loose maybe important maybe OTUs, setting this values lower? I mean how we can know that this will be enough to check diversity in our samples, and not loosing valuable information.

[MaestSi]

I suggest you to perform the analysis with all samples with a reduced number of reads (say 20k or lower, depending on computional resources). Then, you may pick the most alpha-diverse sample and reanalyse it with the full dataset. Since it's only one sample, the RAM should be enough. You may then compare whether what you are missing with the subsampling process is highly relevant to you or not. I expect the most abundant species will be retained in the subsampling, while you may lose information about rare species.
Let me know if this seems reasonable to you.
Best,
SM