Updates

2_process_data/README.md

@@ -2,10 +2,8 @@
 Nextflow pipeline to download and process microbial RNA-seq data from NCBI SRA
 
 ## Setup
-1. Install [Nextflow](https://www.nextflow.io/)
-    1. Check that Java 8 or later is installed using: `java -version`
-    1. Download nextflow to your current directory: `curl -s https://get.nextflow.io | bash`
-    1. Test installation by running: `./nextflow run hello`
+1. Create the environment with all the requirements with the `nextflow_environment.yaml` file:
+    1. `conda env create -f nextflow_environment.yaml --name nextflow`
 1. Install [Docker](https://docs.docker.com/get-docker/)
 1. Prepare the metadata file for your dataset. Use the [download metadata script](../0_download_metadata) to get all metadata for a specified organism. To append local data, you can add new rows to the tsv file and fill out the following columns:
     1. `Experiment`: For public data, this is your SRX ID. For local data, data should be named with a standardized ID (e.g. ecoli_0001)
@@ -14,6 +12,7 @@ Nextflow pipeline to download and process microbial RNA-seq data from NCBI SRA
     1. `Run`: One or more SRR numbers referring to individual lanes from a sequencer. This field is empty for local data.
     1. `R1`: For local data, the complete path to the R1 file. If files are stored on AWS S3, filenames should look like `s3://<bucket/path/to>.fastq.gz`. `R1` and `R2` columns are empty for public SRA data.
     1. `R2`: Same as R1. This will be empty for SINGLE end sequences.
+    1. Convert the tab separated metadata file (.tsv) to a .txt file 
 1. Download your sequence files:
     1. Download FASTA and GFF3 files for your genome and plasmids (if relevant) from NCBI.
     1. Put these in a folder named `sequence_files`, and make sure that this folder only contains files for one organism.

2_process_data/conf/docker.config

@@ -4,9 +4,8 @@ docker {
 }
 
 process {
-
     withLabel: fastq {
-        container = "avsastry/fasterq-dump:1.2"
+        container = "ayyuan/fasterq-dump:1.0"
     }
 
     withLabel: trim_galore {

2_process_data/conf/user.config

@@ -5,5 +5,5 @@ params.organism = "" // Example: "Escherichia_coli"
 params.force = false
 
 // Defaults for organism directory and metadata file
-params.metadata = "" // Must be a tab-separated values (tsv) file
+params.metadata = "" // Must be a tab-separated values file in .txt format
 params.sequence_dir = "" // Must contain a genome.gff3 and genome.fasta file

2_process_data/nextflow_environment.yml

@@ -0,0 +1,203 @@
+name: nextflow
+channels:
+  - bioconda
+  - conda-forge
+  - defaults
+dependencies:
+  - _libgcc_mutex=0.1=main
+  - _openmp_mutex=4.5=1_gnu
+  - anyio=3.5.0=py39h06a4308_0
+  - argon2-cffi=21.3.0=pyhd3eb1b0_0
+  - argon2-cffi-bindings=21.2.0=py39h7f8727e_0
+  - asttokens=2.0.5=pyhd3eb1b0_0
+  - attrs=21.4.0=pyhd3eb1b0_0
+  - babel=2.9.1=pyhd3eb1b0_0
+  - backcall=0.2.0=pyhd3eb1b0_0
+  - beautifulsoup4=4.11.1=py39h06a4308_0
+  - blas=1.0=mkl
+  - blast=2.5.0=hc0b0e79_3
+  - bleach=4.1.0=pyhd3eb1b0_0
+  - boost=1.73.0=py39h06a4308_11
+  - bottleneck=1.3.4=py39hce1f21e_0
+  - brotli=1.0.9=he6710b0_2
+  - brotlipy=0.7.0=py39h27cfd23_1003
+  - bzip2=1.0.8=h7b6447c_0
+  - c-ares=1.18.1=h7f8727e_0
+  - ca-certificates=2022.4.26=h06a4308_0
+  - certifi=2021.10.8=py39h06a4308_2
+  - cffi=1.15.0=py39hd667e15_1
+  - charset-normalizer=2.0.4=pyhd3eb1b0_0
+  - conda=4.12.0=py39h06a4308_0
+  - conda-package-handling=1.8.1=py39h7f8727e_0
+  - cryptography=37.0.1=py39h9ce1e76_0
+  - curl=7.82.0=h7f8727e_0
+  - cycler=0.11.0=pyhd3eb1b0_0
+  - dbus=1.13.18=hb2f20db_0
+  - debugpy=1.5.1=py39h295c915_0
+  - decorator=5.1.1=pyhd3eb1b0_0
+  - defusedxml=0.7.1=pyhd3eb1b0_0
+  - entrypoints=0.4=py39h06a4308_0
+  - executing=0.8.3=pyhd3eb1b0_0
+  - expat=2.4.4=h295c915_0
+  - fontconfig=2.13.1=h6c09931_0
+  - fonttools=4.25.0=pyhd3eb1b0_0
+  - freetype=2.11.0=h70c0345_0
+  - gettext=0.21.0=hf68c758_0
+  - giflib=5.2.1=h7b6447c_0
+  - git=2.34.1=pl5262hc120c5b_0
+  - glib=2.69.1=h4ff587b_1
+  - greenlet=1.1.1=py39h295c915_0
+  - gst-plugins-base=1.14.0=h8213a91_2
+  - gstreamer=1.14.0=h28cd5cc_2
+  - icu=58.2=he6710b0_3
+  - idna=3.3=pyhd3eb1b0_0
+  - intel-openmp=2021.4.0=h06a4308_3561
+  - ipykernel=6.9.1=py39h06a4308_0
+  - ipython=8.3.0=py39h06a4308_0
+  - ipython_genutils=0.2.0=pyhd3eb1b0_1
+  - ipywidgets=7.6.5=pyhd3eb1b0_1
+  - jedi=0.18.1=py39h06a4308_1
+  - jinja2=3.0.3=pyhd3eb1b0_0
+  - joblib=1.1.0=pyhd3eb1b0_0
+  - jpeg=9e=h7f8727e_0
+  - json5=0.9.6=pyhd3eb1b0_0
+  - jsonschema=4.4.0=py39h06a4308_0
+  - jupyter=1.0.0=py39h06a4308_7
+  - jupyter_client=7.2.2=py39h06a4308_0
+  - jupyter_console=6.4.3=pyhd3eb1b0_0
+  - jupyter_contrib_core=0.3.3=py_2
+  - jupyter_contrib_nbextensions=0.5.1=pyhd8ed1ab_2
+  - jupyter_core=4.10.0=py39h06a4308_0
+  - jupyter_highlight_selected_word=0.2.0=py39hf3d152e_1005
+  - jupyter_latex_envs=1.4.6=pyhd8ed1ab_1002
+  - jupyter_nbextensions_configurator=0.4.1=py39hf3d152e_2
+  - jupyter_server=1.13.5=pyhd3eb1b0_0
+  - jupyterlab=3.3.2=pyhd3eb1b0_0
+  - jupyterlab_pygments=0.1.2=py_0
+  - jupyterlab_server=2.12.0=py39h06a4308_0
+  - jupyterlab_widgets=1.0.0=pyhd3eb1b0_1
+  - kiwisolver=1.3.2=py39h295c915_0
+  - krb5=1.19.2=hac12032_0
+  - lcms2=2.12=h3be6417_0
+  - ld_impl_linux-64=2.35.1=h7274673_9
+  - libboost=1.73.0=h3ff78a5_11
+  - libcurl=7.82.0=h0b77cf5_0
+  - libedit=3.1.20210910=h7f8727e_0
+  - libev=4.33=h7f8727e_1
+  - libffi=3.3=he6710b0_2
+  - libgcc-ng=9.3.0=h5101ec6_17
+  - libgfortran-ng=7.5.0=ha8ba4b0_17
+  - libgfortran4=7.5.0=ha8ba4b0_17
+  - libgomp=9.3.0=h5101ec6_17
+  - libnghttp2=1.46.0=hce63b2e_0
+  - libpng=1.6.37=hbc83047_0
+  - libsodium=1.0.18=h7b6447c_0
+  - libssh2=1.10.0=h8f2d780_0
+  - libstdcxx-ng=9.3.0=hd4cf53a_17
+  - libtiff=4.2.0=h85742a9_0
+  - libuuid=1.0.3=h7f8727e_2
+  - libwebp=1.2.2=h55f646e_0
+  - libwebp-base=1.2.2=h7f8727e_0
+  - libxcb=1.14=h7b6447c_0
+  - libxml2=2.9.12=h74e7548_1
+  - libxslt=1.1.34=hc22bd24_0
+  - lxml=4.8.0=py39h1f438cf_0
+  - lz4-c=1.9.3=h295c915_1
+  - markupsafe=2.0.1=py39h27cfd23_0
+  - matplotlib=3.5.1=py39h06a4308_1
+  - matplotlib-base=3.5.1=py39ha18d171_1
+  - matplotlib-inline=0.1.2=pyhd3eb1b0_2
+  - mistune=0.8.4=py39h27cfd23_1000
+  - mkl=2021.4.0=h06a4308_640
+  - mkl-service=2.4.0=py39h7f8727e_0
+  - mkl_fft=1.3.1=py39hd3c417c_0
+  - mkl_random=1.2.2=py39h51133e4_0
+  - munkres=1.1.4=py_0
+  - nbclassic=0.3.5=pyhd3eb1b0_0
+  - nbclient=0.5.13=py39h06a4308_0
+  - nbconvert=6.4.4=py39h06a4308_0
+  - nbformat=5.3.0=py39h06a4308_0
+  - ncurses=6.3=h7f8727e_2
+  - nest-asyncio=1.5.5=py39h06a4308_0
+  - notebook=6.4.11=py39h06a4308_0
+  - numexpr=2.8.1=py39h6abb31d_0
+  - numpy=1.21.5=py39he7a7128_2
+  - numpy-base=1.21.5=py39hf524024_2
+  - openssl=1.1.1o=h7f8727e_0
+  - packaging=21.3=pyhd3eb1b0_0
+  - pandas=1.4.2=py39h295c915_0
+  - pandocfilters=1.5.0=pyhd3eb1b0_0
+  - parso=0.8.3=pyhd3eb1b0_0
+  - pcre=8.45=h295c915_0
+  - pcre2=10.37=he7ceb23_1
+  - perl=5.26.2=h14c3975_0
+  - pexpect=4.8.0=pyhd3eb1b0_3
+  - pickleshare=0.7.5=pyhd3eb1b0_1003
+  - pillow=9.0.1=py39h22f2fdc_0
+  - pip=21.2.4=py39h06a4308_0
+  - prometheus_client=0.13.1=pyhd3eb1b0_0
+  - prompt-toolkit=3.0.20=pyhd3eb1b0_0
+  - prompt_toolkit=3.0.20=hd3eb1b0_0
+  - ptyprocess=0.7.0=pyhd3eb1b0_2
+  - pure_eval=0.2.2=pyhd3eb1b0_0
+  - py-boost=1.73.0=py39ha9443f7_11
+  - pycosat=0.6.3=py39h27cfd23_0
+  - pycparser=2.21=pyhd3eb1b0_0
+  - pygments=2.11.2=pyhd3eb1b0_0
+  - pyopenssl=22.0.0=pyhd3eb1b0_0
+  - pyparsing=3.0.4=pyhd3eb1b0_0
+  - pyqt=5.9.2=py39h2531618_6
+  - pyrsistent=0.18.0=py39heee7806_0
+  - pysocks=1.7.1=py39h06a4308_0
+  - python=3.9.12=h12debd9_0
+  - python-dateutil=2.8.2=pyhd3eb1b0_0
+  - python-fastjsonschema=2.15.1=pyhd3eb1b0_0
+  - python_abi=3.9=2_cp39
+  - pytz=2021.3=pyhd3eb1b0_0
+  - pyyaml=6.0=py39h7f8727e_1
+  - pyzmq=22.3.0=py39h295c915_2
+  - qt=5.9.7=h5867ecd_1
+  - qtconsole=5.3.0=pyhd3eb1b0_0
+  - qtpy=2.0.1=pyhd3eb1b0_0
+  - readline=8.1.2=h7f8727e_1
+  - requests=2.27.1=pyhd3eb1b0_0
+  - ruamel_yaml=0.15.100=py39h27cfd23_0
+  - scikit-learn=1.0.2=py39h51133e4_1
+  - scipy=1.7.3=py39hc147768_0
+  - seaborn=0.11.2=pyhd3eb1b0_0
+  - send2trash=1.8.0=pyhd3eb1b0_1
+  - setuptools=61.2.0=py39h06a4308_0
+  - sip=4.19.13=py39h295c915_0
+  - six=1.16.0=pyhd3eb1b0_1
+  - sniffio=1.2.0=py39h06a4308_1
+  - soupsieve=2.3.1=pyhd3eb1b0_0
+  - sqlalchemy=1.4.32=py39h7f8727e_0
+  - sqlite=3.38.3=hc218d9a_0
+  - stack_data=0.2.0=pyhd3eb1b0_0
+  - terminado=0.13.1=py39h06a4308_0
+  - testpath=0.5.0=pyhd3eb1b0_0
+  - threadpoolctl=2.2.0=pyh0d69192_0
+  - tk=8.6.11=h1ccaba5_1
+  - tornado=6.1=py39h27cfd23_0
+  - tqdm=4.64.0=py39h06a4308_0
+  - traitlets=5.1.1=pyhd3eb1b0_0
+  - typing-extensions=4.1.1=hd3eb1b0_0
+  - typing_extensions=4.1.1=pyh06a4308_0
+  - tzdata=2022a=hda174b7_0
+  - urllib3=1.26.9=py39h06a4308_0
+  - wcwidth=0.2.5=pyhd3eb1b0_0
+  - webencodings=0.5.1=py39h06a4308_1
+  - websocket-client=0.58.0=py39h06a4308_4
+  - wheel=0.37.1=pyhd3eb1b0_0
+  - widgetsnbextension=3.5.2=py39h06a4308_0
+  - xz=5.2.5=h7f8727e_1
+  - yaml=0.2.5=h7b6447c_0
+  - zeromq=4.3.4=h2531618_0
+  - zlib=1.2.12=h7f8727e_2
+  - zstd=1.4.9=haebb681_0
+  - pip:
+    - bio==1.3.8
+    - biopython==1.79
+    - biothings-client==0.2.6
+    - mygene==3.2.2
+prefix: /home/annie/anaconda3/envs/nextflow

3_quality_control/expression_QC_part2.ipynb

@@ -1,5 +1,15 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "toc": true
+   },
+   "source": [
+    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
+    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Expression-Quality-Control-(Part-2)\" data-toc-modified-id=\"Expression-Quality-Control-(Part-2)-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Expression Quality Control (Part 2)</a></span><ul class=\"toc-item\"><li><span><a href=\"#Setup\" data-toc-modified-id=\"Setup-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Setup</a></span><ul class=\"toc-item\"><li><span><a href=\"#Inputs\" data-toc-modified-id=\"Inputs-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span>Inputs</a></span></li><li><span><a href=\"#Load-expression-data\" data-toc-modified-id=\"Load-expression-data-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span>Load expression data</a></span></li><li><span><a href=\"#Load-metadata\" data-toc-modified-id=\"Load-metadata-1.1.3\"><span class=\"toc-item-num\">1.1.3&nbsp;&nbsp;</span>Load metadata</a></span></li></ul></li><li><span><a href=\"#Remove-samples-due-to-poor-metadata\" data-toc-modified-id=\"Remove-samples-due-to-poor-metadata-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Remove samples due to poor metadata</a></span><ul class=\"toc-item\"><li><span><a href=\"#Check-curation\" data-toc-modified-id=\"Check-curation-1.2.1\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>Check curation</a></span></li><li><span><a href=\"#Remove-samples-with-only-one-replicate\" data-toc-modified-id=\"Remove-samples-with-only-one-replicate-1.2.2\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>Remove samples with only one replicate</a></span></li><li><span><a href=\"#Save-this-information-to-the-full-metadata-dataframe\" data-toc-modified-id=\"Save-this-information-to-the-full-metadata-dataframe-1.2.3\"><span class=\"toc-item-num\">1.2.3&nbsp;&nbsp;</span>Save this information to the full metadata dataframe</a></span></li></ul></li><li><span><a href=\"#Check-correlations-between-replicates\" data-toc-modified-id=\"Check-correlations-between-replicates-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Check correlations between replicates</a></span><ul class=\"toc-item\"><li><span><a href=\"#Remove-failed-data-from-log_tpm-files\" data-toc-modified-id=\"Remove-failed-data-from-log_tpm-files-1.3.1\"><span class=\"toc-item-num\">1.3.1&nbsp;&nbsp;</span>Remove failed data from log_tpm files</a></span></li><li><span><a href=\"#Compute-Pearson-R-Score\" data-toc-modified-id=\"Compute-Pearson-R-Score-1.3.2\"><span class=\"toc-item-num\">1.3.2&nbsp;&nbsp;</span>Compute Pearson R Score</a></span></li></ul></li><li><span><a href=\"#Check-that-reference-conditions-still-exist\" data-toc-modified-id=\"Check-that-reference-conditions-still-exist-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Check that reference conditions still exist</a></span></li><li><span><a href=\"#Normalize-dataset-to-reference-conditions\" data-toc-modified-id=\"Normalize-dataset-to-reference-conditions-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Normalize dataset to reference conditions</a></span></li><li><span><a href=\"#Save-final-datasets\" data-toc-modified-id=\"Save-final-datasets-1.6\"><span class=\"toc-item-num\">1.6&nbsp;&nbsp;</span>Save final datasets</a></span></li></ul></li></ul></div>"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -594,7 +604,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "After curation, some samples either did not have enough replicates or metadata to warrant inclusion in this database."
+    "After curation, some samples either did not have enough replicates or metadata to warrant inclusion in this database. Add a `skip` column to the metadata to exclude these samples."
    ]
   },
   {
@@ -1458,7 +1468,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -1472,7 +1482,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.2"
+   "version": "3.7.16"
   },
   "toc": {
    "base_numbering": 1,