Acoustic Telemetry Detection Filtering Pipeline

Reproducible Data Analysis Course, UGent, Q1 2026

This repository contains a Nextflow pipeline for quality control and filtering of detection data from acoustic telemetry hydrophones/receivers.

The pipeline validates input detection data and deployment metadata files, identifies unrealistic detections, filters the detections, and a small summary report explaining which detections were removed, and why.

The workflow is designed to be fully reproducible:

• each computational step runs inside a container,
• no R packages need to be installed locally,
• the same pipeline can run on a laptop, workstation, or High Performance Computing (HPC) environment.

The pipeline uses Nextflow to manage workflow execution and Apptainer/Docker to provide the software environment.

Table of contents

• Scientific background
• Pipeline overview
• Getting started
  • Cloning the repository
  • Setting up your computer
    • Windows users
    • macOS and Linux users
  • Requirements
    • Installing Nextflow
    • Installing Docker
    • Installing Apptainer
• Running the pipeline
  • Testing the pipeline
    • Test run using Docker
    • Test run using Apptainer (HPC)
  • Running the pipeline with your own data
    • Using ETN data
    • Using other acoustic telemetry datasets
• Additional documentation
• Extending the pipeline
• Acknowledgements

Note

New to Nextflow?

A short introduction to Nextflow concepts (pipelines, modules, profiles, containers, and resume functionality) is available docs/nextflow_intro.md. It is not absolutely necessary to understand all the concepts behind nextflow to run this pipeline though 🙂

---

Scientific background

Acoustic telemetry data is collected by researchers worldwide and deposited in databases of the Ocean Tracking Network (OTN), the European Tracking Network (ETN), and others. After having downloaded detection data from acoustic receivers (specialised hydrophones listening for the signals from transmitters that fish get equipped with by researchers), detections need to be cleaned to retain only reaslistic detections for data analyis (e.g. due to acoustic signal collisions there are false detections). Some solutions exist (there are, for instance, custom functions in the glatos and the ATfiltR R packages), but many researchers still use their own, private functions to filter and clean their detection data prior to analysis.

This repository instead focuses on providing a reproducible, portable workflow that combines quality control, detection filtering, and reporting into a single Nextflow pipeline.

For now, the pipeline flags and excludes detections that satisfy at least one of the following:

1. Detection timestamp is before the respective receiver deployment timestamp
2. Detection timestamp is after the respective receiver retrieval timestamp
3. There is no acoustic receiver deployment metadata for the acoustic detection

This is a very simple filter setup for the moment, but this repository is designed to easily extend the detection filter in the future. More information in section Extending the pipeline.

Pipeline overview

The pipeline takes detection and deployment .csv files as input and contains three steps:

detections.csv + deployments.csv
              │
              ▼
01_INPUT_QC
  • validate input files
  • check required columns
              │
              ├── results/qc/qc_report.txt              # small report containing results of checks performed on the input files
              ▼
02_FILTER_DETECTIONS
  • flag unrealistic detections:
      - detection before receiver deployment
      - detection after receiver deployment
      - no matching deployment ID
              │
              ├── results/flagged_detections.csv        # table with all input detections, with new `flag` column referring to the three filter categories
              ▼
03_SUMMARIZE_EXCLUSIONS
  • split flagged detections into retained and excluded detections
  • summarize retained and excluded detections per receiver deployment
              │
              ├── results/filtered_detections.csv       # table with all retained detections
              ├── results/excluded_detections.csv       # table with all excluded detections (handy for troubleshooting)
              └── results/exclusion_summary.csv         # table with retained/excluded detections per receiver deployment

Note: The input validation step requires the assertr R package, which is provided through a custom container (quay.io/lpohl/r-assertr:1.0), which contains the public rocker/tidyverse:4.3.2 R image with the assertr package on top of it. The filtering and summarization steps only require standard tidyverse packages, but to maximize software environment consistency, all three modules use the same container.

---

Getting started

Cloning the repository:

git clone https://github.com/lottepohl/RDA_nextflow_ATdetectionfilter.git  
cd RDA_nextflow_ATdetectionfilter

The repository contains small test datasets (tests/deployments.csv and tests/detections.csv), so the pipeline can be tested immediately. These data are from the European Tracking Network (https://europeantrackingnetwork.org)

---

Setting up your computer

This pipeline is run from a terminal using Nextflow.

No need to become a Linux expert to use this pipeline 🙂. The commands provided in this documentation can be copied and pasted.

Windows users

Nextflow works best in a Linux environment. Therefore, if you use Windows, you best install Windows Subsystem for Linux (WSL), which provides a Linux command-line environment inside Windows.

Install WSL (Windows)

Open PowerShell as Administrator and run:

wsl --install

Restart your computer when prompted.

After installation, you will have a new application called "Ubuntu" where you can run Nextflow. Or, alternatively, you can open a WSL terminal in an Integrated Development Environment (IDE) of your choice, e.g. VS Code:

Terminal → New Terminal

In the terminal panel, click the dropdown arrow next to the + button and select:

Ubuntu (WSL)

The pipeline files remain on your normal Windows computer, WSL only provides the Linux environment that is needed to run Nextflow.

macOS and Linux users

If you use macOS or Linux, you can run the commands directly from the Terminal application.

Requirements

You need:

• Nextflow (>= 23.04 recommended)
• Docker or Apptainer/Singularity (e.g. when working in a HPC environment).

The pipeline itself provides all software dependencies through containers, no need to have R on your computer, for example.

Installing Nextflow

Follow the official Nextflow installation instructions:

Inside a terminal (a WSL terminal, if you use Windows), run:

curl -s https://get.nextflow.io | bash

Then, move Nextflow somewhere on your PATH, for example by:

mkdir -p ~/bin  
mv nextflow ~/bin/  
export PATH=$HOME/bin:$PATH  

Check installation:

nextflow -version  

Installing Docker

Windows and macOS

Install Docker Desktop from: https://www.docker.com/products/docker-desktop/

After installation:

1. Start Docker Desktop.
2. Wait until Docker reports that it is running.
3. If you use WLS on your windows computer:

 Settings → Resources → WLS Integration → [x] Enable integration with my default WSL distro

4. Leave Docker Desktop running while executing the pipeline.

Verify that Docker is available:

docker --version
docker run hello-world

Linux

Install Docker Engine using your Linux distribution's package manager by following the official Docker installation guide: https://docs.docker.com/engine/install/

Verify the installation:

docker --version
docker run hello-world

Installing Apptainer

Apptainer is primarily intended for HPC systems and is often already available as a module.

Check whether it is installed:

apptainer --version

If not, consult your HPC documentation or system administrator.

---

Running the pipeline

The pipeline can be run on either a local computer or a HPC environment.

The analysis steps are identical in both cases; only the container technology changes:

• Docker is recommended for local computers (Windows, macOS, Linux). Docker provides the software environment needed by the pipeline without requiring users to manually install R packages.
• Apptainer (formerly Singularity) is recommended on HPC systems, where Docker is usually not available because it requires elevated privileges.

For users working on a personal computer, Docker is the recommended option.

---

Testing the pipeline

The repository contains a small test dataset in the tests/ directory.

The test profile is a predefined configuration in nextflow.config that automatically provides:

• the paths to the example detections.csv and deployments.csv files
• the correct column names for these files

This allows the pipeline to be tested without additional configuration.

Test run using Docker

nextflow run main.nf -profile test,docker

Test run using Apptainer (HPC)

On an HPC system, first load Nextflow:

module load Nextflow/<your_version>

For example:

module load Nextflow/26.04.3

Then run:

nextflow run main.nf -profile test,apptainer

After completion, the pipeline should create output files in the results/ directory. To verify if the test run was successful, compare the files in your results/ with the files inside the expected_results\ directory.

The remaining examples use Docker. If you are working on an HPC system, simply replace the docker profile with apptainer.

---

Running the pipeline with your own data

The pipeline requires two .csv input files:

1. A detection file (--detections)
2. A receiver deployment file (--deployments)

The pipeline links acoustic detections to receiver deployments using a deployment identifier.

Using ETN data

If your data follows the column naming conventions used by exports from the European Tracking Network (ETN) from the etn R package, you can use the built-in etn profile.

Run:

nextflow run main.nf \
    -profile etn,docker \
    --detections path/to/detections.csv \
    --deployments path/to/deployments.csv

The etn profile assumes the following column names:

Pipeline parameter	ETN column
detection_datetime_col	date_time
detection_deployment_id_col	deployment_id
deployment_id_col	deployment_id
deploy_datetime_col	deploy_date_time
recover_datetime_col	recover_date_time

---

Using other acoustic telemetry datasets

If your .csv files use different column names, these can be provided as parameters when running the pipeline.

Example:

nextflow run main.nf \
    -profile docker \
    --detections detections.csv \
    --deployments deployments.csv \
    --detection_datetime_col detection_time \
    --detection_deployment_id_col receiver_deployment \
    --deployment_id_col deployment_id \
    --deploy_datetime_col deployment_start \
    --recover_datetime_col deployment_end

This allows the pipeline to be used with different database exports without modifying the R scripts.

For frequently used data formats, a new profile can be added to nextflow.config (for instructions, see docs/development.md).

More explanation about input and output files, required columns, and parameters of this pipelines can be found in docs/pipeline_documentation.md.

---

Additional documentation

The README focuses on getting the pipeline running as quickly as possible. Additional documentation is available in the docs/ directory:

Document	Description
docs/nextflow_intro.md	Introduction to Nextflow concepts, including workflows, modules, processes, profiles, containers, and the -resume functionality. Recommended for users who are new to Nextflow.
docs/pipeline_documentation.md	Detailed reference for the pipeline, including input and output files, parameters, profiles, and module descriptions.
docs/hpc_apptainer.md	Running the pipeline on High Performance Computing (HPC) systems using Apptainer, including common troubleshooting tips.
docs/development.md	Guide for extending or modifying the pipeline, for example by adding new filtering rules or creating profiles for additional data formats.

---

Extending the pipeline

If you would like to add more detection filtering rules, or you want to include your own (non-ETN) detection data column parameters, see docs/development.md for a description of how to modify this pipeline.

Acknowledgements

Thank you for reading this README! I would also like to thank my trainers from VIB during the RDA micro-credential, I am confident that my data analysis practices will be more reproducible in the future thanks to this course!