Neuroimaging pipeline manager for the LN2T
About | Installation | Quick Start | Data Organization | CLI Options | Examples | Advanced Documentation
ln2t_tools is a command-line interface (CLI) tool that simplifies the execution of standard neuroimaging pipelines on BIDS-organized datasets. It provides:
- Unified interface: Run multiple neuroimaging pipelines (FreeSurfer, fMRIPrep, QSIPrep, etc.) with consistent syntax
- Container-based reproducibility: All pipelines run in Apptainer containers, ensuring reproducibility and eliminating dependency conflicts
- Automated data discovery: Automatically detects BIDS datasets and participants for streamlined processing
- HPC support: Seamlessly submit jobs to high-performance computing clusters (CECI, etc.)
- Modular architecture: Easily extensible with new tools and pipelines
For a comprehensive guide on data organization, visit the LN2T data repository.
- Python 3.8+
- Apptainer: Must be installed system-wide
- Git (for cloning the repository)
- SSH key (for HPC access, optional)
- Clone the repository:
git clone git@github.com:ln2t/ln2t_tools.git
cd ln2t_tools- Create and activate a virtual environment:
python -m venv venv
source venv/bin/activate- Install the package:
pip install -U pip && pip install -U .- (Optional) Enable bash completion:
Bash completion should be installed automatically during installation. If it's not set up, run:
python -m ln2t_tools.install.post_install
source ~/.bashrcThen reload your shell:
source ~/.bashrcln2t_tools uses Apptainer to run pipelines in containers. Container images are stored in /opt/apptainer by default.
Setup permissions:
sudo mkdir -p /opt/apptainer
sudo chown -R $USER:$USER /opt/apptainer
sudo chmod -R 755 /opt/apptainerCustom directory (optional):
ln2t_tools freesurfer --dataset mydataset --apptainer-dir /path/to/custom/dirSome pipelines require a FreeSurfer license (free academic license available at FreeSurfer Registration).
Default location: ~/licenses/license.txt
Custom location:
ln2t_tools freesurfer --dataset mydataset --fs-license /path/to/license.txtTo use ln2t_tools for submitting jobs to high-performance computing clusters, SSH key-based authentication must be configured. This is a prerequisite for HPC functionality.
SSH Key Requirements:
ln2t_toolsuses SSH keys to authenticate with HPC clusters- Default key location:
~/.ssh/id_rsa - Custom key location can be specified with
--hpc-keyfile <path>
Setup:
# Generate SSH key (if not already present)
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa
# Ensure correct permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pubConfigure your cluster's SSH settings according to your HPC provider's documentation. Most HPC centers provide detailed setup instructions for SSH access. Once configured, you can use HPC features with commands like:
ln2t_tools <pipeline> --dataset <dataset_name> --hpc \
--hpc-username <user> --hpc-hostname <host>The basic syntax for running any pipeline is:
ln2t_tools <pipeline_name> --dataset <dataset_name> [options]ln2t_tools freesurfer --dataset <dataset_name>This command will:
- Detect the latest FreeSurfer version
- Download/install the container if needed
- Discover all subjects in the dataset
- Check for already-processed subjects
- Process remaining subjects sequentially
ln2t_tools freesurfer --dataset <dataset_name> \
--participant-label 001 042Tip: Start with a few subjects to verify the setup before processing large cohorts.
- FreeSurfer -
freesurfer: Cortical reconstruction and surface-based analysis - fMRIPrep -
fmriprep: Functional MRI preprocessing - QSIPrep -
qsiprep: Diffusion MRI preprocessing - QSIRecon -
qsirecon: Diffusion MRI reconstruction - MELD Graph -
meld_graph: Lesion detection
ln2t_tools requires data organized according to the Brain Imaging Data Structure (BIDS) specification. Data are organized into three main categories:
Your primary dataset in BIDS format. This contains all the imaging and physiological data acquired for your study.
Location: ~/rawdata/<dataset_name>
Format: Strictly follows BIDS specification with:
- One folder per participant:
sub-001,sub-002, etc. - Data type folders within each participant:
anat/,func/,dwi/,meg/,mrs/, etc. - Required metadata files at dataset root:
dataset_description.json,participants.tsv
Output data from processing pipelines. Each pipeline creates its own versioned folder.
Location: ~/derivatives/<dataset_name>
Structure:
derivatives/
├── freesurfer_7.3.2/
│ ├── sub-001/
│ ├── sub-002/
│ └── ...
├── fmriprep_20.2.0/
│ ├── sub-001/
│ ├── sub-002/
│ └── ...
└── qsiprep_0.14.0/
├── sub-001/
├── sub-002/
└── ...
Key principles:
- One folder per pipeline with version number
- One subfolder per processed participant
- Version numbers enable multiple pipeline versions on the same dataset
Custom scripts and configuration files for your analysis.
Location: ~/code/<dataset_name>
Recommended structure:
code/
├── README.md # Document your analysis pipeline
├── config/ # Configuration files
│ └── qsirecon_config.yaml
└── analysis/ # Custom analysis scripts
├── preprocessing.py
└── statistical_analysis.R
A minimal example of a properly structured dataset:
~/
├── rawdata/
│ └── <dataset_name>/
│ ├── dataset_description.json
│ ├── participants.tsv
│ └── sub-001/
│ ├── anat/
│ │ ├── sub-001_T1w.nii.gz
│ │ └── sub-001_T1w.json
│ ├── func/
│ │ ├── sub-001_task-rest_bold.nii.gz
│ │ └── sub-001_task-rest_bold.json
│ └── dwi/
│ ├── sub-001_dwi.nii.gz
│ ├── sub-001_dwi.bval
│ └── sub-001_dwi.bvec
│
├── derivatives/
│ └── <dataset_name>/
│ ├── freesurfer_7.3.2/
│ │ └── sub-001/
│ └── fmriprep_20.2.0/
│ └── sub-001/
│
└── code/
└── <dataset_name>/
└── README.md
Before running pipelines, ensure:
- Participant folders: Check that
sub-XXXfolders exist for all expected participants - Data types: Verify expected imaging modalities are present (
func/,dwi/,anat/, etc.) - BIDS compliance: Validate with BIDS Validator available as a
ln2t_toolscommand - Metadata files: Confirm
dataset_description.jsonandparticipants.tsvexist in the raw data root
ln2t_tools <tool> --dataset <dataset> # BIDS dataset name
--participant-label <labels> # Process specific participants
--output-label <label> # Custom output directory label
--version <version> # Specific tool version
--max-instances <n> # Max parallel processes (default: 4)
--tool-args "<args>" # Pass arguments to the tool
--list-missing # Show missing participants
--verbosity <level> # Log level: silent, minimal, verbose, debugln2t_tools --list-datasets # List available BIDS datasets
--list-instances # Show running processes
--hpc-status [JOB_ID] # Check HPC job statusln2t_tools <tool> --dataset <dataset> --hpc \
--hpc-username <user> \ # HPC username
--hpc-hostname <host> \ # Cluster hostname (e.g., lyra)
--hpc-time <HH:MM:SS> \ # Job time limit (default: 24:00:00)
--hpc-mem <memory> \ # Memory allocation (default: 32G)
--hpc-cpus <n> \ # Number of CPUs (default: 8)
--hpc-partition <partition> # HPC partition name--fs-license <path> # FreeSurfer license file path
--apptainer-dir <path> # Apptainer images directory
--fs-version <version> # FreeSurfer version for input data
--hpc-keyfile <path> # SSH private key (default: ~/.ssh/id_rsa)
--hpc-gateway <gateway> # ProxyJump gateway hostnameUse --help after any pipeline name for tool-specific options:
ln2t_tools <pipeline> --helpRun fMRIPrep on a dataset:
ln2t_tools fmriprep --dataset mydata --participant-label 001 002 003Run QSIPrep with custom output label:
ln2t_tools qsiprep --dataset mydata --output-label "my_custom_run"Pass arguments directly to tools using --tool-args:
ln2t_tools qsiprep --dataset mydata --participant-label 001 \
--tool-args "--output-resolution 1.5"Submit jobs to a cluster:
ln2t_tools freesurfer --dataset mydata --hpc \
--hpc-username myuser \
--hpc-hostname lyra \
--hpc-time 10:00:00List participants that still need processing:
ln2t_tools fmriprep --dataset mydata --list-missingThis helps when:
- Resuming after errors
- Managing large cohorts
- Generating batch commands
For more examples and detailed workflows, see Examples.
For advanced topics and detailed configuration, see the following guides:
- Data Import Guide - Convert raw neuroimaging data to BIDS format (DICOM, MRS, Physio, MEG)
- Adding a New Tool - Develop and integrate new neuroimaging pipelines with custom Apptainer recipes
- Instance Management - Control parallel processing limits and resource usage
- MELD Graph Setup - Advanced configuration for focal cortical dysplasia detection
For general information, see docs/index.md.