Skip to content

README.md

Latest commit

 

History

History
210 lines (147 loc) · 8.42 KB

README.md

File metadata and controls

210 lines (147 loc) · 8.42 KB

Echoflow

CI DOI DOI

Do you have Terabytes of unprocessed Sonar data? Use this for easy viewing and inference!

EchoFlow is a three-stage containerised pipeline that converts raw Kongsberg EK80 echosounder files into echograms and DINO ViT attention maps.

Stages:

  1. Conversion (raw) — decodes .raw pings to volume-backscattering strength NetCDF files via pyEcholab.
  2. Pre-processing (preprocessing) — contrast-stretches and tiles echograms as PNGs.
  3. Inference (infer) — runs a DINO Vision Transformer to produce per-patch attention heat-maps.

Features

  • Preprocessing: Prepares the data for inference.
  • Inference: Utilises DINO for visual inference and attention inspection.
  • Docker Support: Run the pipeline in an isolated Docker environment for consistency and ease of use.

Repository Structure

.
├── raw_consumer/                # process raw to xarray
│   ├── Dockerfile.raw            # Dockerfile for volumetric backscatter computing with pyEcholab
│   ├── preprocessing.py          # Script for converting raw file to volumetric backscatter cubes
├── preprocessing/               # Preprocessing components
│   ├── Dockerfile.preprocessing  # Dockerfile for preprocessing
│   ├── preprocessing.py          # Script for preprocessing input data
├── inference/                   # Inference components
│   ├── Dockerfile.infer          # Dockerfile for inference
│   ├── attention_inspect.py      # Script for inspecting attention maps
│   ├── inspect_attention.py      # Main script for running inference and inspection
│   ├── requirements.txt          # Python dependencies for inference demo
│   ├── utils.py                  # Utility functions for inference demo
│   ├── vision_transformer.py     # DINO Vision Transformer model implementation
├── docker-compose.yml            # Docker Compose file to run the entire pipeline
├── entrypoint.sh                 # Entrypoint script for Docker container
├── infer.py                      # Main script to run inference outside Docker
├── run_docker.sh                 # Script to run the pipeline using Docker
├── watchdog.py                   # Script to watch for changes in the pipeline

Installation

Prerequisites

  • Docker ≥ 24
  • Docker Compose v2 (docker compose — note: no hyphen)
  • Git
  • AWS CLI (for downloading the sample input file)

Clone with submodules

git clone --recurse-submodules https://github.com/erlingdevold/EchoFlow.git

If you have already cloned without submodules:

git submodule update --init --recursive

Without Docker (native install)

The raw_consumer stage requires HDF5 and netCDF C libraries:

  • macOS: brew install hdf5 netcdf
  • Ubuntu/Debian: apt-get install libhdf5-dev libnetcdf-dev
  • Docker images include these already — no action needed for containerised use.

Then install Python dependencies for the stages you need:

pip install -r raw_consumer/requirements.txt
pip install -r preprocessing/requirements.txt
pip install -r inference/requirements.txt

Populate input

The command below fetches a publicly available NOAA EK80 test file (~105 MB) into data/input/ and initialises git submodules before building the Docker images.

aws s3 cp --no-sign-request \
  "s3://noaa-wcsd-pds/data/raw/Bell_M._Shimada/SH2306/EK80/Hake-D20230811-T165727.raw" \
  data/input/

touch ./inference/checkpoint.pth

git submodule sync --recursive

Setup

Running with Docker

Run the full pipeline

docker compose up --build

This builds and starts all three stages (Conversion, Pre-processing, Inference) plus the progress monitor.

Run individual stages

You can also run each stage independently:

Stage Command
Stage 1 — Conversion docker compose up --build raw
Stage 2 — Pre-processing docker compose up --build preprocessing
Stage 3 — Inference docker compose up --build infer
All stages (no monitor) docker compose up --build raw preprocessing infer

Each stage reads from and writes to bind-mounted directories under ./data/, so stages can be run in sequence without rebuilding upstream containers.

Monitor dashboard

Once the pipeline is running, a progress monitor is available at http://localhost:8050.

The dashboard is a pipeline progress monitor — it tracks file counts and tail-logs for each stage. It is not an inference viewer. Actual outputs are written to:

  • data/raw_consumer/ — converted NetCDF files (Stage 1 output)
  • data/preprocessing/ — echogram PNGs (Stage 2 output)
  • data/inference/ — attention map PNGs (Stage 3 output)

Performance / parallelism

Stages 1 (conversion) and 2 (pre-processing) use process pools to parallelise work across files — controlled by the MAX_WORKERS env var (defaults to CPU count). Stage 3 (inference) uses batched GPU inference with a configurable BATCH_SIZE (default 4) and auto-detects CUDA when available. Because .raw files are large XML datagrams, file I/O is the primary bottleneck for stages 1–2; adding CPU cores within a node yields proportional throughput gains. Parallelism is intra-node only and there is no built-in cluster or scheduler integration.

Benchmarking

The benchmark script runs the full three-stage pipeline inside Docker on real EK80 data, varying MAX_WORKERS to measure scaling:

# 1. Download 64 test echograms (~6.4 GB) from NOAA public bucket
bash download_bench_data.sh

# 2. Build Docker images
docker compose build

# 3. Install requirements
pip install tyro

# 4. Run benchmark (default: 1,2,4,8 workers)
python benchmark.py

See python benchmark.py --help for options. Reference results on a 12-core Intel i7-12700, 32 GB RAM, RTX 3090 (64 files, 6.7 GB):

Workers Stage 1 (s) Stage 2 (s) Stage 3 (s) Total (s) s/file
1 139.6 514.0 393 1046.6 16.35
2 86.5 275.7 393 755.2 11.80
4 56.3 156.2 393 605.5 9.46
8 56.9 97.2 393 547.1 8.55

Stages 1–2 scale with MAX_WORKERS; stage 3 is GPU-bound. In production, stages overlap via the watchdog. Run python benchmark.py on your hardware to regenerate.

ENV variables

Global

  • MAX_WORKERS — max parallel workers for stages 1–2 (default: CPU count)

watchdog.py

  • LOG_DIR (default: "/data/log")
  • INPUT_DIR (default: "/data/sonar")
  • OUTPUT_DIR (default: "/data/processed")

raw.py (Stage 1)

  • INPUT_DIR (default: "/data/sonar")
  • OUTPUT_DIR (default: "/data/processed")
  • LOG_DIR (default: "log")

preprocessing.py (Stage 2)

  • INPUT_DIR (default: "/data/processed")
  • OUTPUT_DIR (default: "/data/test_imgs")
  • LOG_DIR (default: ".")
  • KEEP_INTERMEDIATES — preserve .nc files after processing (default: "true")

inspect_attention.py (Stage 3)

  • INPUT_DIR (default: "/data/test_imgs")
  • OUTPUT_DIR (default: "/data/inference")
  • LOG_DIR (default: ".")
  • PATCH_SZ (default: 8)
  • ARCH (default: 'vit_small')
  • DOWNSAMPLE_SIZE (default: 5000)
  • BATCH_SIZE — images per GPU forward pass (default: 4)
  • DEVICE — "cuda", "cpu", or auto-detect (default: auto)

Output

The output of the inference step, including generated attention maps and transformed images, will be saved in data/inference/. Each run creates a subdirectory named after the input file for organised output management.

Contributing

See CONTRIBUTING.md for guidelines on reporting bugs, suggesting features, and submitting pull requests.

License

Licensed under the MIT License — see LICENSE for details.

Acknowledgements

This pipeline uses the DINO Vision Transformer for attention-based image analysis. The implementation is based on research from the original DINO paper by Facebook AI Research (FAIR).