🐦 TrainMyBirds — Public Training for WatchMyBirds

This repository provides the public training code for the AI models used in WatchMyBirds, a bird detection and recognition platform. It currently covers classifier training and tuning and will expand to include detection, data processing, active learning, and preannotation workflows.

🔍 EfficientNet classifiers • 🧪 Optuna HPO • 📊 ONNX eval • 📤 Hugging Face export
➡️ Self-contained repo: works from root — all configs, scripts, and utilities included.

🟢 WatchMyBirds App • 🧪 This Repo • 🤗 Model on HF • 🎥 Demo

📚 Classifier Training and Tuning

Overview

This part of the project trains an image classifier to recognize different bird species from images. The training pipeline includes:

• Model selection and configuration using YAML
• Augmentation strategies including MixUp, CutMix, RandAugment, AutoAugment
• Transfer learning from pretrained EfficientNet / EfficientNetV2 models
• Fine-tuning with layer freezing and staged unfreezing
• YAML-driven configuration (see configs/)
• Robust logging with MLflow and automatic output saving to model_registry/

🔧 Hyperparameter Tuning with Optuna

We use Optuna to optimize key hyperparameters such as:

• Learning rate, optimizer type, weight decay
• Image size (guided by GPU OOM profiling)
• MixUp / CutMix probabilities
• Layer freezing depth and dropout
• Early stopping criteria

The search space is defined in model_registry/hyperparams/classifier_hpo_v1_for_optuna.yaml. We tune learning rate, optimizer, dropout, batch size, unfreeze epoch, augmentation, and more.

OOM-aware constraints are applied using model_registry/hardware/classifier_bs_limits.yaml. Trials log metrics like accuracy and loss. The best configurations are exported to model_registry/hyperparams/ for reproducible training.

🐣 Open-Set Validation

Open-set validation evaluates the model’s confidence on unknown bird species using an unlabeled folder provided via --open-val-dir (ImageFolder layout). These images are passed through the model without ground truth labels; the model’s softmax confidences are used to calibrate a rejection threshold.

• Validation samples (known classes) are treated as negatives.
• Open-set samples (unknowns) are treated as positives.

We sweep thresholds and compute a binary F1-score to select a final threshold. The threshold and diagnostics are saved and visualized via:

• threshold_tuning.png
• conf_hist.png
• metrics.yaml

Example Command

python -m optuna_tuner.tune \
  --task classifier \
  --study-name birds_cls_tuning \
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1_for_optuna.yaml \
  --config config/development.yaml \
  --dataset-version v1 \
  --open-val-dir data/dataset/open_val

Goals

• Scientific rigor in evaluation and tuning
• Engineering best practices in modular code structure and configuration
• Reproducibility of ML experiments and results
• Extensibility — this is just the first step of a full ML pipeline

Demo

Curated classifier trained on ~10k images across 29 bird species. Open-set rejection + ONNX deployment ready.

Quick Start

# Create env
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Prepare data (requires detection crops or raw image folder)
python -m classification_training.prepare_data_cls \
  --config config/public_cls.yaml \
  --model-name efficientnet_b3 \
  --dataset-version v1 \
  --output-version cls_v1

# Tune (optional)
python -m optuna_tuner.tune \
  --task classifier \
  --study-name birds_cls_tuning \
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1_for_optuna.yaml \
  --config config/public_cls.yaml \
  --dataset-version v1 \
  --open-val-dir data/dataset/training/classification/images/open_val

# Train
python -m classification_training.train_efficientnet \
  --config config/public_cls.yaml \
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1.yaml \
  --model-name efficientnet_b3 \
  --dataset-version v1

# Evaluate (open-set threshold tuning)
python -m classification_training.posthoc_open_val \
  --model-path model_registry/classifier/<TS>/weights/best.pt \
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1.yaml \
  --config config/public_cls.yaml \
  --open-val-dir data/.../open_val \
  --out-dir outputs/open_val_eval

# Evaluate ONNX
python -m classification_training.onnx_eval \
  --onnx-path model_registry/classifier/<TS>/weights/best.onnx \
  --class-names model_registry/classifier/<TS>/classes.txt \
  --test-dir data/.../test \
  --output-dir outputs/onnx_test_eval \
  --visualize

Repository Structure

• classification_training/ — All classifier training scripts and helpers
  • prepare_data_cls.py — Builds classifier datasets (crops + splits)
  • train_efficientnet.py — Trains EfficientNet models and exports ONNX
  • onnx_eval.py — Evaluates ONNX classifier on labeled test images
  • evaluate_testset.py — Torch/ONNX quick test-set evaluator
  • posthoc_open_val.py — Open‑set evaluation + threshold calibration
  • public_utils.py — Self‑contained public stubs (config, logging, etc.)
  • upload_classifier.py — Public wrapper; logs a hint if upstream tooling absent
• config/ — Example config(s) for the public pipeline
  • public_cls.yaml — Sample configuration with relative paths
• model_registry/ — Local model artifacts (created during training)
  • hyperparams/classifier_hpo_v1.yaml — Example training hyperparams
  • hyperparams/classifier_hpo_v1_for_optuna.yaml — Public Optuna search space
• requirements.txt, environment.yml — Environment setup
• data/ — Place datasets here (see layout below)
• outputs/ — Default location for evaluation outputs and plots

Environment Setup

Choose one of the following:

• Pip
  • python -m venv .venv && source .venv/bin/activate
  • pip install -r requirements.txt
• Conda
  • conda env create -f environment.yml
  • conda activate watchmybirds-public

The requirements include: PyTorch (>=2.1), torchvision, timm, onnx/onnxruntime, numpy, pandas, scikit‑learn, matplotlib, Pillow, PyYAML, tqdm, tensorboard (for logs), GPUtil, OpenCV, imgaug.

Dataset Layout

The classifier expects an ImageFolder‑style structure under your dataset root (dataset_path in the config):

• data/dataset/training/classification/images/train/<class_name>/*.jpg
• data/dataset/training/classification/images/val/<class_name>/*.jpg
• data/dataset/training/classification/images/test/<class_name>/*.jpg

If you start from detection annotations and want to create the classifier crops and splits here, use prepare_data_cls.py.

Configuration

Use config/public_cls.yaml as a starting point. Key fields:

• project_name: Used for bookkeeping and run names.
• dataset_path: Relative path to your dataset root (e.g., data/dataset).
• model_registry: Relative path for runs/artifacts (e.g., model_registry).
• run_logging.output_folder: Where run metadata YAMLs are saved.
• models.classifier.name: EfficientNet backbone (e.g., efficientnet_b3).

The model_registry/hyperparams/ folder contains example hyperparameters:

• classifier_hpo_v1.yaml — A strong starting point for training
• classifier_hpo_v1_for_optuna.yaml — Public Optuna search space (keys align with training)

Data Preparation — classification_training/prepare_data_cls.py

Purpose: build classifier image datasets from annotations and/or existing images and produce split metadata and stats.

Highlights:

• Reads global config via public_utils.load_config (relative paths).
• Determines prepared size (square crop target) from config/limits, or CLI override.
• Builds/updates the following under data/dataset/training/classification/:
  • images/train|val|test/<class>/… — image crops per class and split
  • Optionally images/open_val/<class>/… — open‑set images for threshold calibration
  • configs/cls_data_<project>_<version>.yaml — dataset reference used by training
• Cropping from detections:
  • Converts bounding boxes to centered, padded square crops with OpenCV
  • Resizes crops to the prepared size
• Class balancing:
  • Downsamples classes above max_crops_per_class
  • Augments underrepresented classes using imgaug to approach the target count
• Splitting:
  • Stratified train/val/test splits; optional caps for val/test
• Stats and transparency:
  • Writes summary JSON/YAML and per‑split counts
  • Records dropped rows (if any) and reasons
• CLI (from repo root):
  • python -m classification_training.prepare_data_cls
    --config config/public_cls.yaml
    --model-name efficientnet_b3
    --dataset-version v1
    --output-version cls_v1

Training — classification_training/train_efficientnet.py

Purpose: train EfficientNet classifiers with strong defaults, log metrics/artifacts, and export deployable ONNX.

Key features:

• Model/backbone
  • EfficientNet variants via timm (e.g., efficientnet_b0..b5, tf_efficientnetv2_*)
  • Resolves input size from model defaults or --img-size/YAML
• Data transforms
  • Train: RandomResizedCrop + HorizontalFlip + AutoAugment(IMAGENET) + Normalize + RandomErasing
  • Val: Resize(by crop_pct) + CenterCrop + Normalize
• Mixup/CutMix (optional)
  • Switchable via hyperparams; uses SoftTarget loss when enabled
• Optimizer & schedule
  • AdamW or Adam, warmup (LinearLR) + CosineAnnealingLR
• Progressive unfreezing
  • Trains head first, then unfreezes and rebuilds optimizer at unfreeze_epoch
• Early stopping
  • Monitors loss or acc, with absolute or relative deltas
• Logging
  • TensorBoard logs (in public build, start is logged but not auto‑spawned)
  • Per‑run YAML metadata via RunLogger
• Exports and registry
  • Saves best.pt/last.pt and exports weights/best.onnx
  • Writes metrics.json (accuracy, top‑5, loss, confusion matrix, git hash, notes)
  • Saves pointers in model_registry/classifier/latest_models.json
  • Copies used_hyperparams.yaml and used_dataset.yaml into the run folder
• Open‑set support (optional)
  • If open_val.csv exists under dataset_path, runs CPU open‑set evaluation to log confidence stats
  • Produces open_val_predictions.csv, open_set_eval.yaml, optional histograms
  • Calibrates a rejection threshold and writes calibrated_threshold.yaml

Inputs:

• Config: --config config/public_cls.yaml
• Hyperparams: --hp-yaml model_registry/hyperparams/classifier_hpo_v1.yaml
• Backbone: --model-name efficientnet_b3

Outputs (example):

• model_registry/classifier/<TIMESTAMP>/
  • weights/ → best.onnx, best.pt, last.pt
  • tensorboard/ → TB event files
  • metrics.json, used_hyperparams.yaml, used_dataset.yaml
  • Optional: open_val_predictions.csv, calibrated_threshold.yaml
• model_registry/classifier/latest_models.json (pointer)

CLI example:

• python -m classification_training.train_efficientnet
  --config config/public_cls.yaml
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1.yaml
  --model-name efficientnet_b3
  --dataset-version v1
  --notes "public training run"

Post‑hoc Open‑Set Validation — classification_training/posthoc_open_val.py

Purpose: evaluate a checkpoint (.pt) on an open‑set directory and find a rejection threshold.

What it does:

• Builds a CPU model from timm with provided hp (image_size, dropout)
• Runs inference on open_val_dir, logs confidences, and writes:
  • open_val_predictions.csv — per‑sample results
  • open_set_eval.yaml — summary (N, avg_conf, %below threshold)
  • conf_hist.png, threshold_tuning.png — plots
  • calibrated_threshold.yaml — best F1 threshold (open as positive class)

CLI example:

• python -m classification_training.posthoc_open_val
  --model-path model_registry/classifier//weights/best.pt
  --hp-yaml model_registry/hyperparams/classifier_hpo_v1.yaml
  --config config/public_cls.yaml
  --open-val-dir data/dataset/training/classification/images/open_val
  --dataset-version v1
  --out-dir outputs/open_val_eval

ONNX Evaluation — classification_training/onnx_eval.py

Purpose: run ONNX classifier on labeled test images and compute rich metrics.

What it does:

• Loads ONNX with CPU provider, infers input size from model input tensor
• Builds ImageFolder test set and label mapping (or from a provided classes.txt)
• For each image: preprocess → logits → probs → top‑1 + confidence
• Computes and writes:
  • predictions.csv — path, true_label, pred_label, confidence, is_rejected
  • metrics.yaml — accuracy (overall + non‑rejected), rejection rate, per‑class PRF, top‑k accuracy, best threshold by F1
  • conf_hist.png, threshold_tuning.png — plots
• Optional visualization (--visualize): misclassifications and low‑confidence grids

CLI example:

• python -m classification_training.onnx_eval
  --onnx-path model_registry/classifier//weights/best.onnx
  --class-names model_registry/classifier//classes.txt
  --test-dir data/dataset/training/classification/images/test
  --output-dir outputs/onnx_test_eval
  --threshold 0.5

Results

📦 Public Dataset Access

The training dataset used in this project is based on Creative Commons–licensed bird images from iNaturalist. To ensure compliance and modularity, a reduced public version of the dataset (including attribution, image URLs, hashes, and annotations) is maintained in a separate repository:

👉 WatchMyBirds-Data (GitHub)

Contents

• Bounding box annotations in COCO format
• Attribution information and image hash for deduplication

You can use this CSV to download the training data yourself using any scripting method (e.g., wget, requests). Use the image_hash as filename to save images into data/raw/. Annotation JSON is stored per sample in the approved_annotation column.

📈 Metrics Summary

Dataset	Model	Top-1 Accuracy	Threshold	Rejection Rate
val	efficientnet_b3	94.8%	0.61	5.2%
test	ONNX export	94.2%	0.59	6.4%

📊 Example Plots

• 
• 
• Download metrics.yaml

🔎 Error Analysis & Qualitative Examples

• Misclassified high‑confidence samples (top 25):

  

• Low‑confidence samples (top 25):

  

Key Metrics (from assets/metrics.yaml)

• Top‑1 accuracy (overall): 0.9015
• Accuracy (non‑rejected): 0.9792
• Rejection rate: 0.0794
• F1 (micro): 0.9792, F1 (macro): 0.9788
• Best threshold by F1: 0.10 (F1=0.9412)

Hyperparameters

Use model_registry/hyperparams/classifier_hpo_v1.yaml as a solid baseline for training. For hyperparameter optimization, see model_registry/hyperparams/classifier_hpo_v1_for_optuna.yaml — it defines a public search space aligned with the training script’s expected keys (e.g., num_epochs, adamW_weight_decay). Your Optuna runner should read that YAML and pass the chosen values via CLI or merge into the training YAML before invoking the trainer.

Hugging Face Upload (Placeholder)

Artifacts to upload typically include best.onnx and classes.txt. Use huggingface-cli or huggingface_hub from your environment.

Example commands (replace placeholders):

• export HF_TOKEN=hf_…
• huggingface-cli upload /<repo_name> model_registry/classifier//weights/best.onnx --path-in-repo best.onnx --repo-type model --token "$HF_TOKEN"
• huggingface-cli upload /<repo_name> model_registry/classifier//classes.txt --path-in-repo classes.txt --repo-type model --token "$HF_TOKEN"

classification_training/upload_classifier.py will attempt to call upstream project tooling if present; otherwise it logs a helpful warning.

Reproducibility & Logging

• All scripts use relative paths and log warnings instead of failing when optional services (e.g., TensorBoard) are unavailable.
• Each training run writes:
  • Run YAML metadata into model_registry/run_logs/
  • A timestamped folder under model_registry/classifier/ with weights and artifacts
• public_utils.py centralizes config loading, logging, and small helpers to keep this repo standalone and public‑ready.

Troubleshooting

• Import errors when running scripts directly:
  • Prefer module mode: python -m classification_training.<script> from repo root.
  • Scripts include fallback imports to work when invoked directly.
• CUDA/GPUs:
  • Training uses CUDA if available; ONNX eval runs on CPU by default.
• ONNXRuntime GPU:
  • Replace onnxruntime with onnxruntime-gpu in your environment if needed and ensure CUDA compatibility.

Contribute Labels

Help improve bird recognition! 🐦
Open a GitHub issue titled "Join Labeling" and we’ll invite you to our Label Studio instance.

👉 Join the labeling project

License and Contributions

This public package is a subset of the broader WatchMyBirds pipeline, focused on classification. Contributions and issues are welcome for the public code included here.