Detecting and tracking people and vehicles across 36 fixed surveillance cameras, then stress testing the models under adverse conditions such as rain, fog, darkness and harsh brightness.
This project fine tunes a YOLOv8 detector on the Dana36 dataset and pairs it with a lightweight tracker based on IoU matching to follow each subject frame by frame. This is the kind of pipeline used in real indoor and outdoor surveillance.
The tracker assigns a persistent ID and colour to every detected subject, and counts people versus vehicles live in the top left overlay. Below you can see the baseline model on a clean outdoor scene, and the weather augmented model holding up under heavy rain.
|
Model 1 · clean outdoor scene |
Model 4 · heavy rain 🌧️ |
ℹ️ If the players above do not appear inline, use the full demo gallery below. Every link opens GitHub's built in video viewer.
The goal is a robust surveillance pipeline that answers two questions on video streams. "What or who is in the frame?" (detection) and "Is it the same subject across frames?" (tracking). The work is split into two notebooks.
| Notebook | Content |
|---|---|
1_EDA.ipynb |
Exploratory data analysis. Dataset structure, temporal and spatial analysis (trajectories, presence heatmaps), preprocessing (letterbox resize to 640×640, normalization, per pixel variance study). |
2_IMPLEMENTATION.ipynb |
Model selection (YOLO versus Faster R-CNN), fine tuning strategies, evaluation, and object tracking. |
A full write up is available in report.pdf and presentation.pdf.
1. Data analysis and preprocessing. 36 fixed cameras (27 outdoor, 9 indoor). Subjects are annotated only while moving, so a parked car stops being annotated once it becomes static. Frames are resized to 640×640 with letterboxing to preserve aspect ratio, bounding boxes are converted to YOLO format, and per pixel variance is used to characterize environmental noise such as wind, moving vegetation and light shifts.
2. Model selection. YOLO versus Faster R-CNN. YOLO wins on both accuracy and speed under limited GPU (15 GB VRAM or less):
| Scene | YOLO | Faster R-CNN |
|---|---|---|
| 30 | 23.4 s | 9 min 54 s |
| 08 | 25.6 s | 9 min 17 s |
| 11 | 28.5 s | 10 min 05 s |
| 25 | 25.6 s | 13 min 07 s |
3. Fine tuning. 24 scenes for train and validation, 12 held out for test (03, 04, 05, 08, 18, 22, 23, 24, 27, 31, 33, 36), preserving the indoor and outdoor ratio. The two classes are collapsed to person and vehicle. Splits are chronological (first 80% train, last 20% validation) to avoid near duplicate frames leaking across splits. Four strategies were tried to handle class imbalance and weather robustness (see below).
4. Object tracking. A custom SimpleTracker links detections across frames through IoU matching, with a greedy assignment and a max_age buffer to survive short occlusions. Weather and lighting perturbations (rain, fog, darkness, brightness) are applied on the fly with Albumentations to test robustness.
Four fine tuning runs (20 epochs, YOLOv8n, 640×640), evaluated on the validation set.
| # | Strategy | Precision | Recall | mAP@50 | mAP@50:95 |
|---|---|---|---|---|---|
| 1 | Baseline fine tune | 0.950 | 0.938 | 0.982 | 0.900 |
| 2 | + label rebalancing (drop persons) | 0.914 | 0.924 | 0.975 | 0.880 |
| 3 | + class weighting | 0.795 | 0.939 | 0.937 | 0.859 |
| 4 | + weather and lighting augmentation | 0.856 | 0.922 | 0.946 | 0.861 |
Takeaways
- Model 1 (clean baseline) is the strongest on standard data, with the highest precision and mAP.
- Model 4 trades a little accuracy for robustness to rain, fog and low light, which matters most for real outdoor surveillance.
- Class rebalancing tricks (runs 2 and 3) did not help and were dropped, but they are kept in
runs/for reproducibility.
The two selected models, 1 (baseline) and 4 (weather augmented), are compared side by side in the tracking demos below.
🎥 Tracking demos (videos_tracking/)
36 tracking videos across 5 conditions and 12 scenes, each rendered with both models. Click any scene to play it on GitHub. Indoor scenes are marked 🏠, outdoor 🌳.
| Condition | Model 1 (baseline) | Model 4 (weather augmented) |
|---|---|---|
| ☀️ Normal | 03 🌳 · 08 🌳 · 18 🌳 · 23 🏠 · 27 🏠 | 03 🌳 · 08 🌳 · 18 🌳 · 23 🏠 · 27 🏠 |
| 🌧️ Rain | 03 🌳 · 05 🌳 · 31 🌳 · 36 🌳 | 03 🌳 · 05 🌳 · 31 🌳 · 36 🌳 |
| 🌫️ Fog | 08 🌳 · 23 🏠 · 36 🌳 | 08 🌳 · 23 🏠 · 36 🌳 |
| 🌙 Dark | 04 🌳 · 05 🌳 · 24 🏠 | 04 🌳 · 05 🌳 · 24 🏠 |
| 🔆 Bright | 08 🌳 · 24 🏠 · 36 🌳 | 08 🌳 · 24 🏠 · 36 🌳 |
Deep_Learning_Applied_to_Video_Analytics/
├── source-code/
│ ├── 1_EDA.ipynb # Exploratory data analysis
│ ├── 2_IMPLEMENTATION.ipynb # Detection, fine tuning and tracking
│ ├── eda_useful_functions.py # EDA helpers (heatmaps, trajectories, preprocessing)
│ ├── model_useful_functions.py # Detection, fine tuning, SimpleTracker, video rendering
│ └── runs/ # 4 fine tuning runs (weights, curves, confusion matrices)
├── videos_tracking/ # Tracking demos, {condition}/{scene}/model_{1,4}.mp4
│ ├── normal/ rain/ fog/ dark/ bright/
├── report.pdf # Full report
└── presentation.pdf # Slides
# 1. Install dependencies
pip install -r requirements.txt
# 2. Open the notebooks
jupyter notebook source-code/2_IMPLEMENTATION.ipynbRender a tracking video for any scene (see model_useful_functions.py):
import model_useful_functions as muf
muf.run_image_sequence_tracking_(
image_folder="data/views_resized/03/images",
model_path="runs/4_finetuned_data_aug/weights/best.pt",
output_folder="images_tracking/rain/03/model_4",
video_output_path="videos_tracking/rain/03/model_4.mp4",
fps=10,
rain=True, # dark / bright / rain / fog toggles
)The
data/folder (raw Dana36 frames and annotations) is not shipped here due to size. Point the scripts at your local copy of the dataset.
Dana36 is a multi camera surveillance dataset. 36 fixed cameras (27 outdoor, 9 indoor, some overlapping or zoomed views) film staged scenarios such as people arriving by car, walking and climbing stairs, with bounding box annotations for every moving person and vehicle.
Nayel BENABDESADOK · Deep Learning and Computer Vision project.