PaveScan AI

Geospatial computer-vision pipeline that detects road and pavement damage from vehicle-mounted camera footage and maps every detection to GPS coordinates.

Overview

PaveScan AI turns raw survey footage into actionable road-maintenance data. A camera is mounted on a moving vehicle or bicycle and driven along the roads to be inspected; the system then ingests the captured video, runs a fine-tuned YOLO detector on each frame, and fuses every detection with synchronized GNSS (GPS) data so each defect gets a precise latitude/longitude.

The pipeline supports two capture sources:

• Stereolabs ZED stereo camera (.svo2 recordings) — left-view frames are extracted together with a per-pixel depth map, enabling distance-to-object measurement and depth-aware severity grading.
• Standard cameras / image sequences — frames are processed directly, with GPS read either from image EXIF metadata or from a separate GNSS log.

Detections are classified into general roadside assets (e.g. light poles, paint, scratches, dirt/dents, graffiti, VRI markers) and Dutch road-surface defect categories (asphalt cracking, edge damage, ravelling, unevenness, element paving defects, etc.), reflecting a road-asset inventory use case for a Dutch municipal road authority. Results are exported as annotated frames, a compiled video, and CSV/Excel reports.

Features

• Fine-tuned YOLO object detection for road and roadside damage, with a configurable confidence threshold and optional GPU (CUDA) inference.
• Stereo depth integration — extracts depth maps from ZED .svo2 files to compute the real-world distance from camera to each detected object.
• GNSS geolocation — matches each frame to the nearest GNSS timestamp and derives per-detection latitude/longitude via bearing-and-distance projection, frame-position interpolation, or pixel-to-GPS scaling.
• Severity / depth-of-damage grading — bounding-box-area-based B/C/D classification for damage severity.
• VRI analytics — counts roadside markers (VRIs) per frame and computes inter-VRI distances using the Haversine formula.
• Reporting — generates annotated frames, an MP4 detection video, and detailed CSV + Excel reports.
• Self-training utility — pseudo-labels unlabeled imagery above a confidence threshold and retrains the model for iterative dataset expansion.
• Frame tooling — SVO frame extraction (with frame-interval sampling), batch renaming, image resizing, and dataset validation helpers.

Tech Stack

• Python 3.7+
• Ultralytics YOLO (object detection / training)
• PyTorch (model inference, GPU acceleration)
• OpenCV (frame I/O, drawing, video writing)
• NumPy (depth-map handling)
• pandas / openpyxl (CSV & Excel reporting)
• Pillow (EXIF GPS extraction, image processing)
• Stereolabs ZED SDK (pyzed) (SVO2 stereo frame + depth extraction)

Setup / Run

1. Clone the repository

   git clone https://github.com/ube09/PaveScan-AI.git
   cd PaveScan-AI

2. Create a virtual environment (recommended)

   python -m venv venv
   source venv/bin/activate    # Windows: venv\Scripts\activate

3. Install dependencies

   pip install opencv-python numpy ultralytics pandas openpyxl pillow torch

4. Install the ZED SDK (only needed for .svo2 stereo input) Download and install from stereolabs.com/developers, then install the matching pyzed Python wheel.

5. Configure paths. Each script has a Configuration Section near the top — set MODEL_PATH (e.g. Final_RP_best.pt or Fine_tune.pt), the input path (SVO_FILE_PATH or EXTRACTED_FRAMES_DIR), GNSS_FILE_PATH, the output directories, and CONFIDENCE_THRESHOLD as needed.

6. Run a pipeline. Typical entry points:

   • final.py — end-to-end SVO → depth + bearing-based geolocation → reports (GPU-enabled).
   • process.py / Final-road-pole.py — SVO extraction + detection + reporting variants.
   • updated.py — road-surface-class-only detection with pixel-to-GPS conversion and logging.
   • extract.py / frame_extract.py — extract frames from an SVO file.
   • ml.py — pseudo-labeling and model retraining.

   python final.py

Outputs

• Annotated frames in the processed-frames directory
• A compiled MP4 detection video
• *.csv and *.xlsx reports containing per-detection coordinates, damage type, severity grade, distance-from-camera, VRI counts, and inter-VRI distances

Author

Usama Bin Ejaz — AI/ML Engineer | Data Scientist | Founder, UBE Labs ("AI that ships.")

• GitHub: github.com/ube09
• Website: usamabinejaz.com