Video Understanding Harness

This repository contains the experiment code for testing an open weight reasoner on long video question answering

The system preprocesses each video into timestamped text artifacts then lets a tool calling reasoner inspect targeted video windows through an open weight vision language model

Experiment scope

• preprocessing/ prepares audio transcripts batch summaries and local video artifacts
• harness/ runs the tool calling reasoner over those artifacts
• benchmarks/ loads Hugging Face datasets provisions their videos and scores answers
• modal_app.py serves the reasoner and perception models with vLLM and runs the benchmark on Modal

Frontend and publication assets are intentionally outside the experiment

Models

The default configuration uses

• Qwen/Qwen2.5-32B-Instruct for reasoning and preprocessing summaries
• Qwen/Qwen2.5-VL-72B-Instruct for visual perception
• Groq Whisper for speech transcription

Both Qwen model names can be changed through HF_REASONER_MODEL and HF_PERCEPTION_MODEL

Install

python -m venv .venv
source .venv/bin/activate
pip install -e preprocessing
pip install -e 'harness[bench]'
pip install modal remotezip

Set GROQ_API_KEY for transcription and HF_TOKEN when a Hugging Face dataset requires authentication

Run the model servers on Modal

modal setup
modal secret create vu3-api-keys \
  GROQ_API_KEY=... \
  HF_TOKEN=... \
  VLLM_BEARER_TOKEN=...
modal deploy modal_app.py

Create the vu3-endpoints Modal secret with the deployed endpoint values

HARNESS_REASONER_BASE_URL=https://your-reasoner-endpoint
HARNESS_PERCEPTION_BASE_URL=https://your-perception-endpoint
HARNESS_REASONER_API_KEY=your-vllm-token
HARNESS_PERCEPTION_API_KEY=your-vllm-token

Confirm both servers before starting an evaluation

modal run modal_app.py::vlm_smoke

Run benchmarks

Run a small validation before scaling the evaluation

modal run modal_app.py::validate --dataset egoschema --limit 10
modal run modal_app.py::run_all --n 10 --datasets egoschema,video_mme

The benchmark records answer accuracy wall time tool calls frames clips and perception calls

Supported dataset loaders are

• EgoSchema
• Video MME
• WorldQA

Local workflow

Point the harness at OpenAI compatible reasoner and perception endpoints

export HARNESS_REASONER_BASE_URL=http://localhost:8000
export HARNESS_PERCEPTION_BASE_URL=http://localhost:8001
export HARNESS_REASONER_MODEL=Qwen/Qwen2.5-32B-Instruct
export HARNESS_PERCEPTION_MODEL=Qwen/Qwen2.5-VL-72B-Instruct
export VU3_ARTIFACTS_LOCAL_ROOT=./data/vu3_cache

harness preprocess /path/to/video.mp4 --user-id demo
harness ask VIDEO_ID "What happens after the speaker enters" --user-id demo