MASH-VLM

Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations

CVPR 2025 Highlight

Kyungho Bae^1,2*, Jinhyung Kim¹, Sihaeng Lee¹, Soonyoung Lee¹, Gunhee Lee^1†, and Jinwoo Choi^2†

¹ LG AI Research     ² Kyung Hee University

^† denotes corresponding authors

---

Introduction

MASH-VLM addresses the critical problem of action-scene hallucination in Video-LLMs, where models incorrectly predict actions from scene context or infer scenes from observed actions.

To address this issue, we propose a novel framework that disentangles spatial and temporal representations, enabling more reliable reasoning over both scene context and dynamic actions. As a result, MASH-VLM effectively reduces hallucinated predictions and achieves strong performance across multiple video understanding benchmarks.

We also introduce UNSCENE, a benchmark specifically designed to evaluate action-scene hallucination in Video-LLMs.

---

Release

• Benchmark (UNSCENE): Released (see below)

---

UNSCENE Benchmark

UNSCENE evaluates whether Video-LLMs hallucinate non-existent actions or scenes. It consists of 1,320 video clips across two categories:

Category	Videos	Description
Scene-only	601	Videos with scene content but no salient action. Tests if models hallucinate actions.
Misalignment	719	Videos where action and scene are misaligned. Tests if models hallucinate incorrect scenes.

Videos

Videos are sourced from YouTube. We provide metadata for downloading in data/video_metadata.csv:

filename,youtube_id,start_sec,end_sec,youtube_url,category
6kzqtotkAXg_000367_000380.mp4,6kzqtotkAXg,367,380,https://www.youtube.com/watch?v=6kzqtotkAXg,misalignment

Each row contains the YouTube video ID and the start/end timestamps (in seconds) for the video clip.

Annotations

Ground truth annotations are in annotation/:

File	Count	Description
misalignment_action_positive.json	719	Positive questions about actions (misalignment videos)
misalignment_action_negative.json	719	Negative questions about actions (misalignment videos)
misalignment_scene_positive.json	719	Positive questions about scenes (misalignment videos)
misalignment_scene_negative.json	719	Negative questions about scenes (misalignment videos)
sceneonly_action_positive.json	601	Positive questions about actions (scene-only videos)
sceneonly_action_negative.json	601	Negative questions about actions (scene-only videos)

Annotation format:

{
  "video": "6kzqtotkAXg_000367_000380.mp4",
  "question": "Is there anyone catching fish in the video?",
  "type": "positive"
}

Evaluation

1. Binary (Yes/No) Evaluation

The model is asked a Yes/No question (e.g., "Is there anyone playing soccer in the video?") and its answer is checked against the ground truth.

Two metrics are reported:

• Per-category accuracy: Accuracy for each of the 6 task types independently.
• Both accuracy: A video is correct only when the model answers both the positive and negative question correctly.

python evaluation/eval_binary.py --pred_dir <path_to_prediction_jsons>

Each prediction JSON file should have the following format:

{
  "video_path": "<video_filename>.mp4",
  "question": "Is there anyone playing soccer in the video?",
  "pred": "No, there is no one playing soccer.",
  "task_type": "Sceneonly Action Negative",
  "q_type": "negative",
  "flag": true
}

The video_path (or video_name) field is used to pair positive and negative predictions of the same video for the Both accuracy metric.

See examples/sample_prediction_binary.json for full examples.

2. GPT-based Open-ended Evaluation

The model is prompted with "Describe the video." and generates a free-form description. GPT-3.5-turbo then scores the description against ground truth labels:

• action_score (0-5): How well the predicted description matches the correct action label.
• scene_score (0-5): How well the predicted description matches the correct scene label.

Setup

pip install -r requirements.txt
export OPENAI_API_KEY="your-api-key-here"

Usage

python evaluation/eval_gpt.py --pred_dir <path_to_prediction_jsons>

Each prediction JSON filename must contain Sceneonly or Misalignment (case-sensitive) so that results can be aggregated per category (e.g., Sceneonly_001.json, Misalignment_Action_002.json).

Each prediction JSON file should have the following format:

{
  "pred": "The video shows a person walking on a mountain path...",
  "action_label": "no action",
  "scene_label": "mountain",
  "task_type": "Sceneonly Action Negative"
}

See examples/sample_prediction.json for full examples.

---

Directory Structure

.
├── README.md
├── requirements.txt
├── assets/
│   └── teaser.png
├── evaluation/
│   ├── eval_binary.py                     # Binary (Yes/No) evaluation
│   └── eval_gpt.py                        # GPT-based open-ended evaluation
├── annotation/                             # Ground truth annotations
│   ├── misalignment_action_positive.json
│   ├── misalignment_action_negative.json
│   ├── misalignment_scene_positive.json
│   ├── misalignment_scene_negative.json
│   ├── sceneonly_action_positive.json
│   └── sceneonly_action_negative.json
├── data/
│   └── video_metadata.csv                  # YouTube URLs and timestamps
└── examples/
    ├── sample_prediction.json              # Example prediction format (open-ended)
    └── sample_prediction_binary.json       # Example prediction format (binary)

---

Citation

If you find this work useful, please consider citing:

@inproceedings{bae2025mashvlm,
  title     = {MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations},
  author    = {Bae, Kyungho and Kim, Jinhyung and Lee, Sihaeng and Lee, Soonyoung and Lee, Gunhee and Choi, Jinwoo},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2025}
}

License

• Code: BSD-3-Clause-LG AI Research License (LICENSE)
• Dataset: Creative Commons Attribution-NonCommercial-NoDerivatives-ShareAlike 4.0 International License (CC-BY-NC-ND-SA-4.0)