🚗 VehicleMemBench

An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents

📖 Paper | 🚀 GitHub | 🤗 Dataset

---

Official codebase of VehicleMemBench, a benchmark for evaluating whether agents can recover multi-user preferences from long interaction histories, resolve preference conflicts, and invoke vehicle tools to reach the correct final environment state.

📊 Evaluation Protocols

A. Model Evaluation

This setting evaluates the backbone model under basic memory constructions, such as:

• Raw History (none): Let the models predict without any history information.
• Gold Memory (gold): Provides ground-truth latest user preferences directly, representing the theoretical performance upper bound.
• Recursive Summarization (summary): Compresses history into hierarchical summaries to test reasoning over distilled information.
• Key-Value Store (key_value): Organizes preferences into structured attribute-value pairs to assess precise, indexed retrieval.

B. Memory-System Evaluation

This setting first ingests dialogue history into a memory system and then evaluates whether the agent can retrieve useful memory and execute the correct vehicle actions. The evaluated memory systems include:

• Gold Memory
• Recursive Summarization
• Key-Value Store
• Memobase
• LightMem
• Mem0
• MemOS
• Supermemory

C. Evaluation Metrics

Main metrics used in this repo include:

• Exact State Match: A strict binary success metric that requires the final environment state of models to perfectly match the ground truth.
• Field- and Value-level Metrics: Fine-grained Precision, Recall, and F1 scores that assess whether the correct system fields were modified and if their values were predicted accurately.
• Average Prediction/Tool-call Statistics: The average number of tool calls that measures the execution cost and system overhead per task.

🗂️ Repository Structure

VehicleMemBench/
├── benchmark/      # QA files and history logs
├── environment/    # Vehicle simulator and module definitions
├── evaluation/     # Model and memory-system evaluation code
├── figure/         # README figure assets
├── scripts/        # Example runner scripts
├── requirements.txt
└── README.md

⚙️ Setup

1) Environment

• Python 3.12 recommended

2) Install Dependencies

conda create -n VehicleMemBench python=3.12
conda activate VehicleMemBench
pip install -r requirements.txt

3) Configure API Keys

Set the base model variables before evaluation:

export LLM_API_BASE="..."
export LLM_API_KEY="..."
export LLM_MODEL="..."

For memory-system evaluation, set the backends you want to use:

export MEM0_API_KEY="..."
export MEMOS_API_URL="..."
export MEMOS_API_KEY="..."
export SUPERMEMORY_API_KEY="..."
export MEMOBASE_API_URL=""
export MEMOBASE_API_KEY="..."

🚀 Quick Start

Run commands from the repository root.

A. Model Evaluation

bash scripts/model_test.sh

B. Memory-System Evaluation

bash scripts/memorysystem_test.sh

🧾 Benchmark Data

Released benchmark inputs in this repo mainly include:

• long interaction histories in benchmark/history
• executable QA files in benchmark/qa_data

Typical outputs are written to log/ or memory_system_log/ during evaluation.

🙏 Acknowledgments

This project references and adapts the VehicleWorld repository in building the executable in-vehicle environment.

📝 Citation

If you use VehicleMemBench in your research, please cite:

@misc{chen2026vehiclemembenchexecutablebenchmarkmultiuser,
      title={VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents}, 
      author={Yuhao Chen and Yi Xu and Xinyun Ding and Xiang Fang and Shuochen Liu and Luxi Lin and Qingyu Zhang and Ya Li and Quan Liu and Tong Xu},
      year={2026},
      eprint={2603.23840},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.23840}, 
}

📄 License

This project is licensed under the Apache-2.0 License.