Collab-Overcooked

We propose a new LLM-powered Multi-Agent System (LLM-MAS) benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. Collab-Overcooked extends existing benchmarks from two novel perspectives. First, it provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. Second, it introduces a spectrum of process-oriented evaluation metrics to assess the fine-grained collaboration capabilities of different LLM agents, a dimension often overlooked in prior work.

⚠️ Upcoming Code Restructure for Improved Clarity

We're preparing to restructure our code to significantly improve its readability and logical organization (expected to take 1-2 months). This update is designed to make it much simpler for researchers to follow and understand.

Important points for you:

• No impact on current experiments: This update will not affect any experimental results. If you're currently testing with Collab-Overcooked, you can continue using it without interruption.

• For incremental work: If you plan to build incrementally on Collab-Overcooked and have questions about the current code, please pay close attention to our upcoming updates.

We believe these changes will greatly enhance your experience working with our codebase!

Getting Started

Install

We recommend using the anaconda management environment. Python 3.8 is recommended for this project.

• Install requirements

  • Directly do:

    conda create -n collab-overcooked python=3.8
    conda activate collab-overcooked
    
    pip install -r requirements.txt
    conda install mpi4py==3.1.4  # pip install often fails
    

• Install the game environment overcooked_ai locally.

  cd ./lib/overcooked_ai
  pip install -e .
  

  Notes: Overcooked-AI is a benchmark environment for fully cooperative human-AI task performance, based on the wildly popular video game Overcooked. We made certain modifications based on Overcooked-AI.

Quick Test

The easiest way to test whether the environment is installed correctly is to use gpt-3.5-turbo to test after filling in the openai api secret key.

• Fill in the OpenAI API key at Collab-Overcooked/src/openai_key.txt
• Run the following commands

  cd Collab-Overcooked/src
  python main.py --horizon 3 --order boiled_egg
  

If you can output the environmental visualization map normally, the agents' normal output content, and run through 3 time steps without any errors, then your environment and agent configuration are successful. This will take you about 1.5 minutes, depending on the speed of your connection to OpenAI.

Configure the Open-source LLMs

We recommend using vLLM for local deployment of open-source LLMs.

Evaluation

The evaluation scripts are provided in the "Collab-Overcooked/src" folder. The evaluation process consists of three sequential scripts:

• evaluation.py: Evaluates the environment's output, calculates the metrics for each task, and stores the results in the corresponding task folder.
• organize_result.py: Summarizes the metrics of each task into the statistics_data.csv file.
• convert_result.py: Computes the metrics for each complexity level and stores the results in the converted_data.csv file.

The final evaluation results are stored in converted_data.csv, with the following key fields:

• mean_f1_agent_*: The F1 score for agent * computed using the TES function, considering both correctly matched actions and redundant actions.
• mean_similarity_agent_*: The similarity between the actions generated by agent * and the RATs (Reference Action Templates).
• mean_redundancy_agent_*: The redundancy between the actions generated by agent * and the RATs.
• initiate_collaboration: The ability of the LLM-MAS (Multi-Agent System) to initiate collaboration.
• respond_collaboration: The ability of the LLM-MAS to respond to collaboration.

Experiment Commands

Free-form:

COLLAB_USE_DISCRETE_COMM=0 python main.py --horizon 30 --order boiled_egg \
  --retrival_method recent_k --K 1 --gpt_model gpt-3.5-turbo-0125

Reduced:

COLLAB_USE_DISCRETE_COMM=1 python main.py --horizon 30 --order boiled_egg \
  --retrival_method recent_k --K 1 --gpt_model gpt-3.5-turbo-0125

Always Talk:

COLLAB_USE_DISCRETE_COMM=0 python main.py --horizon 30 --order boiled_egg --retrival_method recent_k --K 1 --gpt_model gpt-3.5-turbo-0125 --comm_baseline always

Pruning Only:

COLLAB_USE_DISCRETE_COMM=0 python main.py --horizon 30 --order boiled_egg --retrival_method recent_k --K 1 --gpt_model gpt-3.5-turbo-0125 --comm_baseline pruning_only --pruning_similarity 0.8

Modify the Environment

The environment settings and logic are stored in the Collab-Overcooked/lib/overcooked_ai folder. Layout files are stored in the data/layouts folder, while the environment logic is in the mdp folder.

• To add more tasks (such as additional recipes or ingredients), you only need to modify the corresponding layout files.
• If you need to add new interactive elements, ensure that you update the environment logic accordingly.

Reference

@inproceedings{zhang2024proagent,
  title={Proagent: building proactive cooperative agents with large language models},
  author={Zhang, Ceyao and Yang, Kaijie and Hu, Siyi and Wang, Zihao and Li, Guanghe and Sun, Yihang and Zhang, Cheng and Zhang, Zhaowei and Liu, Anji and Zhu, Song-Chun and others},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={16},
  pages={17591--17599},
  year={2024}
}

@inproceedings{carroll2019utility,
 title={On the Utility of Learning About Humans for Human-AI Coordination},
 author={Carroll, Micah and Shah, Rohin and Ho, Mark K and Griffiths, Tom and Seshia, Sanjit and Abbeel, Pieter and Dragan, Anca},
 booktitle={Advances in Neural Information Processing Systems},
 pages={},
 volume={32},
 year={2019},
}