ToolBench Dataset

[redwyd]

Hi,

The repository mentions downloading the complete ToolBench dataset (including queries, tools, and reference answers) from the official Google Drive: https://drive.google.com/drive/folders/1TysbSWYpP8EioFu9xPJtpbJZMLLmwAmL .

However, after downloading the files, I am not sure which ones correspond to the queries, tools, and reference answers. Could you please clarify how to construct the final dataset from the downloaded files and which files should be used for each component?

Thanks!

[Tooooa]

Thanks for the question. The downloaded ToolBench files do not need to be merged into a single “final dataset” file. Please keep the original ToolBench data/ directory structure and place it under:

experiments/toolbench/data/data/

The components are mapped as follows:

• Queries:

  • Evaluation queries: data/test_instruction/*.json
  • Full instruction/query files: data/instruction/*_query.json

• Tools:

  • Tool/API definitions and tool environment files: data/toolenv/tools/

• Reference answers:

  • data/answer/G1_answer/
  • data/answer/G2_answer/
  • data/answer/G3_answer/

• Evaluation query IDs:

  • data/test_query_ids/*_test_query_ids.json

For example, with the default split G1_instruction, the query file is:

experiments/toolbench/data/data/test_instruction/G1_instruction.json

the tools are loaded from:

experiments/toolbench/data/data/toolenv/tools/

and the reference answers are taken from:

experiments/toolbench/data/data/answer/G1_answer/

In general, the answer directory is selected by the split prefix:

• G1_* → answer/G1_answer
• G2_* → answer/G2_answer
• G3_* → answer/G3_answer

So the expected local layout is:

experiments/toolbench/data/data/
├── instruction/
├── test_instruction/
├── test_query_ids/
├── toolenv/
│   └── tools/
└── answer/
    ├── G1_answer/
    ├── G2_answer/
    └── G3_answer/

The retrieval/, retrieval_test_query_ids/, and toolllama_G123_dfs_*.json files are part of the upstream ToolBench release, but they are not required for constructing the basic AgentMark ToolBench evaluation dataset unless you are specifically running retriever/open-domain or ToolLLaMA training workflows.