how do you generate response for each test LLM in the paper

[yaoqi-zd]

hi, I find these actually exists two items: "input" and "instruction" for each sample in infobench.json file, like the following

{"id": "user_oriented_task_172", "input": "Crew Members", "category": "Indeed", "instruction": "Provide a list of the skills that may help you find the job.", "decomposed_questions": ["Is the generated text a list of skills?", "Are the skills in the generated text relevant to find the job in the given input?"], "subset": "Easy_set", "question_label": [["Format", "Content"], ["Content"]]}

do you directly concatenate the 'input' and 'instruction' as prompt to test the LLM?

[songxiaoshuai]

I have the same issue; sometimes I find that directly connecting 'instruction' + \n +'input' isn't very smooth.

[KaiQiangSong]

hi, I find these actually exists two items: "input" and "instruction" for each sample in infobench.json file, like the following

{"id": "user_oriented_task_172", "input": "Crew Members", "category": "Indeed", "instruction": "Provide a list of the skills that may help you find the job.", "decomposed_questions": ["Is the generated text a list of skills?", "Are the skills in the generated text relevant to find the job in the given input?"], "subset": "Easy_set", "question_label": [["Format", "Content"], ["Content"]]}

do you directly concatenate the 'input' and 'instruction' as prompt to test the LLM?

@yaoqi-zd @songxiaoshuai
We also notice this issue.
This issue is for the easy set in InfoBench which is inherited from the following work:

SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions
https://arxiv.org/pdf/2212.10560

To better reproduce the results, we didn't change their original settings.
However, when we created the hard set manually, we tried to avoid this issue.