Potential Issue with Lambada evaluation

[iloverdl]

Hi team,
I noticed that in the official evaluation code you release, for Lambada dataset, the correct answer was counted twice at https://github.com/HKUNLP/DiffuLLaMA/blob/main/evaluation/eval-diffugpt.py#L75 and https://github.com/HKUNLP/DiffuLLaMA/blob/main/evaluation/eval-diffugpt.py#L79. This potentially leads to doubling the accuracy metric?

In addition, the official accuracy number reported in GPT2 paper on Lambada dataset was 45.99 for GPT2-S, but in your paper it was 25.9. For GPT2-M the official number is 55.48 but in your paper it was 37.7.

Could you please clarify

• why in the official evaluation code, the number of correct case was counted twice?
• why the accurary for baselines were much lower than the official paper?

Thanks!

[summmeer]

Thank you for pointing this out! I will carefully check this. Any update will be posted here.

[summmeer]

Hi @iloverdl Considering the setting of original GPT2 paper could be different, so we don't use the numbers in the paper. Using lm-harness we have score of 32 & 42 for GPT-S & M. Considering the implementation details could be different and for fair comparision, we still use the numbers computed from eval_llm.py. For DiffuGPT/DiffuLLaMA model's eval, you're right! The repeat two lines should be considered as a bug. The updated numbers:

DiffuGPT-S: 21.6
DiffuGPT-M: 30.3
DiffuLLaMA: 53.9

The numbers are not surpassing their AR counterparts for this task, but other conclusions in paper still hold. We will update this results on readme of this repo and a new arxiv version of this paper.