Benchmark results

[sohviluukkonen]

Hi,

Thank you for open-sourcing the evaluation data! Could you please also release the benchmark results for the different models you compared? Specifically, the values shown in Figure 2 as well as the results for tasks not included in that figure.

Thanks in advance!

[zjlrock777]

the benchmark seems not complete. for example retrosynthesis, solution is the same as question. can you correct this please, thanks!

[jamesbraza]

Hi @sohviluukkonen thanks for the request, I agree and will bump this internally. How do you feel about us adding something like Table 2 from https://arxiv.org/pdf/2407.10362 to the ether0 paper?

---

Hi @zhoujieli thanks for the report, though it's unrelated to this one. Can you open a separate issue for this? Also you'd need to include the question ID you're looking at.

[sohviluukkonen]

Thanks for picking this up. Something like Table 2 would be great. Even better, would be to have it in a computer readable format (e.g., csv).

[sidnarayanan]

@zhoujieli that is expected. The solution column in the dataset defines the reward function and any arguments that need to be forwarded to the reward function. In the case of retrosynthesis, the extra argument to forward is the expected product (see here).

The solution column is parsed by the accuracy_reward reward function, or you can follow the code snippet at the bottom of this section: https://github.com/Future-House/ether0/tree/main?tab=readme-ov-file#benchmark

If you have any follow-up questions, feel free open a new ticket!

[zjlrock777]

@sidnarayanan @jamesbraza Thanks a lot for the responses! I will check out the reward function and open a new ticket.