Questions about the evaluation script

[Zzzzzxl-git]

Hello, may I ask if your evaluation script uses the original evaluation script eval_beir_sbert.py from CodeRagBench? Or have you made any other improvements to the evaluation script? Because I adopted the improved method proposed in the paper for document chunking, yet the evaluation results are worse than those of fixed chunking.

[yilinjz]

Thanks for the question.

Off the top of my head, I remembered there was a key mismatch between CodeRagBench (which uses ‘context’) and astchunk (which uses ’content’) for the code chunks. I’d suggest trying to print out the generation prompt and see if the retrieved chunks are properly passed in (or if the numbers are significantly worse then I think it’s because of this).

[Zzzzzxl-git]

Okay, thank you for your reply. I will try again.