Problem about the recall@20

[srymaker]

What a great benchmark!

I have a problem about the recall@20 metric. In the article, you said that recall@k = 20 is used, and the VSS + LLM Reranker baseline used top-20 results from VSS. I'm wondering if VSS and VSS+LLM rerank shouldn't yield identical recall@20 metrics, since LLM reranking is performed on the top-20 results retrieved by VSS.

[Wuyxin]

Hi,

I just checked our previous experiment logs, it appears that we actually used top-25 results (llm_topk=25) from VSS on synthesized dataset (Table 6 random 10%). Sorry for the confusion. I will fix this typo in the new arxiv version.

And yes, on Table 7, Recall@20 of ada-002 and reranks should be the same, which is the case for the first two datasets. For the last one dataset, I am still checking on why there's a 0.52% difference. It might because we later removed a few questions on STARK-PRIME and didn't update the ada-002 score.. Luckily it doesn't impact the general rankings. I am rerunning the results and will update this as well. Thank you.

[srymaker]

Thank u so much for your answers