I am trying to reproduce the RepoEval retrieval numbers from the cAST paper (GIST-base, 75.0 cAST vs 70.7 fixed-size) as an independent check, using astchunk 0.1.0 — verified byte-identical to e8a4f85 — with max_chunk_size=2000, language="python", metadata_template="coderagbench-repoeval", and CodeRAG-Bench's retrieval/create/repoeval_repo.py corpus at f9e100c. Two things I cannot resolve from the paper, and I would rather ask than guess.
1. Which metric is "Recall@5"? CodeRAG-Bench marks every 50-line window overlapping [context_start_lineno, line_no + 1) as relevant. With 64% of tasks starting at line 0 and a median target span of 135 lines, a query carries ~15.3 relevant windows, so trec_eval recall.5 — what BEIR/pytrec_eval computes in eval_beir_sbert_canonical.py — is bounded above by 5/15.3 ≈ 0.486 (0.554 under the older 20/2 defaults). The reported 0.707 and 0.750 sit above that bound, so I assume the reported quantity is something else.
A hit-rate reading ("any relevant window in the top 5") does land in the right region, but only with the query construction that is commented out in repo2code:
# text = '\n'.join(task["prompt"].split('\n')[-2:])
text = task["prompt"]With the two-line query I measure a fixed-size baseline of 0.779 per-repo / 0.751 pooled, against your reported 0.707. With the live full-prompt query it is 0.991, which is clearly not it. Is the two-line query what you used, and is the metric hit rate rather than trec recall?
2. The Appendix A.2 score mapping. The appendix describes assigning each line the score of its AST chunk and aggregating line scores back onto the baseline chunks, but not the aggregation function. I swept max, mean, and sum. Under the two-line query all three come out well below the fixed-size baseline (−34.8, −27.5, −35.3 hit-rate points); under the full-prompt query max gives +2.3 against your +4.3. So I am fairly sure I have the aggregation wrong.
Would you be willing to share the evaluation/mapping script? Even a sketch of the aggregation step would let me finish the check. Related: issue #3 mentions replacing the chunking in CodeRAG-Bench's /retrieval/create, and #7 notes the context/content key mismatch — both were helpful, but neither pins these two points down.
Related upstream report on the corpus itself: code-rag-bench/code-rag-bench#14 (the function-level split builds 5 repositories, not 6).
To be clear about intent: this is an independent reproduction attempt, not a challenge to the method. Everything above may simply be me running the wrong configuration, and I would like to correct it before writing anything up.
I am trying to reproduce the RepoEval retrieval numbers from the cAST paper (GIST-base, 75.0 cAST vs 70.7 fixed-size) as an independent check, using
astchunk0.1.0 — verified byte-identical toe8a4f85— withmax_chunk_size=2000,language="python",metadata_template="coderagbench-repoeval", and CodeRAG-Bench'sretrieval/create/repoeval_repo.pycorpus atf9e100c. Two things I cannot resolve from the paper, and I would rather ask than guess.1. Which metric is "Recall@5"? CodeRAG-Bench marks every 50-line window overlapping
[context_start_lineno, line_no + 1)as relevant. With 64% of tasks starting at line 0 and a median target span of 135 lines, a query carries ~15.3 relevant windows, so trec_evalrecall.5— what BEIR/pytrec_eval computes ineval_beir_sbert_canonical.py— is bounded above by5/15.3 ≈ 0.486(0.554 under the older 20/2 defaults). The reported 0.707 and 0.750 sit above that bound, so I assume the reported quantity is something else.A hit-rate reading ("any relevant window in the top 5") does land in the right region, but only with the query construction that is commented out in
repo2code:With the two-line query I measure a fixed-size baseline of 0.779 per-repo / 0.751 pooled, against your reported 0.707. With the live full-prompt query it is 0.991, which is clearly not it. Is the two-line query what you used, and is the metric hit rate rather than trec recall?
2. The Appendix A.2 score mapping. The appendix describes assigning each line the score of its AST chunk and aggregating line scores back onto the baseline chunks, but not the aggregation function. I swept
max,mean, andsum. Under the two-line query all three come out well below the fixed-size baseline (−34.8, −27.5, −35.3 hit-rate points); under the full-prompt querymaxgives +2.3 against your +4.3. So I am fairly sure I have the aggregation wrong.Would you be willing to share the evaluation/mapping script? Even a sketch of the aggregation step would let me finish the check. Related: issue #3 mentions replacing the chunking in CodeRAG-Bench's
/retrieval/create, and #7 notes thecontext/contentkey mismatch — both were helpful, but neither pins these two points down.Related upstream report on the corpus itself: code-rag-bench/code-rag-bench#14 (the function-level split builds 5 repositories, not 6).
To be clear about intent: this is an independent reproduction attempt, not a challenge to the method. Everything above may simply be me running the wrong configuration, and I would like to correct it before writing anything up.