Hi SCBench team,
I have developed my own coding agent and would like to evaluate it using the official SlopCodeBench benchmark.
I have two questions:
-
Running my own agent on SCBench
Is it possible to run my custom agent against the existing SlopCodeBench benchmark tasks/test cases and evaluate it using the official
SCBench evaluation pipeline?
If so, could you please point me to the recommended integration/implementation approach for adding a custom agent?
-
Submitting results to the official leaderboard
After running my agent on the benchmark and obtaining the evaluation results, what is the official process for submitting those results so
that my agent can be included as an entry on the SCBench official leaderboard
?
Specifically, should I:
Open a PR in this repository?
Submit the evaluation artifacts/results through another repository?
Contact the maintainers directly?
Or follow another submission process?
I would like to make sure that I run the benchmark using the correct configuration and produce all the required artifacts so that the result can be considered for the official leaderboard.
Thanks!
Hi SCBench team,
I have developed my own coding agent and would like to evaluate it using the official SlopCodeBench benchmark.
I have two questions:
Running my own agent on SCBench
Is it possible to run my custom agent against the existing SlopCodeBench benchmark tasks/test cases and evaluate it using the official
SCBench evaluation pipeline?
If so, could you please point me to the recommended integration/implementation approach for adding a custom agent?
Submitting results to the official leaderboard
After running my agent on the benchmark and obtaining the evaluation results, what is the official process for submitting those results so
that my agent can be included as an entry on the SCBench official leaderboard
?
Specifically, should I:
Open a PR in this repository?
Submit the evaluation artifacts/results through another repository?
Contact the maintainers directly?
Or follow another submission process?
I would like to make sure that I run the benchmark using the correct configuration and produce all the required artifacts so that the result can be considered for the official leaderboard.
Thanks!