Skip to content

Custom Agent Evaluation and Leaderboard Submission #33

Description

@manishdixitbuilds

Hi SCBench team,

I have developed my own coding agent and would like to evaluate it using the official SlopCodeBench benchmark.

I have two questions:

  1. Running my own agent on SCBench
    Is it possible to run my custom agent against the existing SlopCodeBench benchmark tasks/test cases and evaluate it using the official
    SCBench evaluation pipeline?
    If so, could you please point me to the recommended integration/implementation approach for adding a custom agent?

  2. Submitting results to the official leaderboard
    After running my agent on the benchmark and obtaining the evaluation results, what is the official process for submitting those results so
    that my agent can be included as an entry on the SCBench official leaderboard
    ?

Specifically, should I:

Open a PR in this repository?
Submit the evaluation artifacts/results through another repository?
Contact the maintainers directly?
Or follow another submission process?

I would like to make sure that I run the benchmark using the correct configuration and produce all the required artifacts so that the result can be considered for the official leaderboard.

Thanks!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions