It already exists in the registry server, and should be executed from there rather than the code.
We should make sure that the following are fixed first:
And, after #59 is fixed, the remote benchmark definition should use the new label style rather than the existing exact_match style. Doing this will improve the percentage of parsed responses and thus improve benchmark performance.
It already exists in the registry server, and should be executed from there rather than the code.
We should make sure that the following are fixed first:
qt resume, ensure provenance and reproducibility for remote evals #63labelscoring style tocustom_nocodebenchmarks #59And, after #59 is fixed, the remote benchmark definition should use the new
labelstyle rather than the existingexact_matchstyle. Doing this will improve the percentage of parsed responses and thus improve benchmark performance.