Skip to content

Add a response_labels field to exact_match custom_nocode evals #70

Description

@arschles

After #67, the response parsing behavior of custom_nocode evaluations with the exact_match style will change. For example, the PubmedQA built-in did some normalization of responses that the stock exact_match runner will not. This issue brings that back.

We need to add an optional response_labels field exact_match evals. Its purpose would be to support classification-like evaluations where models may return labels with capitalization, punctuation, or short explanatory prefixes. When omitted, exact-match behavior can remain as it is now.

For PubMedQA, for example, the configuration could look like this:

[benchmarks.pubmedqa.style]
type = "exact_match"
golden_column = "final_decision"
response_labels = ["yes", "no", "maybe"]

When response_labels is provided as above, the runner should do the following:

  • Match labels case-insensitively
  • Ignore punctuation surrounding a label
  • Search the first five response tokens
  • Return the canonical label from response_labels
  • Compare the canonical parsed response with the golden value
  • Make response_labels part of the durable step hash (so that changing normalization rules doesn't incorrectly reuse cached exact-match results)
  • Normalize the golden value to the closest response_label, or fail validation if it does not match one of the labels

So, continuing with this example, response_labels = ["yes", "no", "maybe"] would parse as follows:

Model response Parsed response
yes yes
Yes. yes
"NO" no
The answer is maybe. maybe
I cannot determine this. Unparsed
one two three four five yes Unparsed

Also, when response_labels is provided, the demo model should select from the possible values, rather than just generating purely-random text as it does now.

Finally, after this is implementation, we need to add a new PubMedQA version to the benchmark registry (preferably with a version constraint since this is new functionality - see #69 for more) that includes this response_labels field.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions