After #67, the response parsing behavior of custom_nocode evaluations with the exact_match style will change. For example, the PubmedQA built-in did some normalization of responses that the stock exact_match runner will not. This issue brings that back.
We need to add an optional response_labels field exact_match evals. Its purpose would be to support classification-like evaluations where models may return labels with capitalization, punctuation, or short explanatory prefixes. When omitted, exact-match behavior can remain as it is now.
For PubMedQA, for example, the configuration could look like this:
[benchmarks.pubmedqa.style]
type = "exact_match"
golden_column = "final_decision"
response_labels = ["yes", "no", "maybe"]
When response_labels is provided as above, the runner should do the following:
- Match labels case-insensitively
- Ignore punctuation surrounding a label
- Search the first five response tokens
- Return the canonical label from response_labels
- Compare the canonical parsed response with the golden value
- Make
response_labels part of the durable step hash (so that changing normalization rules doesn't incorrectly reuse cached exact-match results)
- Normalize the golden value to the closest
response_label, or fail validation if it does not match one of the labels
So, continuing with this example, response_labels = ["yes", "no", "maybe"] would parse as follows:
| Model response |
Parsed response |
| yes |
yes |
| Yes. |
yes |
| "NO" |
no |
| The answer is maybe. |
maybe |
| I cannot determine this. |
Unparsed |
| one two three four five yes |
Unparsed |
Also, when response_labels is provided, the demo model should select from the possible values, rather than just generating purely-random text as it does now.
Finally, after this is implementation, we need to add a new PubMedQA version to the benchmark registry (preferably with a version constraint since this is new functionality - see #69 for more) that includes this response_labels field.
After #67, the response parsing behavior of
custom_nocodeevaluations with theexact_matchstyle will change. For example, the PubmedQA built-in did some normalization of responses that the stockexact_matchrunner will not. This issue brings that back.We need to add an optional
response_labelsfieldexact_matchevals. Its purpose would be to support classification-like evaluations where models may return labels with capitalization, punctuation, or short explanatory prefixes. When omitted, exact-match behavior can remain as it is now.For PubMedQA, for example, the configuration could look like this:
When
response_labelsis provided as above, the runner should do the following:response_labelspart of the durable step hash (so that changing normalization rules doesn't incorrectly reuse cached exact-match results)response_label, or fail validation if it does not match one of the labelsSo, continuing with this example,
response_labels = ["yes", "no", "maybe"]would parse as follows:Also, when
response_labelsis provided, the demo model should select from the possible values, rather than just generating purely-random text as it does now.Finally, after this is implementation, we need to add a new PubMedQA version to the benchmark registry (preferably with a version constraint since this is new functionality - see #69 for more) that includes this
response_labelsfield.