For benchmarks like PubMedQA, we need some way to restrict the valid golden responses to a static list (e.g. "Yes", "No", and "Maybe"). To do this in custom_nocode benchmarks, we should add a label style so that we can tell the CLI to select one value from a fixed set of labels, even when the dataset does not contain columns that indicate the selection of choices.
With this style = "label" feature, configuration for PubMedQA might look like the following:
[benchmarks.pubmedqa]
type = "custom_nocode"
dataset = {
name = "quantiles/PubMedQA",
config_name = "pqa_labeled",
split = "train",
revision = "<pinned-revision>"
}
model = "random"
prompt_template_file = "prompts/pubmedqa.txt"
[benchmarks.pubmedqa.style]
type = "label"
golden_column = "final_decision"
labels = ["yes", "no", "maybe"]We also might want eval definitions to support custom parsing schemes, with as parser key. For example, this:
parser = { type = "first_tokens", max_chars = 5 }Would be a simpler parser that throws out all characters in the response after the first 5, then uses those for comparison to the golden answer. It could also do other normalization like lower-casing etc...
For benchmarks like PubMedQA, we need some way to restrict the valid golden responses to a static list (e.g.
"Yes","No", and"Maybe"). To do this incustom_nocodebenchmarks, we should add alabelstyle so that we can tell the CLI to select one value from a fixed set of labels, even when the dataset does not contain columns that indicate the selection of choices.With this
style = "label"feature, configuration for PubMedQA might look like the following:We also might want eval definitions to support custom parsing schemes, with as
parserkey. For example, this:Would be a simpler parser that throws out all characters in the response after the first 5, then uses those for comparison to the golden answer. It could also do other normalization like lower-casing etc...