While running conv-26 through our benchmark pipeline, we did a full audit of all 152 ground truths against the source turns. On top of the errors already reported in #27, we found two more incorrect answers and several structural issues worth flagging.
Ground truth errors
q57 — "What symbols are important to Caroline?"
Ground truth says "Rainbow flag, transgender symbol". The cited turn D14:15 says "The rainbow flag mural is important to me ... The eagle symbolizes freedom and pride." The second cited turn D4:1 points to a necklace image captioned as a cross and a heart. None of the cited evidence mentions a transgender symbol. The supported answer would be the rainbow flag mural and the eagle.
q95 — "What is Melanie's hand-painted bowl a reminder of?"
The question attributes the bowl to Melanie, but the cited turn D4:5 is Caroline speaking in first person: "I've got some other stuff with sentimental value, like my hand-painted bowl ... it reminds me of art and self-expression." The answer text itself (art and self-expression) is fine, but the question has the wrong speaker.
Ambiguous list questions
Several conv-26 questions ask broad questions like "What activities does Melanie partake in?" or "What LGBTQ+ events has Caroline participated in?" where the ground truth lists a subset of valid items but the conversation supports more. Affected: q16, q25, q33, q35, q39, q40, q44, q49, q52, q71.
These are tricky to score fairly because any correct system that retrieves more of the conversation will include additional valid items and might get penalized for not matching the expected subset exactly. It would help to either narrow the questions or make the ground truth exhaustive.
Underspecified questions
Eight questions ask counterfactual, political, religious, or future-intent questions without enough conversational evidence to pin down a single answer: q15, q23, q28, q31, q51, q60, q78, q82. For example, q31 asks whether Melanie would be considered LGBTQ and has empty evidence_turn_ids. These work as open-ended discussion probes but not great as single-answer benchmark items.
Metadata issues
- q31 and q47 have empty
evidence_turn_ids
- q38 has two evidence IDs packed into one string ("D8:6; D9:17") instead of two list elements
- q77 is a temporal "When..." question tagged as category 1 instead of category 2
- Several questions cite a setup turn instead of the answer-bearing turn (e.g. q2, q50, q73)
While running conv-26 through our benchmark pipeline, we did a full audit of all 152 ground truths against the source turns. On top of the errors already reported in #27, we found two more incorrect answers and several structural issues worth flagging.
Ground truth errors
q57 — "What symbols are important to Caroline?"
Ground truth says "Rainbow flag, transgender symbol". The cited turn D14:15 says "The rainbow flag mural is important to me ... The eagle symbolizes freedom and pride." The second cited turn D4:1 points to a necklace image captioned as a cross and a heart. None of the cited evidence mentions a transgender symbol. The supported answer would be the rainbow flag mural and the eagle.
q95 — "What is Melanie's hand-painted bowl a reminder of?"
The question attributes the bowl to Melanie, but the cited turn D4:5 is Caroline speaking in first person: "I've got some other stuff with sentimental value, like my hand-painted bowl ... it reminds me of art and self-expression." The answer text itself (art and self-expression) is fine, but the question has the wrong speaker.
Ambiguous list questions
Several conv-26 questions ask broad questions like "What activities does Melanie partake in?" or "What LGBTQ+ events has Caroline participated in?" where the ground truth lists a subset of valid items but the conversation supports more. Affected: q16, q25, q33, q35, q39, q40, q44, q49, q52, q71.
These are tricky to score fairly because any correct system that retrieves more of the conversation will include additional valid items and might get penalized for not matching the expected subset exactly. It would help to either narrow the questions or make the ground truth exhaustive.
Underspecified questions
Eight questions ask counterfactual, political, religious, or future-intent questions without enough conversational evidence to pin down a single answer: q15, q23, q28, q31, q51, q60, q78, q82. For example, q31 asks whether Melanie would be considered LGBTQ and has empty evidence_turn_ids. These work as open-ended discussion probes but not great as single-answer benchmark items.
Metadata issues
evidence_turn_ids