Handle comparison-only answers in structured evaluation

[ChathurangiShyalika]

Description

Some scenarios require comparing two items where the agent might not explicitly provide the expected key. We need a clear rule for when to score an answer as an abstention versus when it should be treated as a clarification or partial response.

Why

Current evaluation can be ambiguous when the answer is based on comparison but does not use the exact expected key format.

Goal

Define a consistent policy for comparison-only scenarios so scoring is stable and predictable.

[jatinkumar300403]

Hi @ChathurangiShyalika, I would like work on this issue. Please find below the design and solution for this issue

Problem faced
In scenarios, where agents are asked to compare two things, agent might reply normally in a English language instead of giving a structured JSON answer with exact matching keys. Right now, the scorer can't tell the difference between- 1) the agent refusing to answer (abstention), 2) the agent giving a valid comparison in words instead of JSON and 3) the agent giving a partial/wrong answer. Hence, all three get scored as 0, which is misleading.

Proposed solution

1. Adding an ‘abstained’ flag which will clearly mark when the agent chose not to answer, different from getting it wrong.
2. Making the scorer smarter for comparisons as when a scenario is a comparison type, allow the scorer to check if the agent picked the right item even if the format doesn't match exactly.
3. Setting up a clear rule – when the agent gives a non-empty answer but no keys match, then check if it's a valid comparison before marking it as a failure first.

How it works
After the scorer tries to match keys and finds zero matches, it checks if the agent actually wrote something meaningful.
• If yes, and valid comparison – give partial credit
• If yes, but invalid – score 0.0(note – not be marked as ‘abstained’)
• If no/empty answer – mark it as ‘abstained = True’

Files to work on
• ‘models.py’ to add ‘abstained’ field,
• ‘static_json.py’ to add logic to detect abstention vs. comparison answers
• and test_ static_json_scorer.py’ to add tests for 3 cases –wrong, comparison(valid) and abstention

If it's fine, would like generate a PR.

[ChathurangiShyalika]

Hi @jatinkumar300403, please go ahead. Thank you

[jatinkumar300403]

@ChathurangiShyalika, I have raised a PR. I request you to please check it and let me know if there is anything to modify/update. Thank you!