The Verification metric

[Tushar-R-Tyagi]

In some unique approaches e.g. where the user does not use any help of the agent, the verification formula is penalising the candidate. Verification logic is a bit AI use heavy which is not fair in every case, I guess we have an edge case here, that needs to be dealt with.

[Tushar-R-Tyagi]

Yes I have verified, this is indeed an edge case and an interesting one actually. I got distracted by AI orchestration so much that I forgot to consider the best case scenario. The case is basically, what if the candidate solves the problem completely by himself, which is very much possible in this problem. But right now, he is being penalised (across three metrics) for not using a tool, he did not need for this specific problem. The ideas that come to my mind to solve this problem are to:

• introduce a layer that only checks for the fixed bugs, whether AI was used or not.. the simplest and the best scenario.
• a second layer that uses the current logic when the agent is used to fix the problem and then score the candidate based on their approach.