Does the llm_rubric judge penalize the agent when the judge call itself fails?

[eng1725]

If the grading model errors out (e.g. a 429 / 5xx) or returns unparseable output while scoring with llm_rubric, does that get counted as a failure against the agent being evaluated, or is it handled some other way?

[itseffi]

No, a judge-side failure does not count against the agent. If the llm_rubric grading call raises (transient API error) or returns output that can't be parsed into a verdict, the verifier abstains: it emits a criterion with weight=0 and required=false, so it can neither drag down the weight-normalized reward nor fail an otherwise-passing run. The underlying error is still recorded on the criterion (error field) for inspection. The parser is also tolerant of prose- or fence-wrapped JSON, so only a genuine failure leads to an abstain rather than a verdict.