Skip to content

eval: emit diagnostic failure_reason + JSONL feedback export from all evaluators #1451

Description

@wildcard

Context

Learning L2 from docs/research/dspy-prompt-optimization.md (PR #1448): DSPy's GEPA optimizer needs a metric that returns a score and textual feedback — the feedback is what a reflection model reasons over when proposing a better prompt. It is also simply better failure reporting for humans.

Today the CI eval target (cargo test --test evaluation, wired to src/evaluation/) scores per-rule boolean pass/fail. EvaluationResult.failure_reason (src/evaluation/models.rs) exists and the rule evaluators populate it, but the strings are prose for a report, have no stable shape, and nothing exports them per case.

Problem

  • src/evaluation/evaluators/correctness.rsevaluate_exact_match / evaluate_command_equivalence / evaluate_pattern_match return (bool, Option<String>) with free-text reasons; posix.rs and safety.rs follow the same pattern.
  • There is no machine-readable per-case record of (input, expected, actual, score, feedback), so neither an optimizer nor a triage script can consume eval failures. Aggregate pass rate is the only artifact.

Proposal

  1. Diagnostic feedback with a stable shape from every evaluator: which rule failed, expected vs actual, the flag/operand diff, and for safety cases the pattern id that fired (or should have).
  2. JSONL export per run — one line per case: {test_id, backend, category, input, expected, actual, passed, score, feedback}. Follow the src/evaluation/sft_export.rs pattern (pure transform from EvaluationResult + TestCase, no IO inside the module; the harness CLI writes the file via a --feedback-jsonl <path> flag or alongside --format json).
  3. No change to pass/fail semantics — this is additive; pass rates on main must not move.

Acceptance criteria

  • Every failing EvaluationResult carries a failure_reason naming the failing rule and an expected/actual pair.
  • cargo test --test evaluation -- --backend static_matcher --feedback-jsonl out.jsonl writes one valid JSONL line per evaluated case.
  • Regression guard: unit tests in src/evaluation/evaluators/ asserting the feedback shape for each rule, plus a contract test that JSONL lines round-trip through serde.
  • Pass rate for static_matcher unchanged.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendRelated to inference backendsbugSomething isn't workingdocumentationImprovements or additions to documentationenhancementNew feature or requestevaluationLLM evaluation harness and quality testingperformancePerformance improvementsplatform/linuxLinux specificsafetyRelated to command safety validation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions