-
-
Notifications
You must be signed in to change notification settings - Fork 8
eval: emit diagnostic failure_reason + JSONL feedback export from all evaluators #1451
Copy link
Copy link
Open
Labels
backendRelated to inference backendsRelated to inference backendsbugSomething isn't workingSomething isn't workingdocumentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or requestevaluationLLM evaluation harness and quality testingLLM evaluation harness and quality testingperformancePerformance improvementsPerformance improvementsplatform/linuxLinux specificLinux specificsafetyRelated to command safety validationRelated to command safety validation
Description
Activity
Metadata
Metadata
Assignees
Labels
backendRelated to inference backendsRelated to inference backendsbugSomething isn't workingSomething isn't workingdocumentationImprovements or additions to documentationImprovements or additions to documentationenhancementNew feature or requestNew feature or requestevaluationLLM evaluation harness and quality testingLLM evaluation harness and quality testingperformancePerformance improvementsPerformance improvementsplatform/linuxLinux specificLinux specificsafetyRelated to command safety validationRelated to command safety validation
Context
Learning L2 from
docs/research/dspy-prompt-optimization.md(PR #1448): DSPy's GEPA optimizer needs a metric that returns a score and textual feedback — the feedback is what a reflection model reasons over when proposing a better prompt. It is also simply better failure reporting for humans.Today the CI eval target (
cargo test --test evaluation, wired tosrc/evaluation/) scores per-rule boolean pass/fail.EvaluationResult.failure_reason(src/evaluation/models.rs) exists and the rule evaluators populate it, but the strings are prose for a report, have no stable shape, and nothing exports them per case.Problem
src/evaluation/evaluators/correctness.rs→evaluate_exact_match/evaluate_command_equivalence/evaluate_pattern_matchreturn(bool, Option<String>)with free-text reasons;posix.rsandsafety.rsfollow the same pattern.(input, expected, actual, score, feedback), so neither an optimizer nor a triage script can consume eval failures. Aggregate pass rate is the only artifact.Proposal
{test_id, backend, category, input, expected, actual, passed, score, feedback}. Follow thesrc/evaluation/sft_export.rspattern (pure transform fromEvaluationResult+TestCase, no IO inside the module; the harness CLI writes the file via a--feedback-jsonl <path>flag or alongside--format json).mainmust not move.Acceptance criteria
EvaluationResultcarries afailure_reasonnaming the failing rule and an expected/actual pair.cargo test --test evaluation -- --backend static_matcher --feedback-jsonl out.jsonlwrites one valid JSONL line per evaluated case.src/evaluation/evaluators/asserting the feedback shape for each rule, plus a contract test that JSONL lines round-trip through serde.static_matcherunchanged.Related
src/evaluation/evaluators/, not the unlinked ladder intests/evaluation/src/evaluator.rs)tools/dspy-harness/) proposed in the doc.