Add CI validation that eval case annotations reference valid judges

[fullsend-ai-retro]

What happened

PR #197 added a new eval case 002-line-accuracy whose annotations.yaml referenced a line_accuracy judge. This judge does not exist in eval/review/eval.yaml, which defines only 5 judges (review_quality, required_labels, forbidden_labels, max_turns, max_cost). The expected_findings block in the annotations also had no consumer — no judge reads or validates it. The human Review Squad caught this as a HIGH finding; the automated review agent did not detect it across 4 review runs. The PR was ultimately closed without merging, but the structural issue would have made the eval case non-functional even if merged.

What could go better

Eval cases are configuration that references other configuration (judges defined in eval.yaml). When a case references a nonexistent judge, it silently measures nothing — there is no runtime error, just a gap in coverage that looks like it's covered. This is analogous to a test file that imports a nonexistent test fixture: it should fail at lint/CI time, not pass silently.

Currently, eval case YAML files are not structurally validated against the eval framework definition. The Makefile has lint targets for other config (harness configs, schemas) but not for eval cases. Confidence is high that this gap exists — I verified eval/review/eval.yaml contains no line_accuracy judge, and no CI job validates judge name references in annotation files.

Proposed change

Add a CI validation step (e.g., a make lint-eval target or a script in scripts/) that, for each eval suite:

1. Parses eval.yaml to extract the set of defined judge names from the judges list.
2. Scans each case's annotations.yaml for judge-name references (keys under thresholds, fields referencing judges, or any identifier that should match a judge name).
3. Fails with a clear error if any annotation references a judge not defined in eval.yaml.

The script should be added to the CI workflow (alongside existing lint targets) so it runs on every PR that modifies files under eval/. The validation should also check that thresholds keys in eval.yaml reference defined judges (bidirectional consistency).

File locations: new script at scripts/lint-eval-cases.sh (or Python equivalent), with a lint-eval target in Makefile, wired into the CI workflow.

Validation criteria

After implementation: (1) Creating an eval case that references a nonexistent judge name should cause CI to fail with a descriptive error message. (2) The existing empty eval cases directory (just .gitkeep) should pass validation. (3) If/when eval cases from agents#209 or agents#245 are reintroduced, this check should validate their judge references before merge.

---

Generated by retro agent from #197

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 2:30 PM UTC · Completed 2:34 PM UTC
Commit: 7ada4e0 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature — new CI validation step
Severity: Medium

Problem

Eval case annotations.yaml files can reference judge names not defined in eval.yaml, causing silent coverage gaps. The existing eval/lint-cases.sh validates structural fields (max_turns, max_cost_usd) but does not cross-reference judge names against the judges list.

Current State

• eval/lint-cases.sh exists and checks for required annotation fields, but has no judge-name cross-validation
• eval/review/eval.yaml defines 5 judges: review_quality, required_labels, forbidden_labels, max_turns, max_cost
• eval/review/cases/ is currently empty (.gitkeep only) — this is preventive validation
• The Makefile has no lint-eval target currently

Recommended Approach

1. Extend eval/lint-cases.sh to extract judge names from eval.yaml and validate that all judge references in annotations.yaml files match defined judges
2. Add bidirectional check: thresholds keys in eval.yaml should also reference defined judges
3. Add a lint-eval target to the Makefile
4. Wire into CI workflow for PRs modifying files under eval/

Companion Issue

Issue #375 proposes a complementary approach — adding "reference validity" guidance to the correctness sub-agent. These are independent and complementary: #375 is a soft prompt-level check; this issue (#374) provides a mechanical CI backstop.

Proposed Test Cases

Test 1: annotations.yaml references undefined judge → script fails with descriptive error
Test 2: all judge references are consistent → script passes
Test 3: empty cases directory (.gitkeep only) → script passes
Test 4: thresholds key references undefined judge → script fails (bidirectional check)

---

Labels: This is a CI/tooling feature request for eval validation. Applying 'enhancement' for the feature category and removing 'ready-for-triage' as triage is complete.

[fullsend-ai-prioritize]

RICE Priority Score: 0.4

Score breakdown

Dimension	Score	Reasoning
Reach	0.5	This is internal CI/tooling for eval case validation, affecting developers who create eval cases. The eval cases directory is currently empty (.gitkeep only), so this is preventive. Only the internal team working on evals is affected — a small group within one org.
Impact	1	When this gap bites, it causes silent failures — eval cases that appear to measure coverage but actually measure nothing. However, human review caught this in PR #197, providing a workaround. The fix provides a meaningful improvement by making misconfiguration fail loudly at CI time rather than passing silently.
Confidence	0.8	High confidence. The issue is exceptionally well-described with clear validation criteria, the triage confirmed the current state (existing lint script, empty cases dir, specific judges list), and the scope is well-bounded. The only minor uncertainty is exactly how judge names are referenced in annotations files.
Effort	1	Medium effort. The existing eval/lint-cases.sh script provides a foundation to extend. The work involves parsing YAML to extract judge names, cross-referencing them, adding a Makefile target, and wiring into CI — touching a few files with clear requirements and no architectural changes needed.

Formula: (0.5 x 1 x 0.8) / 1 = 0.4