Review eval suite has zero coverage — reintroduce review eval cases after post-review 422 fix lands

[fullsend-ai-retro]

What happened

PR #205 removed the only review eval case (001-clean-approve) to unblock the merge queue. The case failed because fullsend post-review returns 422 errors from inline comments citing line numbers outside diff hunks (issue #193). The removal was the right short-term fix, but the review eval suite now has zero cases. The run-functional.sh script exits 0 with a warning when no cases exist, meaning the review agent has no functional test coverage and regressions will go undetected.

What could go better

The removal was a necessary tactical fix, and the issue body explicitly noted the case should be reintroduced after the 422 fix lands. However, there's no tracking issue to ensure the case is actually reintroduced. The underlying fix is tracked by issue #193, but closing #193 won't automatically trigger case reintroduction. Medium confidence: this is a gap in the tracking workflow — the case could remain absent indefinitely if no one remembers to reintroduce it.

Proposed change

File a tracking issue in fullsend-ai/agents to reintroduce the 001-clean-approve review eval case (or an equivalent) once issue #193 is resolved. The issue should block on #193 and include the original case's annotations.yaml and input.yaml content for reference. Additionally, consider adding a CI lint check in eval/lint-cases.sh that warns (not errors) when an eval suite defined in eval/<agent>/eval.yaml has zero cases in its cases/ directory — this creates visibility without blocking merges.

Validation criteria

A tracking issue exists for reintroducing the review eval case. The review eval suite has at least one passing case within 30 days of issue #193 being resolved.

---

Generated by retro agent from #205

[fullsend-ai-triage]

🤖 Triage · ❌ Terminated · Started 3:34 PM UTC · Ended 3:37 PM UTC
Commit: 12bd957 · View workflow run →

[fullsend-ai-triage]

Status update

The primary concern — zero review eval coverage — is resolved. The eval/review/cases/ directory now contains 4 cases (001-risk-low-typo-fix, 002-risk-high-auth-change, 003-protected-path-downgrade, 004-tool-owned-codecov-filename). The validation criteria ("at least one passing case within 30 days of #193 resolution") are met.

The previously identified blockers have been resolved:

• fullsend#2569 (upstream root cause): closed 2026-07-21
• PR fix(#193): apply outcome labels even when review submission fails #195 (resilience fix): closed without merge (superseded by upstream fix)
• Issue post-review: 422 on inline comment outside diff hunk prevents label application #193: upstream dependency resolved; issue remains open but no longer blocks this work

Remaining work

The secondary proposal — adding a zero-case warning to eval/lint-cases.sh — has not been implemented. Currently the script validates case structure but silently passes when no cases exist. Adding a warning (not error) when $CASES_DIR has zero subdirectories would prevent silent regression to zero coverage.

# Proposed test: lint-cases.sh with empty cases/ directory
mkdir -p /tmp/test-eval/agent/cases
cp eval/review/eval.yaml /tmp/test-eval/agent/eval.yaml
# Should emit warning to stderr about zero cases, exit 0
./eval/lint-cases.sh agent 2>&1 | grep -q 'WARNING.*zero cases'

---

Labels: Removing blocked since all prerequisites are resolved; adding enhancement since the remaining work is a new lint check feature.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

Previous run

Triage Assessment

This issue is well-described with clear scope: reintroduce the 001-clean-approve review eval case (or equivalent) once the underlying 422 failure is resolved, and optionally add a CI lint check for zero-case eval suites.

However, progress is blocked on prerequisite work that has not yet landed:

Blocking dependencies

1. Issue post-review: 422 on inline comment outside diff hunk prevents label application #193 (this repo, OPEN, blocked) — the 422 on inline comments outside diff hunks that caused the eval case to fail in the first place. This issue has two fixes:

   • Fix 2 (resilience): PR fix(#193): apply outcome labels even when review submission fails #195 is open and would make post-review.sh apply outcome labels even when the review submission returns 422. If this merges, the eval case may become viable again since the required_labels judge would pass.
   • Fix 1 (root cause): blocked on the upstream fullsend repo.

2. post-review 422 regression: diff-hunk validation does not cover all edge cases fullsend#2569 (upstream, OPEN, ready-to-code) — the root cause fix for the 422 regression in fullsend post-review's diff-hunk validation. Until this lands, inline comments on lines outside diff hunks will continue to trigger 422 errors.

3. PR fix(#193): apply outcome labels even when review submission fails #195 (this repo, OPEN) — the most likely near-term unblock. Once it merges, the eval case could potentially be reintroduced even before the upstream root cause fix, since labels would be applied despite the 422.

Recommendation

Once PR #195 merges, reassess whether the eval case can be reintroduced with the resilience fix alone. If the required_labels judge passes when labels are applied despite a 422, the case can come back immediately. Otherwise, wait for the upstream fix in fullsend-ai/fullsend#2569.

The secondary proposal (CI lint check in eval/lint-cases.sh that warns when an eval suite has zero cases) is independent and could be split into a separate issue to make progress now.

Blocked by:

• post-review: 422 on inline comment outside diff hunk prevents label application #193
• fix(#193): apply outcome labels even when review submission fails #195
• post-review 422 regression: diff-hunk validation does not cover all edge cases fullsend#2569

---

Labels: This is blocked on prerequisite work (#193 and upstream #2569). The review-agent label applies since this is about review eval coverage.

[fullsend-ai-prioritize]

RICE Priority Score: 1.6

Score breakdown

Dimension	Score	Reasoning
Reach	1	This is a test coverage gap for the review agent. While it directly affects the development team, all users of the review agent are indirectly impacted because regressions could ship undetected. The review agent is one component (not platform-wide), so this affects a moderate number of users.
Impact	1	Without any functional eval cases, regressions in the review agent will go undetected until they surface in production. This is a meaningful gap in quality assurance (medium impact) but not directly blocking any user workflow today — the review agent still functions, it's just untested.
Confidence	0.8	High confidence. The issue is exceptionally well-described with clear scope, identified blocking dependencies (#193, PR #195, upstream #2569), and a thorough triage assessment. The only uncertainty is timing — when blockers will resolve — but the work itself is well-defined.
Effort	0.5	Simple once unblocked. The original eval case content already exists and just needs to be reintroduced. The optional CI lint check is a small script addition. The main complexity is the dependency chain, not the implementation work itself.

Formula: (1 x 1 x 0.8) / 0.5 = 1.6