Triage agent should treat absent eval coverage as a prerequisite for issues that modify agent skill behavior

[fullsend-ai-retro]

What happened

The triage agent (run 29104087079) triaged issue #108 — a proposal to modify the review skill's behavior on rebase-only pushes — as sufficient with ready-to-code. This triggered the code agent, which produced PR #111. The human reviewer rejected the PR because the review skill has no eval coverage to validate behavioral changes. The triage agent did not check for eval coverage existence and treated the issue as ready for implementation.

What could go better

The triage agent already checks for blocking dependencies in step 2b (duplicate and blocking relationship search). It could extend this to check whether the affected skill has eval infrastructure. For the agents repo specifically, modifying skills/pr-review/SKILL.md without eval/review/eval.yaml existing is a prerequisite gap — the change cannot be safely deployed without validation. If the triage agent had emitted action: 'prerequisites' noting the need for review eval coverage, the code agent would not have been dispatched. Confidence: medium — this is specific to the agents repo's eval maturity. Once review evals exist, this gate would be satisfied and issues could proceed normally. The check might be too repo-specific for the general triage agent prompt, but it could be added to repo-level context (AGENTS.md or CLAUDE.md).

Proposed change

Add a section to AGENTS.md (the repo's agent-facing instructions) that documents the eval requirement for skill changes:

## 8. Eval coverage requirement for skill changes

Changes to agent skill behavior (`skills/*/SKILL.md`) require eval coverage for the affected skill. Before labeling a skill-modification issue as ready-to-code, verify that eval infrastructure exists (`eval/{skill-name}/eval.yaml`). If no evals exist for the target skill, the issue is blocked on eval creation.

Currently, only the triage skill has eval coverage (`eval/triage/`). Issues proposing behavioral changes to review, code, fix, or retro skills should be treated as having a prerequisite on eval infrastructure creation.

This approach leverages the triage agent's existing repo-context reading (step 2a: it reads CLAUDE.md and AGENTS.md) rather than modifying the general triage agent prompt. The triage agent would see this constraint during its context-gathering phase and factor it into its prerequisite analysis.

Validation criteria

On the next triage of an issue proposing behavioral changes to a skill that lacks eval coverage (e.g., review, code, fix), the triage agent should emit action: 'prerequisites' or note the eval gap in its reasoning rather than labeling the issue ready-to-code. Verify by checking the next 2 triage runs on skill-modification issues.

---

Generated by retro agent from #111

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 7:18 PM UTC · Completed 7:21 PM UTC
Commit: b8a817e · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: medium · Category: feature

Problem

The triage agent promoted issue #108 to ready-to-code without verifying that the target skill (review) has eval coverage. The resulting PR #111 was rejected by a human reviewer because behavioral changes to agent skills cannot be safely validated without evals. Currently only eval/triage/ exists — review, code, fix, and retro skills have no eval infrastructure.

Root Cause

AGENTS.md — the repo-level context the triage agent reads during step 2a — has no guidance about eval coverage requirements for skill changes. Without this constraint, the triage agent has no way to know that skill-modification issues should be gated on eval existence.

Recommended Fix

Add section 8 to AGENTS.md with the eval coverage requirement text proposed in the issue body. This is a narrow documentation change that leverages the triage agent's existing context-reading behavior.

Proposed Test Case

Add an eval case to eval/triage/cases/ with a synthetic issue
proposing a behavioral change to skills/pr-review/SKILL.md.

Expected: triage agent emits action: prerequisites (or notes the
eval gap in reasoning) rather than ready-to-code.

Edge case: issue targeting a skill WITH eval coverage (e.g., triage)
should still be eligible for ready-to-code.

Related

• Issue Retro agent should assess eval coverage prerequisites when proposing behavioral changes to agent skills #115 is a companion retro issue targeting the retro agent's skill definition — complementary, not a duplicate.
• PR feat(#108): short-circuit review on rebase-only pushes #111 (closed, not merged) was the failed implementation that motivated this issue.

[fullsend-ai-prioritize]

RICE Priority Score: 1.6

Score breakdown

Dimension	Score	Reasoning
Reach	1	This affects the internal agent workflow (triage, code, review agents) and the team maintaining the agents repo. The improvement prevents wasted agent runs for anyone modifying skills without eval coverage — currently 4 out of 5 skills (review, code, fix, retro) lack evals, so skill-modification issues for these are affected. Moderate reach within this repo's contributors and automated agents.
Impact	1	Medium — the issue documents a concrete case (PR #111) where the triage agent incorrectly promoted an issue to ready-to-code, triggering a wasted code agent run that was subsequently rejected by a human reviewer. Adding this guardrail provides a noticeable workflow improvement by preventing wasted compute and review cycles, but workarounds exist (human reviewers catch the gap).
Confidence	0.8	High — the issue is exceptionally well-described with a specific proposed change (add section 8 to AGENTS.md), clear validation criteria, a concrete motivating example (PR #111), and a triage summary confirming the analysis. The scope is narrow and well-bounded.
Effort	0.5	Simple — the core change is adding a documentation section to AGENTS.md. The proposed text is already drafted in the issue body. An optional eval test case is suggested but the primary deliverable is a small, well-scoped documentation addition.

Formula: (1 x 1 x 0.8) / 0.5 = 1.6