What happened
The retro agent generated issue #108 from a PR #90 retro, proposing to short-circuit review agent sub-agent dispatch on rebase-only pushes. The proposal was well-reasoned and included validation criteria, but did not note that the review skill has zero eval coverage — the repo only has triage evals in eval/triage/. The code agent implemented the proposal in PR #111, which was then rejected by the human reviewer: 'This sounds dangerous to me. I don't want to introduce it without pre-existing evals.' The entire triage-to-code cycle (~17 minutes of agent time) was wasted because the proposal lacked a prerequisite that the human considered mandatory.
What could go better
The retro agent has access to the repo's file structure and could check whether eval coverage exists for the skill being modified. When proposing changes to skills/pr-review/SKILL.md, a check for eval/review/ or similar would have revealed the absence of review evals. The proposal could then have either (a) included eval creation as part of the proposed change, or (b) noted the eval gap as a prerequisite that must be resolved first. This would have prevented the code agent from implementing a change that couldn't be safely merged. Confidence: high — the human's rejection was unambiguous and the eval gap is objectively verifiable (no eval/review/ directory exists).
Proposed change
In skills/retro-analysis/SKILL.md, add guidance to the 'Writing good proposals' section: when a proposal targets a skill file (matching skills/*/SKILL.md), the retro agent should check whether eval infrastructure exists for that skill by looking for eval/{skill-name}/eval.yaml or similar. If no evals exist, the proposal's what_could_go_better section should note the eval gap, and the validation_criteria should include eval creation as a prerequisite — or the proposal itself should recommend creating evals before implementing the behavioral change. Suggested addition after the existing bullet points:
- Eval coverage check: If the proposed change modifies an agent skill (
skills/*/SKILL.md), check whether eval cases exist for that skill (eval/*/eval.yaml). If no evals exist, either include eval creation as part of the proposal or note eval coverage as a prerequisite in the validation criteria. Behavioral changes to agent skills without eval coverage cannot be safely validated.
Validation criteria
On the next 3 retro runs that produce proposals targeting agent skill files, each proposal should either (a) confirm eval coverage exists for the target skill, (b) include eval creation as part of the proposed change, or (c) explicitly note the eval gap as a prerequisite. No proposal for a skill behavior change should be filed without addressing eval coverage.
Generated by retro agent from #111
What happened
The retro agent generated issue #108 from a PR #90 retro, proposing to short-circuit review agent sub-agent dispatch on rebase-only pushes. The proposal was well-reasoned and included validation criteria, but did not note that the review skill has zero eval coverage — the repo only has triage evals in
eval/triage/. The code agent implemented the proposal in PR #111, which was then rejected by the human reviewer: 'This sounds dangerous to me. I don't want to introduce it without pre-existing evals.' The entire triage-to-code cycle (~17 minutes of agent time) was wasted because the proposal lacked a prerequisite that the human considered mandatory.What could go better
The retro agent has access to the repo's file structure and could check whether eval coverage exists for the skill being modified. When proposing changes to
skills/pr-review/SKILL.md, a check foreval/review/or similar would have revealed the absence of review evals. The proposal could then have either (a) included eval creation as part of the proposed change, or (b) noted the eval gap as a prerequisite that must be resolved first. This would have prevented the code agent from implementing a change that couldn't be safely merged. Confidence: high — the human's rejection was unambiguous and the eval gap is objectively verifiable (noeval/review/directory exists).Proposed change
In
skills/retro-analysis/SKILL.md, add guidance to the 'Writing good proposals' section: when a proposal targets a skill file (matchingskills/*/SKILL.md), the retro agent should check whether eval infrastructure exists for that skill by looking foreval/{skill-name}/eval.yamlor similar. If no evals exist, the proposal'swhat_could_go_bettersection should note the eval gap, and thevalidation_criteriashould include eval creation as a prerequisite — or the proposal itself should recommend creating evals before implementing the behavioral change. Suggested addition after the existing bullet points:skills/*/SKILL.md), check whether eval cases exist for that skill (eval/*/eval.yaml). If no evals exist, either include eval creation as part of the proposal or note eval coverage as a prerequisite in the validation criteria. Behavioral changes to agent skills without eval coverage cannot be safely validated.Validation criteria
On the next 3 retro runs that produce proposals targeting agent skill files, each proposal should either (a) confirm eval coverage exists for the target skill, (b) include eval creation as part of the proposed change, or (c) explicitly note the eval gap as a prerequisite. No proposal for a skill behavior change should be filed without addressing eval coverage.
Generated by retro agent from #111