Code agent should verify root cause hypothesis before implementing

[fullsend-ai-retro]

What happened

On PR #195, the code agent built a 254-line fix for #193's hypothesis (422 from out-of-hunk comments). The human discovered the 422s were actually from self-review. Entire PR cycle wasted.

What could go better

AGENTS.md says 'verify claims about root cause' but this isn't a concrete step. Issue #193 linked a workflow run — examining its error messages would have revealed the real cause. Confidence: MEDIUM.

Proposed change

Add to skills/code-implementation/SKILL.md: when an issue describes a root cause with linked evidence, state the hypothesis, check available evidence, and if it contradicts, comment on the issue instead of implementing.

Validation criteria

Next 3 code runs on bug issues with linked evidence should show hypothesis verification in the JSONL trace.

---

Generated by retro agent from #195

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 11:51 AM UTC · Completed 11:55 AM UTC
Commit: 7d0bfa9 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Assessment: Sufficient for implementation · Severity: Medium · Category: Feature

Problem

The code-implementation SKILL.md lacks a concrete procedure for verifying root cause hypotheses against linked evidence before implementing a fix. The current guidance in step 2 ("verify it against the code before relying on it") is too vague to trigger structured investigation. In the motivating incident (PR #195 / issue #193), the code agent built a 254-line fix based on an incorrect hypothesis when the linked workflow run's error messages would have revealed the actual cause.

Root Cause

No actionable verification step exists in the SKILL.md that says: state the hypothesis, check linked evidence, and stop if it contradicts. Without this, the agent treats the triage hypothesis as ground truth.

Recommended Fix

Add an explicit hypothesis verification sub-step to the code-implementation SKILL.md (within step 2 or between steps 7–8) that requires the agent to:

1. State the root cause hypothesis from the issue/triage
2. Check linked evidence (workflow logs, error messages, PR comments)
3. If evidence contradicts the hypothesis, comment on the issue instead of implementing

Related Issues

• Issue Code agent should verify that suggested implementation approaches are semantically compatible with the use case #229 (verify implementation approach compatibility) — adjacent but distinct concern
• Issue Triage agent should promote to ready-to-code when code investigation confirms an actionable root cause #290 (triage agent promotion) — addresses the triage agent, not the code agent

Proposed Test Criteria

For the next 3 code agent runs on bug issues with linked evidence:
- The JSONL trace should show a step where the agent explicitly states
  the root cause hypothesis
- The trace should show the agent examining linked evidence
  (workflow runs, error logs, PR comments)
- If evidence contradicts the hypothesis, the agent should stop and
  comment rather than implement

---

Labels: This is a feature request for the code agent's implementation skill. Applying 'code-agent' for area ownership and 'enhancement' for issue type. Removing 'ready-for-triage' as triage is complete.

[fullsend-ai-prioritize]

RICE Priority Score: 3.6

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	This affects all code agent runs on bug issues that include linked evidence (workflow runs, error logs, PR comments). While not every issue has linked evidence, bug fixes are a core use case for the code agent, so a meaningful subset of all runs across all users/customers would benefit from this guardrail.
Impact	1.5	The motivating incident (PR #195) demonstrates that failing to verify root cause wastes entire PR cycles — 254 lines of code built on a wrong hypothesis. Preventing this is a significant efficiency gain, though it improves quality rather than unblocking core functionality.
Confidence	0.8	High confidence — the issue is well-described with a concrete motivating example, a clear proposed change (add a sub-step to SKILL.md), and specific validation criteria. The triage summary confirms the scope is small and well-understood.
Effort	0.5	This is a small, well-scoped documentation change to skills/code-implementation/SKILL.md — adding an explicit hypothesis verification sub-step. No architectural changes, no multi-file refactoring, just a targeted addition to an existing skill file.

Formula: (1.5 x 1.5 x 0.8) / 0.5 = 3.6