Fix agent fails silently when it cannot produce output within iteration limit

[fullsend-ai-retro]

What happened

Three /fix commands were issued on PR fullsend-ai/fullsend#473 (May 2 by waynesun09 with detailed instructions, May 6 by ascerra twice). All three fix runs in fullsend-ai/.fullsend failed with 'validation failed after 2 iteration(s)' — the agent exited with code 1 without writing fix-result.json. No commits were pushed, and no feedback was posted to the PR explaining the failure. The humans who invoked /fix received no indication that the fix agent had tried and failed.

What could go better

Two problems compound here: (1) the fix agent couldn't complete the work in 2 iterations, and (2) no failure notification was posted to the PR. The first problem may be related to issues #615/#408 (HUMAN_INSTRUCTION shell expansion) or simply task complexity exceeding the iteration budget. The second problem means humans waited days without knowing the fix failed. Confidence on root cause: medium — I can see the failures but cannot access the agent's reasoning trace to confirm why it exited with code 1.

Proposed change

The fix agent's post-script (or the harness itself) should post a comment on the PR when a fix run fails validation, including the failure reason and iteration count. Something like: '⚠️ Fix agent failed after 2 iterations without producing results. View run. You may need to apply changes manually or re-run with /fix --force.' This is likely a change to internal/scaffold/fullsend-repo/scripts/post-fix.sh or the harness validation failure path.

Validation criteria

The next fix agent failure on any PR should result in a comment posted to the PR within 5 minutes of the failure, with a link to the Actions run. Verify over the next 3 fix agent failures across the org.

---

Generated by retro agent from #473

[github-actions]

fullsend triage is working on this — view logs

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Bug

Problem

When the fix agent's validation loop fails after exhausting its iteration budget, no comment is posted to the PR. This is because the harness (internal/cli/run.go, lines 258–261) intentionally skips the post-script when validation fails — a security constraint from ADR 0022's zero-trust model (don't process unvalidated agent output). However, there is no alternative failure notification path, so humans who invoked /fix receive no feedback.

Root Cause

The harness has two exit paths after agent completion:

• Validation passes → post-script runs → PR comment posted ✓
• Validation fails → post-script skipped → no notification ✗

The validation failure error only surfaces in GitHub Actions logs, which most users won't check proactively.

Recommended Fix

Add an on_failure_script (or equivalent) to the harness YAML schema that runs a lightweight notification script when validation fails. This script should NOT process agent output (respecting ADR 0022) but should post a minimal failure notice to the PR with:

• The failure reason and iteration count
• A link to the Actions run
• Guidance to apply changes manually or retry

Alternatively, add failure notification logic directly in the run.go defer block — when h.ValidationLoop != nil && !validationPassed, post a comment via gh CLI before returning the error.

Related Issues

• post-code.sh: comment on issue when agent determines no changes needed agents#505 — same pattern for the code agent (post-code.sh)
• Failed review agent runs should update the status comment on the PR #837 — same pattern for the review agent

These are not duplicates (different agents/scripts) but could benefit from a unified approach.

Proposed Test Case

// Test: validation failure triggers PR notification
//
// Setup: Configure a harness with validation_loop (max_iterations=1)
//        and a validation script that always exits 1.
//
// Assert:
//   1. runAgent() returns a "validation failed" error
//   2. A failure notification was dispatched with correct PR number,
//      repo name, and iteration count
//   3. The full post-script was NOT executed (ADR 0022 respected)
//
// Edge cases:
//   - Notification is NOT sent when validation passes
//   - Notification includes the Actions run URL when available
//   - Notification works even if no fix-result.json was produced

[github-actions]

fullsend code is working on this — view logs

[ifireball]

/fs-prioritize

[fullsend-ai-prioritize]

RICE Priority Score: 2.4

Score breakdown

Dimension	Score	Reasoning
Reach	2	This affects all users who invoke /fix on PRs across the org. The triage comment confirms the same silent-failure pattern exists for the code agent (#757) and review agent (#837), making this a platform-wide UX gap. Scored 2 (most active users across orgs).
Impact	1.5	Users waited days without knowing the fix agent had failed, with no workaround other than manually checking Actions logs. This is a significant pain point that erodes trust in the agent system — users can't tell if the agent is still working or has silently failed. Scored 1.5 (significant pain point).
Confidence	0.8	The issue is well-described with clear reproduction (3 failures on PR #473), the triage agent identified the exact root cause in run.go (validation failure skips post-script), and proposed a concrete fix path. The only gap is whether the on_failure_script approach or the defer-block approach is preferable. Scored 0.8 (high).
Effort	1	The triage comment pinpoints the exact code location (internal/cli/run.go lines 258-261) and the fix is well-scoped: add failure notification logic in the defer block or introduce an on_failure_script in the harness schema. Touches a few files but the change is straightforward. Scored 1 (medium).

Formula: (2 x 1.5 x 0.8) / 1 = 2.4