Add evidence-for content check to retro validation loop

[fullsend-ai-retro]

What happened

PR #85 added a post-script gate in scripts/post-retro.sh to reject retro proposals with evidence-for title patterns. The gate works correctly but runs after the sandbox is destroyed — the agent cannot fix its output at that point. The retro-analysis skill instructs the agent not to file these proposals, but the agent ignores the instruction frequently enough that the comment references fullsend-ai/fullsend#3881 as the tracking issue. When a proposal is filtered, the agent's token budget for generating that proposal is wasted.

What could go better

The validation loop (harness/retro.yaml → scripts/validate-output-schema.sh) runs inside the sandbox with up to 2 retry iterations. If evidence-for proposals were caught during validation, the agent would receive a clear error message (e.g., "FAIL: proposal[0] title matches evidence-for pattern — fold this into your summary instead") and could rewrite the proposal as a summary note or a different, non-duplicate proposal.

This would recover value from the agent's analysis work rather than silently discarding it. The post-script gate would remain as defense-in-depth for cases where the agent fails to fix its output within the allowed iterations.

Confidence: High — the validation loop mechanism is well-established across all agents, and the evidence-for pattern matching logic already exists in post-retro.sh. The main risk is that adding content validation to the loop could slow it down, but the check is a simple regex match on 1-3 proposal titles, which is negligible.

Note: GitHub API was unavailable during this retro, so I could not verify whether an open issue already covers this improvement.

Proposed change

Create a retro-specific validation script (e.g., scripts/validate-retro-output.sh) that:

1. Calls the existing scripts/validate-output-schema.sh for schema validation
2. If schema validation passes, checks each proposal title against the evidence-for patterns (reuse the same regex from post-retro.sh lines 87-89)
3. Fails with a clear message if any proposal matches, telling the agent to fold the evidence into the summary field instead

Update harness/retro.yaml to use the new validation script:

validation_loop:
  script: scripts/validate-retro-output.sh
  schema: schemas/retro-result.schema.json
  max_iterations: 2

Keep the existing post-script gate in post-retro.sh as defense-in-depth — it catches anything the validation loop misses.

Validation criteria

1. When a retro agent generates a proposal with an evidence-for title pattern, the validation loop catches it and the agent rewrites or removes the proposal before the sandbox exits.
2. The post-script gate in post-retro.sh still functions as a fallback — existing tests continue to pass.
3. Over the next 10 retro runs, the number of proposals silently filtered by the post-script gate decreases (ideally to zero, since the validation loop catches them first).

---

Generated by retro agent from #85

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 3:09 PM UTC · Completed 3:13 PM UTC
Commit: b8a817e · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature

Problem

The evidence-for proposal gate in post-retro.sh runs after the sandbox is destroyed. When the retro agent generates an evidence-for proposal, it is silently filtered — the agent never gets a chance to self-correct and rewrite the proposal. The token budget spent generating that proposal is wasted, and any useful evidence in the analysis is lost rather than being folded into the summary.

Root Cause

The retro harness validation loop (harness/retro.yaml) currently uses the generic scripts/validate-output-schema.sh, which only checks JSON Schema compliance. No content-level checks run inside the sandbox, so evidence-for proposals pass validation and are only caught by the post-script gate after sandbox exit.

Recommended Fix

Create scripts/validate-retro-output.sh that wraps the existing schema validator and adds evidence-for pattern matching (reusing the regex from post-retro.sh lines 87–89). Update harness/retro.yaml to use the new script. Keep the post-script gate as defense-in-depth.

Companion issue #102 proposes reinforcing the prohibition in the agent prompt — these are complementary approaches.

Proposed Test Case

# Test cases for validate-retro-output.sh:
1. PASS: proposals with non-matching titles (e.g., "Improve evidence gathering")
2. FAIL: proposal title "Evidence for #123 — memory leak"
3. FAIL: proposal title "Evidence: repeated timeout in CI"
4. FAIL: proposal title "Additional evidence of flaky test"
5. PASS: schema validation errors are still surfaced correctly
   (script delegates to validate-output-schema.sh first)

---

Labels: This is a feature enhancement to the retro agent's validation loop. Applying 'enhancement' for the improvement type and 'ready-to-code' since the issue is fully specified with clear implementation steps, existing patterns to follow, and no open questions.