What happened
On PR #6982, the review agent ran 3 times with 2 iterations each (6 total iterations). In 5 of 6 iterations, sub-agents (correctness, challenger, docs-currency, style-conventions, intent-coherence, risk-assessment, contracts) completed with detailed findings — including HIGH-severity bugs — but the outer orchestrator never wrote agent-result.json. The harness reported FAIL: output/agent-result.json not found.
Failed runs showed 0 turns and $0.00 cost despite 400+ sub-agent tool calls, indicating the orchestrator session never completed its synthesis phase. Runs 33785884243 and 33797773532 failed both iterations. Run 33802280060 failed iteration 1 but succeeded on iteration 2.
The failed runs reviewed code with more correctness issues (the human was still fixing bugs). The successful iteration reviewed cleaner code with fewer sub-agent findings. Agents repo: fullsend-ai/agents@10560252b0ad.
What could go better
The orchestrator's synthesis step — aggregating sub-agent results and writing the output file — is a single point of failure. When sub-agents produce larger volumes of findings, the orchestrator appears more likely to fail. This correlation is suggestive but not certain (n=3 runs).
The 0-turn / $0-cost signature suggests the orchestrator session may be crashing, timing out, or hitting a context limit before it processes any sub-agent results. Even the successful run failed on its first iteration, indicating persistent fragility.
The impact was significant: review sub-agents found HIGH-severity bugs in the code agent's output during the first failed run, but this feedback never reached the human. The human spent ~3 hours manually finding and fixing these same bugs without review feedback. If the first review had succeeded, it would have surfaced actionable findings within ~45 minutes of PR creation.
Proposed change
Investigate the review orchestrator's synthesis phase (defined in agents/review.md and the review harness at harness/review.yaml in fullsend-ai/agents) for failure modes that prevent agent-result.json from being written. Potential fixes:
- Graceful degradation in the agent definition: If the orchestrator cannot fully synthesize sub-agent results, write a partial
agent-result.json with raw sub-agent findings rather than producing no output.
- Harness-level fallback: In
scripts/validate-output-schema.sh or a post-iteration step, detect when sub-agents completed (tool calls > 0) but orchestrator produced no output (turns = 0), and extract sub-agent findings as a fallback result.
- Diagnostic logging: Add structured logging when the orchestrator session ends without writing output, capturing the termination reason (timeout, context overflow, error) to enable targeted fixes.
As an immediate diagnostic step, download and analyze the fullsend-review artifacts from the failed runs (3.1–3.3 MB each) to determine why the orchestrator exited without producing output despite sub-agents completing.
Validation criteria
Monitor the next 30 review agent runs on fullsend-ai/fullsend. The rate of agent-result.json not found validation failures should drop from the observed ~83% per-iteration (5/6) to below 15%. The review agent should produce actionable output on at least 90% of runs where sub-agents complete successfully.
Generated by retro agent from fullsend-ai/fullsend#6982
What happened
On PR #6982, the review agent ran 3 times with 2 iterations each (6 total iterations). In 5 of 6 iterations, sub-agents (correctness, challenger, docs-currency, style-conventions, intent-coherence, risk-assessment, contracts) completed with detailed findings — including HIGH-severity bugs — but the outer orchestrator never wrote
agent-result.json. The harness reportedFAIL: output/agent-result.json not found.Failed runs showed 0 turns and $0.00 cost despite 400+ sub-agent tool calls, indicating the orchestrator session never completed its synthesis phase. Runs 33785884243 and 33797773532 failed both iterations. Run 33802280060 failed iteration 1 but succeeded on iteration 2.
The failed runs reviewed code with more correctness issues (the human was still fixing bugs). The successful iteration reviewed cleaner code with fewer sub-agent findings. Agents repo:
fullsend-ai/agents@10560252b0ad.What could go better
The orchestrator's synthesis step — aggregating sub-agent results and writing the output file — is a single point of failure. When sub-agents produce larger volumes of findings, the orchestrator appears more likely to fail. This correlation is suggestive but not certain (n=3 runs).
The 0-turn / $0-cost signature suggests the orchestrator session may be crashing, timing out, or hitting a context limit before it processes any sub-agent results. Even the successful run failed on its first iteration, indicating persistent fragility.
The impact was significant: review sub-agents found HIGH-severity bugs in the code agent's output during the first failed run, but this feedback never reached the human. The human spent ~3 hours manually finding and fixing these same bugs without review feedback. If the first review had succeeded, it would have surfaced actionable findings within ~45 minutes of PR creation.
Proposed change
Investigate the review orchestrator's synthesis phase (defined in
agents/review.mdand the review harness atharness/review.yamlinfullsend-ai/agents) for failure modes that preventagent-result.jsonfrom being written. Potential fixes:agent-result.jsonwith raw sub-agent findings rather than producing no output.scripts/validate-output-schema.shor a post-iteration step, detect when sub-agents completed (tool calls > 0) but orchestrator produced no output (turns = 0), and extract sub-agent findings as a fallback result.As an immediate diagnostic step, download and analyze the
fullsend-reviewartifacts from the failed runs (3.1–3.3 MB each) to determine why the orchestrator exited without producing output despite sub-agents completing.Validation criteria
Monitor the next 30 review agent runs on fullsend-ai/fullsend. The rate of
agent-result.json not foundvalidation failures should drop from the observed ~83% per-iteration (5/6) to below 15%. The review agent should produce actionable output on at least 90% of runs where sub-agents complete successfully.Generated by retro agent from fullsend-ai/fullsend#6982