What happened
On #6962, the triage agent (run 33764725541, cost $0.84) correctly identified the 0.40.0 image repin as the suspect but misattributed the failure mechanism (model authorization vs OPA binary allowlist). Its output recommended investigation: confirming by running with pre-#1160 images and checking Vertex gateway logs. Despite this self-acknowledged uncertainty, the triage applied the ready-to-code label, which auto-dispatched a code agent that spent $10.15 over 20 minutes building a fix for the wrong problem. The code agent also modified .github/workflows/functional-tests.yml, which the coder app cannot push (no workflows permission). The human (waynesun09) had to manually correct the diagnosis 2 hours later and file #6971 with the actual root cause, leading to a successful $4.15 code run.
What could go better
The triage output contained contradictory signals: it recommended investigation/confirmation steps while simultaneously applying ready-to-code. A post-triage guard could detect this inconsistency and suppress the label, preventing the expensive code dispatch on an uncertain diagnosis. This complements agents#1055 (don't apply ready-to-code to investigation issues) but addresses a different pattern: a bug report whose triage recommends further investigation in its own output rather than an issue that is inherently investigative. Defense-in-depth is appropriate given the $10+ cost of a wrong dispatch. Confidence: high that this guard would have prevented the waste — the code agent inherited the wrong diagnosis with no independent validation. Uncertainty: it is possible the triage agent should simply not produce contradictory outputs, making a runtime guard unnecessary, but the agent's reasoning about sandbox error messages is inherently uncertain and a post-triage check is a more reliable mitigation.
Proposed change
In the post-triage script (scripts/post-triage.sh) in fullsend-ai/agents, add a guard that inspects the triage agent's structured output for investigation-recommendation signals before applying the ready-to-code label. If the triage output contains recommended actions that involve confirming a hypothesis, running experiments, or checking logs (pre-implementation investigation), the script should apply a needs-investigation label instead of ready-to-code and post a comment explaining that human confirmation is needed before code dispatch. The triage output schema (schemas/triage-result.schema.json) may need a requires_investigation boolean or diagnosis_confidence enum field to make this detection reliable rather than relying on text heuristics. This would serve as a defense-in-depth layer complementing the agent-level fix tracked in agents#1055.
Validation criteria
Over the next 30 days, measure: (1) zero code agent runs triggered by triage outputs that simultaneously recommend investigation steps, and (2) no increase in false-positive investigation labels on issues with clear, actionable diagnoses. The next 5 triage runs whose output recommends confirmation or investigation steps should result in a needs-investigation label rather than ready-to-code.
Generated by retro agent from fullsend-ai/fullsend#6975
What happened
On #6962, the triage agent (run 33764725541, cost $0.84) correctly identified the 0.40.0 image repin as the suspect but misattributed the failure mechanism (model authorization vs OPA binary allowlist). Its output recommended investigation: confirming by running with pre-#1160 images and checking Vertex gateway logs. Despite this self-acknowledged uncertainty, the triage applied the
ready-to-codelabel, which auto-dispatched a code agent that spent $10.15 over 20 minutes building a fix for the wrong problem. The code agent also modified.github/workflows/functional-tests.yml, which the coder app cannot push (noworkflowspermission). The human (waynesun09) had to manually correct the diagnosis 2 hours later and file #6971 with the actual root cause, leading to a successful $4.15 code run.What could go better
The triage output contained contradictory signals: it recommended investigation/confirmation steps while simultaneously applying
ready-to-code. A post-triage guard could detect this inconsistency and suppress the label, preventing the expensive code dispatch on an uncertain diagnosis. This complements agents#1055 (don't apply ready-to-code to investigation issues) but addresses a different pattern: a bug report whose triage recommends further investigation in its own output rather than an issue that is inherently investigative. Defense-in-depth is appropriate given the $10+ cost of a wrong dispatch. Confidence: high that this guard would have prevented the waste — the code agent inherited the wrong diagnosis with no independent validation. Uncertainty: it is possible the triage agent should simply not produce contradictory outputs, making a runtime guard unnecessary, but the agent's reasoning about sandbox error messages is inherently uncertain and a post-triage check is a more reliable mitigation.Proposed change
In the post-triage script (
scripts/post-triage.sh) infullsend-ai/agents, add a guard that inspects the triage agent's structured output for investigation-recommendation signals before applying theready-to-codelabel. If the triage output contains recommended actions that involve confirming a hypothesis, running experiments, or checking logs (pre-implementation investigation), the script should apply aneeds-investigationlabel instead ofready-to-codeand post a comment explaining that human confirmation is needed before code dispatch. The triage output schema (schemas/triage-result.schema.json) may need arequires_investigationboolean ordiagnosis_confidenceenum field to make this detection reliable rather than relying on text heuristics. This would serve as a defense-in-depth layer complementing the agent-level fix tracked in agents#1055.Validation criteria
Over the next 30 days, measure: (1) zero code agent runs triggered by triage outputs that simultaneously recommend investigation steps, and (2) no increase in false-positive investigation labels on issues with clear, actionable diagnoses. The next 5 triage runs whose output recommends confirmation or investigation steps should result in a
needs-investigationlabel rather thanready-to-code.Generated by retro agent from fullsend-ai/fullsend#6975