feat(triage): add not-planned action to close out-of-scope issues

[rh-hemartin]

Summary

Add not-planned action to triage agent for closing issues as out-of-scope, invalid, or spam.

Related Issue

Closes fullsend-ai/fullsend#2205

Changes

• Schema: Add not-planned to action enum in triage-result.schema.json
• Post-script: Add not-planned case that applies label and closes issue with reason not planned
• Control labels: Add not-planned to CONTROL_LABELS array
• Agent prompt: Document when to use not-planned vs other actions
• Documentation: Add not-planned to control labels table in docs/agents/triage.md
• Tests: Add 6 new tests covering not-planned action behavior

Test plan

• All existing tests pass
• New tests cover: comment posting, label application, blocked/needs-info removal, issue closing, missing comment validation
• Schema validates as valid JSON
• Lint checks pass

🤖 Generated with Claude Code

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 6:45 AM UTC · Completed 6:57 AM UTC
Commit: 04de034 · View workflow run →

[qodo-code-review]

PR Summary by Qodo

Add not-planned triage action to label and close out-of-scope issues

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Extend triage result schema with a not-planned action for out-of-scope/invalid/spam issues.
• Teach post-triage automation to apply not-planned and close issues with reason "not planned".
• Document usage guidance and add automated coverage for schema + post-triage behavior.

Diagram

graph TD
  A["Triage agent prompt"] --> B["Triage result JSON"] --> C["Schema validation"] --> D["post-triage.sh"] --> E{{"GitHub Issue"}}
  A --> F["Triage docs"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Generic `close` action with `close_reason` field

• ➕ More extensible (add future close reasons without expanding the enum repeatedly)
• ➕ Keeps schema expressive (reason is explicit rather than implied by action name)
• ➖ Bigger schema + script refactor and more validation logic
• ➖ Less user-friendly prompt semantics vs a single explicit not-planned action

2. Drive closure via `label_actions` only (pipeline reacts to labels)

• ➕ Keeps agent action space smaller
• ➕ Centralizes policy in the pipeline rather than action handlers
• ➖ Less explicit intent (labels used as control-plane signals are easy to misuse)
• ➖ Requires additional pipeline logic beyond this script; harder to test end-to-end here

Recommendation: Keep the explicit not-planned action as implemented. It’s the smallest cross-cutting change that clearly encodes intent, matches existing patterns (control labels + post-action close like duplicate), and is easy to validate/test. The generic close action is a reasonable future refactor if multiple close reasons are expected to grow.

Files changed (5) +69 / -3

Enhancement (1) +16 / -2

post-triage.shHandle 'not-planned': apply label, require comment, close issue +16/-2

Handle 'not-planned': apply label, require comment, close issue

• Adds 'not-planned' to control labels, implements an action handler that requires a comment, removes 'blocked'/'needs-info', applies 'not-planned', and closes the issue with reason "not planned".

scripts/post-triage.sh

Tests (2) +29 / -0

post-triage-test.shAdd end-to-end tests for 'not-planned' behavior +25/-0

Add end-to-end tests for 'not-planned' behavior

• Introduces 6 tests verifying comment posting, label application, removal of conflicting labels, issue closure, and failure when 'comment' is missing.

scripts/post-triage-test.sh

validate-output-schema-test.shAdd schema validation test for 'not-planned' output +4/-0

Add schema validation test for 'not-planned' output

• Adds a positive test case ensuring a 'not-planned' triage result validates against the updated JSON schema.

scripts/validate-output-schema-test.sh

Documentation (1) +23 / -0

triage.mdDocument 'not-planned' action semantics and JSON example +23/-0

Document 'not-planned' action semantics and JSON example

• Adds a new section defining when to use 'not-planned' and when not to (vs 'insufficient', 'duplicate', 'prerequisites'). Includes a concrete JSON payload example emphasizing a respectful explanatory comment.

agents/triage.md

Other (1) +1 / -1

triage-result.schema.jsonAllow 'not-planned' in triage action enum +1/-1

Allow 'not-planned' in triage action enum

• Extends the 'action' enum to include 'not-planned', enabling schema validation of the new triage outcome.

schemas/triage-result.schema.json

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (1)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. Stale not-planned label persists ✓ Resolved 🐞 Bug ≡ Correctness

Description

post-triage.sh adds the new not-planned control label, but neither pre-triage.sh nor other
action handlers remove it, so a reopened/re-triaged issue can end up with not-planned plus another
control label (e.g., ready-to-code). This breaks the pipeline’s mutual-exclusion expectations and
can misroute downstream automation that keys off control labels.

Code

scripts/post-triage.sh[R344-352]

+  not-planned)
+    if [[ -z "${COMMENT}" ]]; then
+      echo "ERROR: action is 'not-planned' but no comment provided" >&2
+      exit 1
+    fi
+    remove_label "blocked"
+    remove_label "needs-info"
+    add_label "not-planned"
+    ;;

Relevance

⭐⭐⭐ High

Repo prioritizes control-label consistency; PR #40 added guards preventing contradictory label
actions, so stale-label fix likely accepted.

PR-#40

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

post-triage.sh now applies not-planned, but pre-triage.sh does not delete or verify removal of
not-planned during its reset, and none of the other action handlers remove not-planned either.
Therefore, once an issue is labeled not-planned, the label can persist into future runs and
coexist with other control labels.

scripts/post-triage.sh[78-92]
scripts/post-triage.sh[101-123]
scripts/post-triage.sh[245-333]
scripts/post-triage.sh[344-352]
scripts/pre-triage.sh[30-45]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new `not-planned` control label is applied by `post-triage.sh` but is not cleared by `pre-triage.sh` and is not removed by other action handlers. This allows stale `not-planned` labels to persist across future triage runs (e.g., when an issue is reopened), creating conflicting control-label state.

## Issue Context
- `pre-triage.sh` is the pipeline’s label baseline reset script.
- `post-triage.sh` action handlers generally remove incompatible control labels (e.g., `blocked`, `needs-info`), but do not clear `not-planned` for non-`not-planned` actions.

## Fix Focus Areas
- scripts/pre-triage.sh[30-45]
- scripts/post-triage.sh[101-358]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. scripts/ and agents/ changed 📜 Skill insight § Compliance

Description

This PR modifies protected governance/infrastructure paths (agents/ and scripts/), which require
explicit human review and must not be auto-approved. A compliance finding is required whenever
protected paths are changed.

Code

scripts/post-triage.sh[R344-353]

+  not-planned)
+    if [[ -z "${COMMENT}" ]]; then
+      echo "ERROR: action is 'not-planned' but no comment provided" >&2
+      exit 1
+    fi
+    remove_label "blocked"
+    remove_label "needs-info"
+    add_label "not-planned"
+    ;;
+

Relevance

⭐⭐ Medium

Some compliance concerns accepted for workflow changes (PR #25), but no clear precedent for
scripts/agents protected-path rule.

PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The diff introduces new/modified logic under scripts/ and modifies an agent prompt under
agents/, both of which are explicitly listed as protected paths that must trigger a compliance
finding.

scripts/post-triage.sh[344-353]
agents/triage.md[178-200]
Skill: pr-review

---

3. Docs omit not-planned label ✓ Resolved 📜 Skill insight ⚙ Maintainability

Description

The triage pipeline now treats not-planned as a control label, but docs/triage.md still lists
only the older control labels even though the harness directs users to that document. This leaves
user-facing documentation out of sync with post-triage.sh, so operators and skill authors lack
guidance on interpreting/configuring the new outcome and label expectations.

Code

scripts/post-triage.sh[82]

+CONTROL_LABELS=("needs-info" "ready-to-code" "duplicate" "feature" "blocked" "triaged" "question" "not-planned")

Relevance

⭐⭐⭐ High

Team previously fixed stale triage docs when adding question control label (PR #8); also accepts
doc clarifications (PR #25).

PR-#8
PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The implementation in scripts/post-triage.sh includes not-planned in CONTROL_LABELS,
indicating it is pipeline-managed, but docs/triage.md does not include not-planned in its
control-label list/table or the embedded issue-labels guidance. Additionally, the harness
configuration points users to docs/triage.md for triage documentation, so the omission directly
impacts the primary user reference despite the code’s updated behavior.

scripts/post-triage.sh[78-83]
docs/triage.md[32-44]
docs/triage.md[79-83]
harness/triage.yaml[1-4]
scripts/post-triage.sh[78-92]
Skill: code-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Update the user-facing triage documentation to reflect that `not-planned` is now a pipeline-managed control label: `docs/triage.md` should include `not-planned` alongside the other control labels and clarify that it should not be recommended via `label_actions`.

## Issue Context
`scripts/post-triage.sh` now manages a `not-planned` action/label as part of `CONTROL_LABELS`, but `docs/triage.md` still documents only the older control labels (including in its control-label table and `issue-labels` guidance). The harness configuration also references `docs/triage.md` as the triage agent documentation, so keeping it accurate is important for operators and skill authors interpreting outcomes and configuring label expectations.

## Fix Focus Areas
- docs/triage.md[32-46]
- docs/triage.md[79-83]
- harness/triage.yaml[1-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

Review

Well-structured addition of the not-planned triage action. The implementation is consistent and complete across all touchpoints:

• Schema — not-planned added to the action enum. No conditional allOf rule needed (base required fields suffice, matching the question pattern).
• Post-script — Case handler validates comment, removes blocked/needs-info, adds label. Issue close uses hardcoded --reason "not planned". Comment posted via gh issue comment with --body-file - (safe stdin piping).
• Pre-triage — Label reset loop and verification query both include not-planned, maintaining the mutual-exclusion invariant.
• Control labels — not-planned added to CONTROL_LABELS array, preventing agent manipulation via label_actions.
• Agent prompt — Anti-premature-closure guardrail requires citing specific scope boundaries. Clear documentation of when to use vs. not use not-planned.
• Tests — Six post-triage tests (comment, label, blocked removal, needs-info removal, close, missing-comment validation) plus one schema validation test and one control-label refusal test. Coverage matches existing action test patterns.
• Eval cases — Positive case (005: out-of-scope chat/video request) and negative guardrail case (006: in-scope MFA request) provide evaluation coverage.

No issues found across correctness, security, intent, style, documentation, or cross-repo contract dimensions.

Findings

Medium

• [protected-path] agents/triage.md, scripts/post-triage.sh, scripts/pre-triage.sh, scripts/post-triage-test.sh, scripts/validate-output-schema-test.sh — This PR modifies files under protected paths (agents/, scripts/). The PR links to an issue and explains the rationale. Human approval is always required for protected-path changes, regardless of context.

---

Labels: PR modifies triage agent prompt, post-script, pre-script, and schema.

Previous run

Review

Well-structured addition of the not-planned triage action. All prior review findings have been resolved. The implementation is consistent and complete across all touchpoints:

• Schema — not-planned added to the action enum. No conditional allOf rule needed (base required fields suffice, matching the question pattern).
• Post-script — Case handler validates comment, removes blocked/needs-info, adds label. Issue close uses hardcoded --reason "not planned". Comment posted via gh issue comment with --body-file - (safe stdin piping).
• Pre-triage — Label reset loop and verification query both include not-planned, maintaining the mutual-exclusion invariant.
• Control labels — not-planned added to CONTROL_LABELS array, preventing agent manipulation via label_actions.
• Agent prompt — Anti-premature-closure guardrail requires citing specific scope boundaries. Clear documentation of when to use vs. not use not-planned.
• Tests — Six post-triage tests (comment, label, blocked removal, needs-info removal, close, missing-comment validation) plus one schema validation test. Coverage matches existing action test patterns.
• Eval cases — Positive case (005: out-of-scope chat/video request) and negative guardrail case (006: in-scope MFA request) provide evaluation coverage.

No issues found across correctness, security, intent, style, documentation, or cross-repo contract dimensions.

Findings

Medium

• [protected-path] agents/triage.md, scripts/post-triage.sh, scripts/pre-triage.sh, scripts/post-triage-test.sh, scripts/validate-output-schema-test.sh — This PR modifies files under protected paths (agents/, scripts/). The PR links to an issue and explains the rationale. Human approval is always required for protected-path changes, regardless of context.

Previous run (2)

Review

Well-structured addition of the not-planned triage action. All prior review findings have been resolved — pre-triage.sh now includes not-planned in both the label reset loop and the verification query, and docs/triage.md is updated with the control labels table row and list entry.

The implementation is consistent and complete across all touchpoints:

• Schema — not-planned added to the action enum. No conditional allOf rule is needed since the action requires only the base fields (action, reasoning, comment), matching the question action pattern.
• Post-script — Case handler mirrors question (comment required, removes blocked/needs-info, adds label). Close logic mirrors duplicate with --reason "not planned". Comment posted via plain gh issue comment (correct — only sufficient uses sticky comments).
• Pre-triage — Label reset loop and verification query both include not-planned, maintaining the mutual-exclusion invariant.
• Control labels — not-planned added to CONTROL_LABELS array, preventing agents from manipulating it via label_actions.
• Agent prompt — Clear documentation of when to use not-planned vs other actions, with explicit "Do NOT use" guidance.
• Tests — Six post-triage tests (comment posting, label application, blocked removal, needs-info removal, issue closing, missing-comment validation) plus one schema validation test. Coverage matches the question and duplicate test patterns.

No issues found across correctness, security, intent, style, documentation, or cross-repo contract dimensions.

---

Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

• agents/triage.md
• scripts/post-triage-test.sh
• scripts/post-triage.sh
• scripts/pre-triage.sh
• scripts/validate-output-schema-test.sh

Previous run (3)

Review

Well-structured addition of the not-planned triage action. The implementation follows established patterns consistently: the case handler mirrors the question action's label cleanup, the issue close mirrors the duplicate action's close pattern, tests are comprehensive (6 post-triage tests + 1 schema validation test), and the agent prompt clearly documents when to use not-planned vs other actions.

One consumer of the triage action enum was not updated in this PR.

Findings

1. scripts/pre-triage.sh — label reset loop and verification omit not-planned (medium)

pre-triage.sh resets triage labels before each re-triage run to enforce mutual exclusivity (Story 2, #125). The reset loop (line 30) strips needs-info, ready-to-code, duplicate, feature, and question — but not not-planned:

for label in needs-info ready-to-code duplicate feature question; do

The verification query (line 35-36) also does not check for not-planned:

select(.name == "needs-info" or .name == "ready-to-code" or .name == "duplicate" or .name == "feature" or .name == "question")

Impact: If an issue closed as not-planned is reopened and re-triaged, the stale not-planned label persists alongside whatever new triage label is applied, violating the mutual-exclusion invariant. The duplicate label (which also triggers issue closure) is in the reset loop, so not-planned should be too.

Remediation: Add not-planned to both the for loop on line 30 and the jq select filter on line 36.

2. docs/triage.md — control labels table not updated (low)

The control labels table in docs/triage.md lists needs-info, ready-to-code, triaged, duplicate, and blocked but does not include not-planned. The PR body claims documentation was updated, but docs/triage.md is not in the changed files.

Remediation: Add a row: | \not-planned` | The issue is out of scope, invalid, or spam. The post-script closes the issue with reason "not planned". |`

Notes

• The linked issue (#2205) returns HTTP 404 from the API. This may be a private issue or a reference to a different repository — not flagged as a finding but worth verifying the reference.
• The schema does not add a conditional allOf rule for not-planned, which is correct — the base required: ["action", "reasoning", "comment"] already enforces all needed fields, matching the question action's pattern.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:38 AM UTC · Completed 9:47 AM UTC
Commit: 35a059c · View workflow run →

[waynesun09]

Multi-agent review (Claude, Gemini, Codex reviewers, findings verified before posting). The mechanical wiring is clean — schema enum, CONTROL_LABELS, the pre-triage reset loop and its verification jq filter, close ordering, and gh issue close --reason "not planned" were all independently confirmed correct and consistent with the existing duplicate flow; both test suites pass. Two medium findings posted inline; no critical/high issues found.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:49 AM UTC · Ended 8:01 AM UTC
Commit: 5cd495a · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:02 AM UTC · Completed 8:14 AM UTC
Commit: 525b015 · View workflow run →

[waynesun09]

Multi-agent review (Claude ×2, Grok). Both prior findings verified as fixed. One non-blocking nit below.

Assisted-by: Claude (review ×2), Grok (review)

[waynesun09]

[medium, non-blocking] missing-test-coverage

Existing tests verify the control label guard for question (line 315), ready-to-code (line 338), and feature (line 342), but no test verifies that not-planned is refused when added via label_actions. The guard logic works (same is_control_label() function), but the test pattern isn't followed for the new label.

Suggested fix: Add a not-planned-control-label-refused test in scripts/post-triage-test.sh:

run_test_stdout "not-planned-control-label-refused" \
  '{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Fix crash","severity":"high","category":"bug","problem":"Crash","root_cause_hypothesis":"Buffer overflow","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Fix buffer","proposed_test_case":"test_crash"},"comment":"## Triage Summary\n\nReady.","label_actions":{"reason":"Tried to set not-planned label.","actions":[{"action":"add","label":"not-planned"}]}}' \
  "::warning::Refused to add control label 'not-planned' -- control labels are managed by the triage pipeline"

Assisted-by: Grok (review)

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:54 AM UTC · Completed 8:10 AM UTC
Commit: ab33774 · View workflow run →