feat(review-agent): skip review dispatch that cannot produce a review - #6587
feat(review-agent): skip review dispatch that cannot produce a review#6587guyoron1 wants to merge 1 commit into
Conversation
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
1 similar comment
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
Site previewPreview: https://59df9f30-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Three cases where the review agent is dispatched, runs a full inference, and cannot say anything useful (ADR 0096). Drafts no longer trigger an automatic review on open or push — the review fires once the PR is marked ready. A fullsend-no-review label suppresses automatic dispatch, mirroring the fullsend-no-fix label the fix agent already honours. And a PR whose changed files are all documentation prose under docs/ is skipped with a notice in the job summary. That last pattern is deliberately narrow, because bash case globs match '/': docs/* would swallow the TypeScript under docs/.vitepress and the normative event schema, and a bare *.md would swallow skills/*/SKILL.md, AGENTS.md and CLAUDE.md — markdown that is executable agent instruction rather than prose. ADRs are excluded because a new ADR is among the things most worth reviewing, and lockfiles are not skippable at all: npm resolves from package-lock.json, so a lockfile-only diff can repoint a transitive dependency without touching package.json. A truncated file listing never skips either, since GitHub caps the files endpoint at 3000 entries and stops paginating without erroring. An explicit /fs-review bypasses all three skips. The draft and label checks are mirrored into the per-org scaffold, which docs/contributing/workflow-contracts.md requires for stage routing. The docs skip is not: that installation mode is deprecated (ADR 0044) and the requirement is scoped to routing, payload construction, and secret threading, none of which this step is. Rebased onto current main; renumbered the new ADR from 0091 to 0096 to avoid a collision with 0091-per-agent-runtime-model-effort.md, which landed upstream after this branch was cut. Signed-off-by: guy oron <goron@redhat.com>
05d40b9 to
ef01fdd
Compare
PR Summary by QodoSkip unnecessary automatic review dispatches
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo
1. Accepted ADR lacks architecture update
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| Three additive skips in `reusable-dispatch.yml`'s "Determine stage" step. | ||
|
|
||
| ### 1. Draft skip |
There was a problem hiding this comment.
2. Adr records three decisions 📜 Skill insight ⚙ Maintainability
ADR 0096 separately decides draft suppression, label-based suppression, and documentation-only suppression. These are three independently applicable routing decisions and must not be recorded as one ADR.
Agent Prompt
## Issue description
ADR 0096 records three distinct review-routing decisions in one Decision section.
## Issue Context
Each ADR must record exactly one decision; create separate ADRs with their own context and consequences.
## Fix Focus Areas
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[52-112]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| title: "96. Skip provably unnecessary review dispatch" | ||
| status: Accepted |
There was a problem hiding this comment.
3. Accepted adr lacks architecture update 📜 Skill insight ⚙ Maintainability
ADR 0096 is newly accepted, but this PR does not modify docs/architecture.md with a linked Decided: entry. The accepted routing decision therefore is not reflected in the repository's living architecture overview.
Agent Prompt
## Issue description
The new Accepted ADR has no corresponding architecture overview update.
## Issue Context
Add a surgical `Decided:` entry under the dispatch-related component and link ADR 0096; update related problem documents where the decision resolves an open question.
## Fix Focus Areas
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[1-21]
- docs/architecture.md[46-60]
- docs/problems/agent-architecture.md[1-1]
- docs/problems/code-review.md[1-1]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| - `/fs-review` remains a complete escape hatch for all three skips. A | ||
| maintainer can always force a review on a draft, a no-review-labeled PR, or | ||
| a prose-only PR by commenting. | ||
| - `fullsend-no-review` currently requires manual label creation and |
There was a problem hiding this comment.
6. Consequences use multiple sentences 📜 Skill insight ⚙ Maintainability
Several Consequences bullets contain multiple sentences and extended implementation rationale. The required format is 3–5 bullets with exactly one sentence per bullet.
Agent Prompt
## Issue description
The ADR Consequences bullets contain multiple sentences.
## Issue Context
Keep 3–5 bullets and rewrite every bullet as one concise sentence.
## Fix Focus Areas
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[114-137]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| Builds on [ADR 0034](0034-centralized-shim-routing-via-dispatch.md) | ||
| (centralized dispatch routing, and the `fullsend-no-fix` label pattern this | ||
| ADR mirrors) and [ADR 0054](0054-require-authorization-on-all-agent-dispatch-paths.md) | ||
| (authorization on all agent dispatch paths — unaffected by this change). |
There was a problem hiding this comment.
7. Adr links outside context 📜 Skill insight ⚙ Maintainability
The related ADR cross-references are placed between Status and Context rather than inside the Context section. This also introduces a free-standing block that does not follow the ADR template's section order.
Agent Prompt
## Issue description
Related ADR references appear outside the Context section and disrupt the required template structure.
## Issue Context
Preserve the references but incorporate them concisely into the Context section.
## Fix Focus Areas
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[19-30]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| | `ready-for-merge` | The review agent approved the PR. No blocking findings. | | ||
| | `requires-manual-review` | The review agent found issues that require human judgment — it could not confidently approve or reject. | | ||
| | `rejected` | The review agent rejected the PR and the post-script closed it. | | ||
| | `fullsend-no-review` | Prevents automatic (bot-triggered) review runs on this PR. Mirrors the [fix agent](fix.md)'s `fullsend-no-fix` label. Explicit `/fs-review` commands are unaffected. Currently applied manually (no `/fs-review-stop` command yet — see [ADR 0096](../ADRs/0096-skip-provably-unnecessary-review-dispatch.md)). | |
There was a problem hiding this comment.
8. Planned command lacks issue link 📜 Skill insight ≡ Correctness
The not-yet-implemented /fs-review-stop command is documented as follow-up work without a `> **Planned:**` callout or issue link. Readers cannot trace the planned feature and may mistake the prose for an informal commitment.
Agent Prompt
## Issue description
The guide mentions the unimplemented `/fs-review-stop` feature without the required planned-feature callout and tracking link.
## Issue Context
Create or reference the relevant issue and use the exact `> **Planned:**` blockquote format.
## Fix Focus Areas
- docs/agents/review.md[61-61]
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[74-82]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| if [[ "${EVENT_ACTION}" == "ready_for_review" || "${PR_IS_DRAFT}" != "true" ]] && ! has_label "fullsend-no-review" "${PR_LABELS}"; then | ||
| if [[ "${PR_USER_LOGIN}" =~ \[bot\]$ ]] || is_event_actor_authorized "${PR_USER_LOGIN}" triage; then | ||
| STAGE="review" | ||
| fi |
There was a problem hiding this comment.
9. Per-org changes not disclosed 📘 Rule violation § Compliance
The PR modifies routing in the deprecated per-org scaffold, but its description does not explicitly disclose that deprecated mode is affected or reference ADR 0044. The required deprecation warning is therefore absent from the review context.
Agent Prompt
## Issue description
The PR changes deprecated per-org installation scaffold behavior without the required PR-description disclosure.
## Issue Context
Update the PR description to state that deprecated per-org installation behavior is touched and explicitly reference ADR 0044.
## Fix Focus Areas
- internal/scaffold/fullsend-repo/.github/workflows/dispatch.yml[196-210]
- docs/ADRs/0096-skip-provably-unnecessary-review-dispatch.md[126-134]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| if [[ -z "${PR_NUMBER}" || "${PR_NUMBER}" == "null" ]]; then | ||
| exit 0 | ||
| fi | ||
| FILES=$(gh api "repos/${SOURCE_REPO}/pulls/${PR_NUMBER}/files" --paginate -F per_page=100 --jq '.[].filename') || { |
There was a problem hiding this comment.
10. Files request uses post 🐞 Bug ≡ Correctness
Adding -F per_page=100 makes gh api use POST unless --method GET is specified, but the pull-request files endpoint is a GET operation. The request therefore enters the fail-open handler without setting skipped=true, so documentation-only PRs continue dispatching reviews.
Agent Prompt
## Issue description
The changed-files request supplies `-F per_page=100` without explicitly selecting GET, causing `gh api` to send a POST request and making the documentation-only skip ineffective.
## Issue Context
The existing failure handler intentionally continues review dispatch when the API call fails, so this error does not fail the workflow but prevents every intended docs-only skip.
## Fix Focus Areas
- .github/workflows/reusable-dispatch.yml[555-558]
- internal/scaffold/workflow_call_alignment_test.go[653-654]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| if [[ -z "${PR_NUMBER}" || "${PR_NUMBER}" == "null" ]]; then | ||
| exit 0 | ||
| fi | ||
| FILES=$(gh api "repos/${SOURCE_REPO}/pulls/${PR_NUMBER}/files" --paginate -F per_page=100 --jq '.[].filename') || { |
There was a problem hiding this comment.
11. Renames bypass prose filter 🐞 Bug ≡ Correctness
The API query extracts only each file's new filename, so renaming an ADR or executable VitePress source into a docs/*.md path is classified as prose-only. Such a PR is skipped even though it removes or relocates content the change explicitly declares review-worthy.
Agent Prompt
## Issue description
The documentation-only filter ignores pull-file rename metadata and checks only the destination filename. A substantive file renamed into a matching Markdown path can therefore suppress review.
## Issue Context
Pull-file rename records include the old path separately as `previous_filename`. Either reject renamed files conservatively or classify both old and new paths, while preserving the 3000-file truncation safeguard.
## Fix Focus Areas
- .github/workflows/reusable-dispatch.yml[555-575]
- internal/scaffold/workflow_call_alignment_test.go[645-669]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep — 5 MEDIUM findings, 4 posted inline below; the one below targets a file not in this diff.
MEDIUM — User-facing bugfix guide still says closing the PR is the way to stop review (docs/guides/user/bugfix-workflow.md:169)
Verified on the PR head: docs/guides/user/bugfix-workflow.md:169 reads 'review is triggered automatically by PR events (pull_request_target), so closing the PR is the way to stop review dispatch'. This PR adds the fullsend-no-review label specifically as the lever to stop automatic review, and adds draft/docs-only skips, but the guide is not in the diff. The trigger descriptions at :83 ('review agents take over automatically'), :130/:134 ('Triggered by: pull_request_target events (PR opened, push to PR branch, ...)') and :183 ('Push commits ... the review agents will re-review') likewise do not mention the new skip exceptions. docs/agents/review.md was updated; this user-facing guide was not.
Suggestion: Update the 'Stopping automation' bullet at :169 to name the fullsend-no-review label as the way to stop automatic review (closing the PR is no longer the only option), and add a short note to the Stage 3 trigger description (:134) that drafts, fullsend-no-review-labeled PRs and docs-prose-only diffs are skipped unless /fs-review is used.
| [[ -z "$f" ]] && continue | ||
| case "$f" in | ||
| docs/ADRs/*) skippable=false; break ;; | ||
| docs/*.md) ;; |
There was a problem hiding this comment.
MEDIUM — docs/*.md skip swallows normative contract and contributing markdown the ADR says it protects
bash case globs match /, so the docs/*.md) ;; arm treats every nested markdown file under docs/ as skippable prose. Verified with the step's exact case block on the PR head: docs/normative/normalized-event/v1/README.md (NormalizedEvent v1 routing-input spec, ADR 0061), docs/normative/normalized-event/v1/jira-poll-adapter.md, docs/normative/prescript-output/v1/README.md (pre-script output contract) and docs/contributing/workflow-contracts.md (the sync contract governing the file this PR edits) all resolve to SKIP. ADR 0096 line 95 argues docs/* is dangerous precisely because it would swallow docs/normative/**, but the chosen pattern still swallows the normative prose specs alongside the JSON schema. A PR changing the documented meaning of a NormalizedEvent field without touching the JSON would receive no automatic review. docs/agents/review.md:47 documents only the ADR exclusion.
Suggestion: Add explicit non-skippable arms before docs/*.md: docs/normative/*|docs/contributing/*) skippable=false; break ;; (or invert to an allow-list of low-stakes prose directories such as docs/guides/*). Update ADR 0096 §3 and docs/agents/review.md:47 to state exactly which directories are excluded, and extend TestReviewRoutingDocsSkip to pin the new arms.
|
|
||
| - Fewer wasted review runs: drafts, `fullsend-no-review`-labeled PRs, and | ||
| documentation-prose-only PRs no longer trigger automatic review dispatch, | ||
| reducing inference cost with no loss of coverage — nothing that would have |
There was a problem hiding this comment.
MEDIUM — ADR 0096 asserts 'no loss of coverage' without evidence or threat-model evaluation
The Consequences bullet states 'no loss of coverage — nothing that would have been usefully reviewed is skipped' as settled fact, and the PR body says the work started from 'tallying dispatches', but no tally, cost figure, or sample of skipped PRs appears in the PR or ADR. Verified counter-evidence on head: (a) the glob already skips docs/normative/*.md and docs/contributing/*.md contract prose (see the comment at reusable-dispatch.yml:572); (b) the review agent's charter in fullsend-ai/agents docs/review.md lists 'docs currency' as a review dimension, and docs/agents/*.md + docs/contributing/*.md are the user-facing product surface — a docs-only follow-up to docs/agents/review.md (the page this PR edits) would itself be skipped; (c) docs/contributing/design-decisions.md:5 ranks external prompt injection as the top threat, and docs-only PRs are a low-friction path to land instruction-like prose that later agents read, with the automated reviewer disabled for exactly that path. The draft and label skips are self-evidently zero-loss; the docs skip is a judgement call shipped as fact. This is distinct from the existing threads on ADR format/length.
Suggestion: Either attach the dispatch tally (count and share of docs-only review runs over a stated window) and record in Consequences that the trade-off was evaluated against the injection threat model, or narrow the skip to an allow-list of low-stakes prose paths. In both cases soften the 'no loss of coverage' wording to describe the actual trade-off.
| The `/fs-review` command does not accept arguments. The review agent also runs automatically when a PR is opened, | ||
| synchronized (new commits pushed), or moved out of draft by a user with triage-level repository permission or higher. | ||
|
|
||
| ### Automatic skips |
There was a problem hiding this comment.
MEDIUM — fullsend-ai/agents docs/review.md 'Triggers' section is now stale (needs a companion PR in the agents repo)
Verified against the live fullsend-ai/agents main: docs/review.md 'Triggers' states the review agent runs automatically when 'A PR/MR is opened' and 'New commits are pushed to a PR/MR (synchronized)', with no draft, fullsend-no-review, or docs-only caveat, and its Control labels table has no fullsend-no-review row. After this PR those statements are wrong for drafts, labeled PRs and docs-prose-only PRs. That document lives in fullsend-ai/agents (ADR 0058 split) and cannot be fixed from this repo; the PR and ADR reference no companion change.
Suggestion: Open a companion PR in fullsend-ai/agents updating docs/review.md Triggers (draft exclusion, fullsend-no-review label, docs-only skip with the exact exclusions) and its Control labels table, and link it from this PR and ADR 0096. Do not attempt to fix it via internal/scaffold/ here.
| assert.Contains(t, s, "id: docs-lockfile-check") | ||
| assert.Contains(t, s, "steps.docs-lockfile-check.outputs.skipped != 'true'", | ||
| "job-level stage output must account for the docs skip") | ||
| assert.Contains(t, s, `if: steps.route.outputs.stage == 'review' && steps.role-check.outputs.skipped != 'true' && steps.agent-check.outputs.skipped != 'true'`, |
There was a problem hiding this comment.
MEDIUM — TestReviewRoutingDocsSkip pins substrings, not behaviour: passed green against a 404ing step and does not lock the /fs-review bypass guard
Verified on head. (1) The if: assertion at :651 checks only if: steps.route.outputs.stage == 'review' && steps.role-check.outputs.skipped != 'true' && steps.agent-check.outputs.skipped != 'true' and omits the trailing && github.event_name != 'issue_comment'; if that guard were removed the test still passes and the docs skip would apply to explicit /fs-review runs, contradicting ADR 0096's escape-hatch guarantee. (2) The gh-api pin at :653 matches the current command, which (per the existing thread at reusable-dispatch.yml:555) issues a POST and 404s, so the test passed CI while the feature it guards can never fire. (3) The two NotContains assertions at :663/:665 check literal strings (*.md|docs/*, package-lock.json|yarn.lock) that no plausible regression would produce, so they are trivially true. The PR body's claim that the tests were 'verified meaningful by reverting each guard' holds for TestReviewRoutingSkips' regexes but not for this function. hack/lint-workflow-size's own error text says logic belongs in scripts/ where it can be tested.
Suggestion: Add an explicit assertion that the docs-lockfile-check if: includes github.event_name != 'issue_comment'. When fixing the POST bug (the pin at :653 must change anyway), extract the step body to .github/scripts/check-docs-only-diff.sh and add a shell test that runs it with a stubbed gh on PATH covering: docs-only listing, a docs/ADRs/ entry, a docs/normative/ entry, a 3000-line listing, and a gh failure (fail-open). Keep the YAML test as a wiring check only, and cite a real dispatch run where the skip fired.
2 participants
Heyaa : )
This started from tallying dispatches where the review agent runs a full inference and cannot produce a useful review. Three cases, now skipped before dispatch:
fullsend-no-reviewlabel — mirroring thefullsend-no-fixlabel the fix agent already honours (ADR 0034 pattern).docs/*.mdminusdocs/ADRs/**, with a notice in the job log and summary.An explicit
/fs-reviewbypasses all three.The docs glob is deliberately narrow, because the obvious patterns are dangerous:
docs/*would swallow the TypeScript underdocs/.vitepress/and the publisheddocs/normative/**schema;*.mdwould swallowSKILL.md/AGENTS.md— markdown that is executable agent instruction, not prose; and a new ADR is among the things most worth reviewing.Two things are never skippable: lockfiles (a lockfile-only diff can repoint a transitive dependency's
resolved/integritywithout touchingpackage.json— supply-chain relevant, so Renovate PRs keep their review) and truncated file listings (GitHub caps/pulls/{n}/filesat 3000 entries without erroring).TestReviewRoutingSkips/TestReviewRoutingDocsSkippin the routing conditions and assert the dangerous broad globs are absent — verified meaningful by reverting each guard and watching them fail.go test ./internal/scaffold/andpre-commit run --all-filespass, including the ADR lint hooks.Note: the ADR is 0096 — 0090 and 0091 were both taken upstream while this branch was in flight. Known follow-up, disclosed in the ADR:
fullsend-no-reviewrequires manual label creation; there's no/fs-review-stopcommand yet, unlike/fs-fix-stop.