Skip to content

claude-blocking-review reports green on workflow-only PRs, so the required check passes unreviewed #173

Description

@twistedmelonman

The gap

claude-blocking-review.yml:344-366 short-circuits when any changed file matches .github/workflows/*.yml|*.yaml:

echo "skip=true" >> "$GITHUB_OUTPUT"
echo "skip_reason=workflow-self-modification" >> "$GITHUB_OUTPUT"
echo "**Verdict:** SKIPPED (workflow-self-modification)" >> "$GITHUB_STEP_SUMMARY"
exit 0

The reason is sound: anthropics/claude-code-action refuses to run against a PR that modifies the reviewer, and without the skip the step fails with a misleading "review did not complete." The problem is the reporting, not the skip.

Because this is a required check, exit 0 makes the check report SUCCESS. On the PR page a workflow-only change is indistinguishable from a reviewed-and-passed change — same green tick. The one class of change that can alter CI enforcement itself is the class that silently bypasses review.

Why it matters more than it looks

The comment notes this "hits EVERY Dependabot-generated PR that bumps the pin, plus any manual caller-workflow edit." Since the fleet moved to floating @v3 first-party refs, caller-workflow edits are rarer — but that cuts the wrong way: it means the remaining workflow-only PRs are disproportionately deliberate changes to enforcement behaviour, which is precisely what most deserves review.

"A real review runs on the next non-workflow PR after merge" is true but does not cover this: by then the change is already on main and the required check has already reported green.

This is an instance of the false-OK pattern — a gate that reports success while measuring nothing.

Options

  1. Neutral instead of success. Report the check as neutral rather than passing. Needs a check against branch protection: a neutral conclusion may or may not satisfy "required," so this must be measured before adopting, not assumed.
  2. Distinguishable green. Keep exit 0 but make the check name or summary carry the skip, so a human scanning the PR sees "SKIPPED (workflow-self-modification)" rather than an unqualified tick. Cheapest, and does not risk blocking the merge path.
  3. Review the diff out-of-band. Run the review in a job that does not use claude-code-action against itself — e.g. review the diff as text in a separate job — so workflow PRs get a real verdict.

Option 2 is the minimum. Option 1 is the correct fix if branch protection tolerates neutral.

Provenance

Carried in session handoffs since at least 2026-09-11 as "claude-blocking-review returning a green SKIP on workflow-only PRs (STILL NOT FILED as an issue)." Verified against the current file 2026-09-18 before filing; the behaviour is unchanged and no existing issue covers it (#481, #451 and #427 are different skip paths in run-review.sh).

https://claude.ai/code/session_011awg91UvzUos9YoXHJ2e8B

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions