CI cannot AI-review a workflow-touching PR, but its required check still reports SUCCESS

[twistedmelonman]

Summary

claude-code-action refuses to run on any PR that touches .github/workflows/. The claude-review job handles this by skipping the review step — but the job still reports SUCCESS, and on most repos claude-review / run-review is a required check.

So a PR that modifies CI workflows passes its required AI review having reviewed nothing.

Observed

smartwatermelon/github-workflows#165, run 34550692109:

DOC_SKIP: true
DOC_SKIP_REASON: workflow-self-modification

##[notice] PR modifies .github/workflows/ (.github/workflows/claude-assistant.yml)
          — claude-code-action refuses to run by design. Skipping; real review
          will run on the next non-workflow PR after merge.

Step timings for that job:

Step	Result
5. Check for doc-only diff	success
10. Run Claude Code Review	skipped
13. Check review verdict	success

Job conclusion: success. Elapsed: 7 seconds (01:26:53 → 01:26:58).

For contrast, a genuine review on the same workflow — claude-config run 34551173017 — spends 96 seconds in step 10 (01:34:02 → 01:35:38). The 7s/96s gap is the tell.

That PR changed run-standards.sh, a reusable workflow, and a test file. None of it was AI-reviewed in CI.

Why the skip itself is correct

The underlying refusal is a sound security boundary: a PR that can modify the workflow reviewing it could neutralise its own review. claude-code-action declining is right, and this issue is not a request to bypass it.

What is wrong

Three things, in order of severity:

1. A required check reports SUCCESS without performing its function. This is the false-OK shape that has bitten repeatedly in this fleet (see the false-OK pattern notes). Green means "reviewed and passed" everywhere else; here it means "not reviewed".
2. The deferral does not do what it says. "Real review will run on the next non-workflow PR after merge" reviews a different diff, after the risky change has already landed on main. The workflow change is never reviewed, by anything, ever.
3. The blind spot is exactly inverted. The class of change most worth reviewing — CI/CD plumbing, permissions, triggers, secrets handling — is precisely the class CI structurally cannot review.

Relationship to the CI-deterministic decision

This is independent support for the decision recorded at docs/STATUS.md:203 ("No judgment reviewer stays in CI"), on structural rather than cost grounds.

Local hooks have no such constraint. #165's workflow diff was reviewed locally — code-reviewer and adversarial-reviewer both PASS, plus a codebase-mode review — because local review reads the diff directly instead of running inside the permission boundary the change would alter.

Options

• A — Fold into W3 and drop it. Once standards-check is required and claude-blocking-review.yml is removed, this disappears with it. Zero extra work; the gap persists until W3 lands.
• B — Fail the check instead of passing it. Make the workflow-self-modification path report failure (or neutral, where branch protection accepts it), forcing a human to acknowledge the change went unreviewed. Honest, but noisy on every workflow PR, and neutral-vs-failure interacts with required-check config per repo.
• C — Surface it without gating. Keep SUCCESS but post a PR comment naming the unreviewed files, so the reviewer knows. Cheapest; relies on someone reading it.

Recommend A, with C as a stopgap if W3 slips — the gap is real but its window is bounded by work already planned.

Verification notes

• DOC_SKIP_REASON: workflow-self-modification was read from the run log, not inferred from the step name.
• The skip is specific to workflow-touching PRs. It is not universal: the same workflow performed a real 96-second review on a non-workflow PR in the same hour.
• No evidence of token/session limits in these runs — the Verify CLAUDE_CODE_OAUTH_TOKEN is configured step passes wherever it executes. A separate concern about session-limited tokens was raised but is not what this run shows.

Found while investigating standards-check failure rates (github-workflows#165, dev-env#115).

[twistedmelonman]

Still live, and confirmed today on a PR that was actually merged — smartwatermelon/github-workflows#166, which modified .github/workflows/standards-check.yml.

Run 34566467346, job claude-review / run-review:

Step	Result
5. Check for doc-only diff	success
6. Check for Dependabot PR	skipped
9. Verify CLAUDE_CODE_OAUTH_TOKEN is configured	skipped
10. Run Claude Code Review	skipped
11. Probe GitHub API after review failure	skipped

Job conclusion: SUCCESS. Required check satisfied. Nothing reviewed. I merged that PR on the strength of a green check that had reviewed zero lines.

What changed today

1. standards-check does not have this failure mode. On the same PR #166 — a workflow-touching PR — it ran and passed for real:

claude-review / run-review           = success   <- reviewed nothing
standards-check / run-standards-check = success  <- actually ran
guard-no-checkout                     = success

claude-code-action refuses to run on workflow-touching PRs by design; shellcheck, markdownlint and zizmor have no such restriction. So the W3 flip gives these repos a required check that still means something on exactly the PRs where claude-review goes hollow.

2. Andrew's direction (2026-09-11) is to deprecate CI-side AI review, keeping it only for isolated cases. That reframes this issue: the fix is probably not "make the skip report failure" but "stop depending on a check with this failure mode."

Why it still matters

W3 is an add, not a replace — claude-review / run-review is still required on dev-env, claude-config, dotfiles and every other protected repo. Until it is removed, the hole described here is open on every workflow-touching PR fleet-wide.

Two coherent end states:

• Remove claude-review from required_status_checks once standards-check has ridden enough PRs, which closes this by deletion. Consistent with the deprecation direction and with deprecate extensive claude instructions and hopes and dreams in favor of determinitive non-skippable checks #114.
• Keep it and make the skip report failure, which closes this by fixing the gate. Only worth building if AI review stays required.

The second is wasted work if the first is where this is heading. Worth deciding before anyone implements a fix.

Related: #89 (closed today — .github now requires standards-check instead of getting the AI review workflow), #128 (W3 flip detail), #114.

[twistedmelonman]

Closed on four repos; the rest is gated on a constraint worth recording

Took the "remove claude-review from required_status_checks" branch from the previous comment. It is done where it is safe to do, which turned out to be four repos, not the fleet.

Fleet survey (2026-09-11)

42 protected repos require claude-review / run-review. Only three also required standards-check / run-standards-check: dev-env, twistedmelonman/claude-config, twistedmelonman/dotfiles.

Count correction for anything that cites an earlier number: smartwatermelon/claude-config and smartwatermelon/dotfiles are owner redirects to twistedmelonman, so a naive gh repo list sweep double-counts them. Both names resolve to the same repo and the same protection record.

Why this cannot be a blanket removal

dependabot-auto-merge.yml:240 merges with gh pr merge --auto. The workflow's own comment states the dependency:

# required. `gh pr merge --auto` means the merge only happens after
# all status checks pass; failing CI leaves the PR open indefinitely.

On a repo whose only required check is claude-review, removing it leaves zero required checks — and --auto then merges every Dependabot PR the moment it opens, with nothing gating it. That is a strictly worse false-OK than the one this issue describes: #116 is a check that reviews nothing, that would be no check at all.

So the removal is safe exactly where another required check remains behind it.

What changed

github-workflows turned out to be a fourth safe repo. It already runs standards-check / run-standards-check via self-standards-check.yml, green on five consecutive PRs — including #166, the PR named in the comment above, where claude-review went hollow on the same commit that standards-check reviewed for real. It was simply never promoted to required. Sequence there was: add standards-check to required, confirm, then drop claude-review.

Final state on all four:

Repo	Required checks
smartwatermelon/dev-env	standards-check / run-standards-check
twistedmelonman/claude-config	standards-check / run-standards-check
twistedmelonman/dotfiles	standards-check / run-standards-check
smartwatermelon/github-workflows	standards-check / run-standards-check

Verified by re-reading each protection record after the change and diffing it against the pre-change copy with the contexts list removed: no collateral drift. enforce_admins (false), strict (true), required_conversation_resolution (true), review settings and force-push/deletion flags are all unchanged. Pre-change JSON for all four was saved before the first PATCH.

github-workflows is the repo where this issue actually bit, and it is now closed there.

Residual

38 repos still require claude-review as their only required check, so the hole stays open on every workflow-touching PR there. Removing it on those is blocked on giving them a required check that survives the removal — which is W3 steps 2-5, already planned. Option A from the original post, with the dependency now stated in mechanical terms rather than as a preference for sequencing.

Leaving this open to track those 38.