Skip to content

test(ci): pin the no-visual-change waiver so it cannot rot - #553

Open
chaitanyagiri wants to merge 1 commit into
mainfrom
ci/pin-the-evidence-waiver
Open

chaitanyagiri wants to merge 1 commit into
mainfrom
ci/pin-the-evidence-waiver

Conversation

@chaitanyagiri

Copy link
Copy Markdown
Owner

The escape already exists. This pins it.

.github/workflows/pr-evidence.yml has had the no-visual-change waiver since the gate shipped, at lines 55 to 59: checked before any parsing, returning early, exact label membership, case insensitive. The bot's own failure comment already tells the author about it. The label already exists in this repository.

I used it on #547 today and Before / after evidence passed in three seconds.

So this PR does not build the escape. pr-evidence.yml is not modified by this commit at all. It does the two things that were actually missing.

1. It was completely untested

A branch that lets a required status check pass, with nothing asserting when it may and may not fire, is the dangerous shape. It can rot in two directions and neither is loud:

  • it stops working, and a CI tweak or a typo has no way through a required check
  • it starts always firing, and the gate is decorative on every PR and nobody notices

Eight tests, in the style already in this file: the label name, the ordering and the guard are read out of the workflow rather than restated, so editing the workflow carelessly fails here instead of on a contributor's PR.

test guards against
the label still matches the template and the bot comment the three copies drifting apart
the waiver is checked BEFORE the evidence is computed a waived PR still getting a "missing evidence" comment
the waiver returns early rather than falling through passing the branch but failing anyway
it FIRES with the label, at any position the escape silently breaking
it is case insensitive labels are typed by hand
it does NOT fire without the label the always-pass rot
it needs the WHOLE label needs-no-visual-change walking through

Mutation results, run before committing

mutation tests failing
rename the label in the workflow only 3
substring match instead of exact membership 6
remove the early return 1
move the waiver after the evidence parsing 1

Baseline is 15 pass, 0 fail.

2. The template only mentioned it inside an HTML comment

So it never rendered. There is now a visible note under ## Evidence naming the label and saying it is the supported way past the check.

Two further tests prove that adding visible text changed nothing: an unfilled template still fails the gate, and the note itself is not mistaken for evidence.

Why 42 PRs are stuck

Since that was the reason for the work: not a missing escape. Of 93 open pull requests, exactly one carries the label (#498). The escape works and nobody is applying it.

Labelling them is a maintainer action and is not in this change.

Before and after

No visual surface. This is a test file and three lines of template text, so the no-visual-change label applies to this PR by its own rule.

Evidence that the tests do their job is the mutation table above: four ways of breaking the waiver, all caught.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PogxmT9ULRQSYbfs1oigLx

… template

THE ESCAPE ALREADY EXISTS. `.github/workflows/pr-evidence.yml` has had the
`no-visual-change` waiver since the gate shipped, at lines 55 to 59, checked
before any parsing and returning early. The bot's own failure comment already
tells the author about it, and the label already exists in the repository. I
used it on #547 today and the check passed in three seconds.

So this change does NOT build the escape. It does the two things that were
actually missing.

ONE. IT WAS COMPLETELY UNTESTED. A branch that lets a required status check pass,
with nothing asserting when it may and may not fire, is the dangerous shape: it
can rot in two directions and neither is loud. If it stops working, a CI tweak or
a typo has no way through a required check. If it starts always firing, the gate
is decorative on every pull request and nobody notices.

Eight tests, in the style already in this file: the label name, the ordering and
the guard are read OUT of the workflow rather than restated, so editing the
workflow carelessly fails here instead of on a contributor's pull request.

  the label still matches the template and the bot comment
  the waiver is checked BEFORE the evidence is computed
  the waiver returns early rather than falling through
  it FIRES with the label, at any position in the list
  it is case insensitive, because labels are typed by hand
  it does NOT fire without the label          <- the always-pass guard
  it needs the WHOLE label, so `needs-no-visual-change` does not walk through

All four mutations go red, checked before committing:
  rename the label in the workflow only            3 tests fail
  substring match instead of exact membership      6 tests fail
  remove the early return                          1 test fails
  move the waiver after the evidence parsing       1 test fails

TWO. THE TEMPLATE ONLY MENTIONED IT INSIDE AN HTML COMMENT, so it never rendered.
There is now a visible note under `## Evidence` naming the label and saying it is
the supported way past the check. Two further tests prove that adding visible
text changed nothing: an unfilled template still fails, and the note itself is
not mistaken for evidence.

`.github/workflows/pr-evidence.yml` IS NOT MODIFIED BY THIS COMMIT.

WHY 42 PULL REQUESTS ARE STUCK, since that was the reason for the work: not a
missing escape. Of 93 open pull requests exactly one carries the label. The
escape works and nobody is applying it. Labelling them is a maintainer action
and is not in this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PogxmT9ULRQSYbfs1oigLx
@chaitanyagiri chaitanyagiri added the no-visual-change Maintainer waiver: this change has no observable before/after label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-visual-change Maintainer waiver: this change has no observable before/after

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant