Skip to content

ci: scope a pull request's conformance to the cases it changed - #176

Merged
naxty merged 2 commits into
mainfrom
naxty/gcpConformancePR
Aug 31, 2026
Merged

ci: scope a pull request's conformance to the cases it changed#176
naxty merged 2 commits into
mainfrom
naxty/gcpConformancePR

Conversation

@naxty

@naxty naxty commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

Conformance doesn't run on pull requests. discover-tests is gated on github.event_name != 'pull_request', which skips the matrix, pre-cleanup and post-cleanup with it. A new or edited fixture is unverified until it reaches main, and the only early check is hand-dispatching debug-conformance.yml with the case names typed out.

What changes

ci.yml resolves its conformance scope from the event:

Event Scope
push to main / workflow_dispatch every case in testdata/ (117)
pull request only the cases the PR touched
pull request + full-conformance label every case in testdata/

A PR touching no fixture resolves to zero and skips conformance-tests, pre-cleanup and post-cleanup entirely — a plumbing or refactor PR stops paying for 117 live GCP lifecycles.

Unlike the Azure sibling (formae-plugin-azure#129), there is no curated matrix file here: full scope stays the testdata/ discovery ci.yml already did, moved into the script unchanged.

Mapping rules

In scripts/ci/conformance-scope.sh:

  • Added / copied / modified / renamed testdata/*.pkl are in scope. Deletions ignored — nothing left to run.
  • An -update / -replace companion maps back to its case, since one matrix entry drives the whole lifecycle. Editing only bucket-update.pkl still runs bucket — which is exactly what this PR does to itself, see Verification.
  • testdata/config/ is shared Pkl, not fixtures. Git's pathspec glob matches /, so -- 'testdata/*.pkl' also catches testdata/config/vars.pkl; without an explicit top-level-only filter it would resolve to a bogus case named config/vars and fail the run. Filtered, with a test.
  • A changed companion with no base fixture fails the run rather than filtering to nothing and passing green.

.github/conformance-pr-skip.txt — mechanism only, no file

Names in this file are subtracted from a PR's scope, for cases that cannot pass in CI regardless of the diff. The file is not in this PR. The AlloyDB and VPN-gateway families are the obvious candidates on cost and runtime, but they run on push today and I have no evidence they are PR-unrunnable, so seeding them would be a claim I can't back. Add lines when a case proves it.

A name in the file matching no fixture fails the run, so the two can't drift into disagreement. The file does not affect push or nightly.

Known gap

A change under pkg/ or testdata/config/vars.pkl affects every resource but touches no fixture, so it resolves to an empty scope. Two backstops: label the PR full-conformance, or rely on push-to-main, which still runs everything. Documented in CONTRIBUTING.md rather than auto-escalated — escalating a one-line vars.pkl edit to 117 live cases costs more than it catches.

Verification

scripts/ci/conformance-scope_test.sh — 13 cases against throwaway git repos, no cloud creds, wired into the test-unit job so it can't rot:

ok   added fixture is in scope
ok   -update companion maps back to its case
ok   -replace companion maps back too
ok   editing only the companion still runs the case
ok   deletion alone is out of scope
ok   non-fixture change is out of scope
ok   shared testdata/config Pkl is not a case
ok   several cases are deduped and sorted
ok   skip-listed case is subtracted
ok   every touched case skip-listed leaves an empty scope
ok   unknown skip-list name fails loudly
ok   orphan companion fails loudly
ok   push and full-conformance label use the whole matrix

13 passed, 0 failed

Resolved against real history too. Replaying 6ad92ea ("20 more resource types") as if it were a PR reproduces its fixture set rather than the whole matrix:

6ad92ea as a PR -> 22  (artifactregistry-rule, backend-service-signed-url-key,
                        dataproc-sessiontemplate, ... workflows-workflow)
push            -> 117 (whole testdata/ matrix)
this branch     -> 1   (bucket)

This PR is its own smoke test. The second commit edits testdata/bucket-update.pkl and nothing else under testdata/, so the companion→base rule is what puts bucket — and only bucket — on the matrix. If the CI run below shows one conformance job named bucket, the mechanism works end to end.

That second commit is also a real fixture fix: bucket-update.pkl carried the comment "This is the mutable field being tested (changed from false to true)" next to enabled = false, the same value as bucket.pkl. The update phase only ever changed a label; the versioning path it claimed to cover was never exercised. versioning is a direct field in bucket_body_builder.go, so the PATCH already carries it — the fixture just never asked for a different value.

ci.yml parses; reuse lint clean (588/588).

Notes for review

  • The fetch-depth ternary trap. ${{ event_name == 'pull_request' && 0 || 1 }} evaluates to 1 on a pull request, because 0 is falsy in a GitHub expression and || then takes the right branch. Hardcoded 0 with a comment saying why.
  • The count gate is load-bearing. An empty matrix vector is a hard error in Actions, not a skip, so if: needs.discover-tests.outputs.count != '0' on conformance-tests and pre-cleanup is what makes "this PR touched no fixture" green instead of red.
  • Concurrency is unchanged and still a rough edge. Every conformance workflow shares gcp-conformance-tests because they target one project. GitHub keeps only one run queued per group, so a third concurrent run evicts the waiting one and it reports cancelled, not failed. PR-triggered conformance makes that more likely. Documented in CONTRIBUTING.md; splitting the cheap checks out of the group is a separate change.
  • Fork PRs can't read the GCP secrets. Unchanged by this PR, but it's now reachable on the pull_request path.

naxty added 2 commits August 28, 2026 17:14
Conformance did not run on pull requests at all: discover-tests was gated on
`github.event_name != 'pull_request'`, so a new or edited fixture stayed
unverified until it reached main and the only early check was dispatching
debug-conformance.yml by hand with the case names typed out.

ci.yml now resolves its scope from the event via
scripts/ci/conformance-scope.sh:

  push / workflow_dispatch          -> every case in testdata/ (117)
  pull request                      -> only the cases the PR touched
  pull request + full-conformance   -> every case in testdata/

A PR touching no fixture resolves to zero and skips conformance-tests,
pre-cleanup and post-cleanup entirely, so a plumbing PR stops paying for 117
live GCP lifecycles.

Mapping rules: added/copied/modified/renamed testdata/*.pkl are in scope;
deletions are ignored; an -update / -replace companion maps back to its case;
testdata/config/ is shared Pkl, not a fixture (git's pathspec glob matches '/',
so it has to be filtered out explicitly); a changed companion with no base
fixture fails the run rather than filtering to nothing and passing green.

.github/conformance-pr-skip.txt is optional and absent today - names in it are
subtracted from a PR's scope, and a name in it that matches no fixture fails
the run so the two cannot drift.

conformance-scope_test.sh covers all of this against throwaway git repos with
no cloud credentials, and runs in the test-unit job so it cannot rot.
bucket-update.pkl carried the comment "This is the mutable field being tested
(changed from false to true)" next to `enabled = false` - the same value as
bucket.pkl. The update phase therefore only ever changed a label, and the
versioning path it claimed to cover was never exercised.

`versioning` is a direct field in bucket_body_builder.go, so the PATCH already
carries it; the fixture just never asked for a different value.
@naxty
naxty merged commit 92ccb4c into main Aug 31, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant