Add fail-safe change-aware CI test routing (#533) - #534
Merged
Merged
Conversation
validate.yml now routes each PR to FULL (default) or FAST, which omits only tests.integration.* when every changed path is on a small, evidence-backed allowlist. The router runs from the PR base SHA, every push to main runs FULL, and a guard test keeps integration temp-root copies and REPO_ROOT reads off the allowlist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review F1: the separate `route` job added ~8s of serial latency to every PR. The `test` job now extracts the base router through a separate blobless clone under $RUNNER_TEMP (checkout under test unchanged), and FAST runs that same base-extracted router. The tier is written last so any failure leaves it unset, which runs FULL. Review F2: the policy now describes the integration-input guard as a narrow tripwire and names what it cannot see. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #533
What
testjob invalidate.ymlnow runs in one of two tiers. FULL is the default and runs exactly today's command and environment. FAST runs the same discovery without thetests.integration.*tests, and is picked only when every changed path is on the allowlist inscripts/validation/ci_test_route.py. ARoute tests (FAST/FULL)step insidetestmakes a separate blobless clone under$RUNNER_TEMP, so the checkout under test is unchanged. From it, the step extracts the router from the PR base SHA and routes on a three-dot--no-renamesdiff. The FAST step also runs that base copy of the router. A missing router, a failed fetch, a git or router error, an empty diff, or a non-PR event all give FULL. The step iscontinue-on-error, and the router writestier=last, so any failure leaves the tier unset. The integration step runs when the tier is!= 'fast'. Pushes tomainnow run FULL. The step summary shows the tier, the reason, and the first path that forced FULL.policies/validation-and-clean-exit.md("Routed CI tests") is the canonical description. README, CONTRIBUTING, and.github/AUTOMATION.mdsummarize it and link there. The temp-root copy lists are now oneTEMP_ROOT_INPUTSconstant intests/integration/packaging/_shared.py, with no behavior change. A guard test fails if that constant, or a literalREPO_ROOT / "…"read undertests/integration/**, overlaps the allowlist. The policy describes this guard as a narrow tripwire, not proof: adding a path to the allowlist still needs evidence gathered by hand, and FULL on every push tomainis the safety net.CHANGELOG.md(seedocs/RELEASE.md)Validation
ci_test_route.py run-fastlocally: 4,951 tests OK in ~56s. The FAST IDs are the discovered IDs minus exactly the integration IDs.bash -eagainst a localfile://remote. An allowlist-only diff gavefast. A mixed diff gavefulland namedshared/…as the path that forced it. A base with no router, a push event, and a bad SHA all gavefull.teststarts together with the parity jobs. This PR's base has no router, so it runs FULL. Still to validate after merge: a.gitignore-only PR gives FAST, adocs/**-only PR and a mixedREADME.md+shared/PR give FULL, and the first push tomainruns FULL.Review
local-code-reviewon7ffa478: REVIEW CLEAN (P0: 0, P1: 0, P2: 2). F1: the separateroutejob added about 8s to every PR. F2: the policy overstated what the guard covers. Both are fixed in85b2838.local-code-reviewre-review on85b2838: REVIEW CLEAN (P0: 0, P1: 0, P2: 0).🤖 Generated with Claude Code