Skip to content

Add fail-safe change-aware CI test routing (#533) - #534

Merged
amirbena merged 3 commits into
mainfrom
chore/change-aware-ci-test-routing
Sep 24, 2026
Merged

amirbena merged 3 commits into
mainfrom
chore/change-aware-ci-test-routing

Conversation

@amirbena

@amirbena amirbena commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Fixes #533

What

  • Behavior / contracts: The required test job in validate.yml now runs in one of two tiers. FULL is the default and runs exactly today's command and environment. FAST runs the same discovery without the tests.integration.* tests, and is picked only when every changed path is on the allowlist in scripts/validation/ci_test_route.py. A Route tests (FAST/FULL) step inside test makes a separate blobless clone under $RUNNER_TEMP, so the checkout under test is unchanged. From it, the step extracts the router from the PR base SHA and routes on a three-dot --no-renames diff. The FAST step also runs that base copy of the router. A missing router, a failed fetch, a git or router error, an empty diff, or a non-PR event all give FULL. The step is continue-on-error, and the router writes tier= last, so any failure leaves the tier unset. The integration step runs when the tier is != 'fast'. Pushes to main now run FULL. The step summary shows the tier, the reason, and the first path that forced FULL.
  • Governance / policy: policies/validation-and-clean-exit.md ("Routed CI tests") is the canonical description. README, CONTRIBUTING, and .github/AUTOMATION.md summarize it and link there. The temp-root copy lists are now one TEMP_ROOT_INPUTS constant in tests/integration/packaging/_shared.py, with no behavior change. A guard test fails if that constant, or a literal REPO_ROOT / "…" read under tests/integration/**, overlaps the allowlist. The policy describes this guard as a narrow tripwire, not proof: adding a path to the allowlist still needs evidence gathered by hand, and FULL on every push to main is the safety net.
  • Packaging / portability: None
  • Changelog: generated at release from the two lines below — never edit CHANGELOG.md (see docs/RELEASE.md)
  • Release category: none
  • Release entry:

Validation

  • Relevant validation was run, or the reason it could not be run is stated.
  • Router unit tests and workflow/guard policy tests: 29 tests OK. The two edited integration modules also pass.
  • ci_test_route.py run-fast locally: 4,951 tests OK in ~56s. The FAST IDs are the discovered IDs minus exactly the integration IDs.
  • I replayed the route step with bash -e against a local file:// remote. An allowlist-only diff gave fast. A mixed diff gave full and named shared/… as the path that forced it. A base with no router, a push event, and a bad SHA all gave full.
  • On CI, the route step takes under a second, and test starts together with the parity jobs. This PR's base has no router, so it runs FULL. Still to validate after merge: a .gitignore-only PR gives FAST, a docs/**-only PR and a mixed README.md + shared/ PR give FULL, and the first push to main runs FULL.

Review

  • local-code-review on 7ffa478: REVIEW CLEAN (P0: 0, P1: 0, P2: 2). F1: the separate route job added about 8s to every PR. F2: the policy overstated what the guard covers. Both are fixed in 85b2838.
  • local-code-review re-review on 85b2838: REVIEW CLEAN (P0: 0, P1: 0, P2: 0).

🤖 Generated with Claude Code

amirbena and others added 3 commits September 24, 2026 18:00
validate.yml now routes each PR to FULL (default) or FAST, which omits
only tests.integration.* when every changed path is on a small,
evidence-backed allowlist. The router runs from the PR base SHA, every
push to main runs FULL, and a guard test keeps integration temp-root
copies and REPO_ROOT reads off the allowlist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review F1: the separate `route` job added ~8s of serial latency to every
PR. The `test` job now extracts the base router through a separate
blobless clone under $RUNNER_TEMP (checkout under test unchanged), and
FAST runs that same base-extracted router. The tier is written last so
any failure leaves it unset, which runs FULL.

Review F2: the policy now describes the integration-input guard as a
narrow tripwire and names what it cannot see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amirbena
amirbena merged commit 5d6ac1b into main Sep 24, 2026
10 checks passed
@amirbena
amirbena deleted the chore/change-aware-ci-test-routing branch September 24, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fail-safe change-aware CI test routing

1 participant