ci: a corpus-drift job checks the vendored corpus against its committed baseline (#1034)

[niko86]

Summary

A validator change that moved findings over the vendored python-ags4 corpus merged green, because nothing in this repo ran corpus-qa's baseline --check. The only runner was the dev satellite's compliance.yml, against its own older copy of the crate and its own baseline. This PR adds a corpus-drift job to ci.yml. A PR that moves findings now has to re-freeze pyags4-vendor.json in the same PR.

What changed

• .github/workflows/ci.yml
  • A new corpus filter in the changes job, published as changes.outputs.corpus. It covers:
    • the validator and its path deps (parse, reference, types; cargo tree -p laterite-ags4-validator names no other laterite crate);
    • laterite-ags4-hostopts, which parses the --dict-version the baseline is captured under;
    • the corpus-qa crate, which holds the baseline;
    • the vendored fixtures;
    • rust-packages/Cargo.lock;
    • the new script;
    • ci.yml and build-accel.
  • A new corpus-drift job with the same !cancelled() / filter-failed-means-run if: as rust/test.
    • It builds only laterite-ags4-corpus-qa (debug, sccache via build-accel).
    • It runs crawl --root <fixtures> --all … -q, then baseline --check <baseline> --show-warnings --show-fyi --no-check-files … --no-input, both with --json.
    • The exit code of baseline --check is captured, so the summary line still prints when the check is red.
• tools/check_corpus_drift.py (new, stdlib) turns the two JSON outputs into the verdict:
  • Every run prints one line: corpus-drift: N files checked (M by content), walk_skipped …, not selected …, copy errors …, only_in_committed …, only_in_current …, changed …. "not selected" counts files under the fixture root that the crawl did not take (today that is PROVENANCE.md).
  • On drift it names each moved fixture by file name and sha prefix, lists the - removed / + added findings, and prints the re-freeze commands (--out in place of --check). It exits 1.
  • If the binary's exit code and its JSON disagree, it exits 2 and trusts neither. An error exit from baseline --check is passed through, and the summary still prints.
• tests/test_check_corpus_drift.py (new) drives report() through clean, drift, added/severity, only-in-either-side, exit/JSON disagreement, an errored check, and walk-skip/copy-error counts.
• tests/test_paths_filter.py
  • The corpus filter must fire on a validator-only, parse, reference, types, corpus-qa, baseline, fixture, lockfile and script path.
  • It must not fire on web/, the docs site, the wiki, laterite-node or forge's src/.
  • A new picomatch cross-check series entry (see below).
• .github/rulesets/main-protection.expected.json lists corpus-drift (with integration_id: 15368, like the other Actions contexts).
• rust-packages/laterite-ags4-corpus-qa/baselines/README.md
  • Removed the false claims ("the per-PR value gate" pointing at compliance.yml, and "what CI runs").
  • It now names the corpus-drift job, its trigger and the skip behaviour, and keeps the local check and re-freeze instructions.
• ags-wiki/tools/laterite-ags4-corpus-qa.md said this baseline was --checked per-PR in the satellite's compliance.yml. That was false for this tree. It now names the job and the script.
• Changelog entry under unreleased.added, with CHANGELOG.md regenerated.

Acceptance criteria

• A validator-only PR runs the job; a web/-only PR skips it, and the skip satisfies the required check.
  • test_corpus_fires_on_what_moves_findings / test_corpus_skips_what_cannot pin the filter.
  • The job uses the same job-level if: shape as rust and test. A job skipped by if: reports success to a required check.
  • This PR touches ci.yml, so corpus-drift runs on this PR itself.
• Proven red / green. See the fail-first evidence below.
• The summary line prints on every run. It prints on the clean run, the drift run and the errored run (test_an_errored_check_still_prints_the_summary, test_a_verdict_that_disagrees_with_the_exit_code_is_refused).
• test_every_executed_tool_fires_its_job and the rest of pytest tests pass. The buildless subset as CI runs it: 899 passed, 26 skipped.
• The check name is in main-protection.expected.json. See the maintainer action below.
• The baseline README no longer claims CI that isn't true. It names the job and its trigger, and keeps the re-freeze instructions.

Fail-first evidence

• The job's own run: block, run locally, extracted from ci.yml with RUNNER_TEMP pointed at a scratch dir:
  • Committed baseline: exit 0.

    no drift: 83 contents match the committed baseline.
    corpus-drift: 83 files checked (83 by content), walk_skipped 0, not selected 1, copy errors 0, only_in_committed 0, only_in_current 0, changed 0
    

  • One finding deleted from the baseline (the AGS Format Rule 8 LOCA L76 f1 entry of 4.1-rule8-1.ags, restored with git checkout afterwards): exit 1.

    Finding drift against the committed baseline:
    
      4.1-rule8-1.ags (0d994a046dca): findings -> findings
        + AGS Format Rule 8 LOCA L76 f1
    
    If this change is intended, re-freeze the baseline in THIS PR ...
      cargo run ... crawl --root rust-packages/laterite-ags4-forge/vendor/pyags4-tests --all --corpus-dir "$QA" --no-input -q
      cargo run ... baseline --out rust-packages/laterite-ags4-corpus-qa/baselines/pyags4-vendor.json --show-warnings --show-fyi --no-check-files --corpus-dir "$QA" --no-input
    ...
    corpus-drift: 83 files checked (83 by content), walk_skipped 0, not selected 1, copy errors 0, only_in_committed 0, only_in_current 0, changed 1
    

• tests/test_check_corpus_drift.py was run before the tool existed: all 9 cases errored on the missing module.
• The new test_paths_filter.py cases were run against ci.yml without the corpus filter: 15 failed (14 for the missing filter, 1 for the stale series count). With the filter, 88 passed.
• Picomatch cross-check. _matches was compared with the real picomatch 4.0.7 (dot: true) over 1868 paths x 62 patterns: zero mismatches. It was falsified first: a planted _matches inversion on .rs paths gave 14136 mismatches.

Maintainer action at merge: the live ruleset

Add the status check corpus-drift (source: GitHub Actions) to the live main protection ruleset (19425032). Until then it gates nothing.

How the nightly ruleset-drift behaves while the fixture is ahead of the live ruleset.

• It normalises both sides through normalize.jq and diffs them. The only difference is the corpus-drift entry (checked locally against the live ruleset with gh api).
• So the nightly ruleset-drift job fails, and the nightly tracking issue picks that up. This is expected until the ruleset is updated.
• Its ::error:: text suggests regenerating the snapshot from the live ruleset. Don't follow that here: it would delete the new entry. Add the check to the live ruleset instead.
• If the UI records the context without integration_id: 15368 (that is, without choosing GitHub Actions as the source), the nightly diff will show exactly that field. Either re-add it with the source set, or regenerate the snapshot once the check is live.

Gates run

• tools/check_changelog.py --base origin/main: OK (no shipped surface).
• ruff check .: clean.
• ruff format --check .: clean.
• ty check: clean.
• pytest tests -m "not needs_env" -o addopts="" -p no:benchmark: 899 passed, 26 skipped.
• check_doc_refs.py --check: OK.
• check_issue_refs.py: OK.
• gen_changelog.py --check: OK.
• gen_crate_graph.py --check: OK.
• gen_wiki_tables.py --check: OK.
• Wiki reindex.py --check: OK.
• Wiki lint.py --since origin/main: clean.
• actionlint: no new findings (one pre-existing SC2046 in another job).
• No Rust source changed, so no cargo fmt/clippy/test run was needed beyond building corpus-qa.

Closes #1034

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!