Skip to content

test: cover the exit-code wiring of the corpus checks - #2

Merged
yakisoba0728 merged 2 commits into
mainfrom
test/corpus-check-main-wiring
Aug 18, 2026
Merged

test: cover the exit-code wiring of the corpus checks#2
yakisoba0728 merged 2 commits into
mainfrom
test/corpus-check-main-wiring

Conversation

@yakisoba0728

Copy link
Copy Markdown
Owner

The gap was not where it looked

The premise going in was that validate_corpus.py, check_decode_errors_corpus.py and check_metrics_baseline.py print SKIP and exit 0 without a corpus, the trap CONTRIBUTING.md warns about. Measured: all three exit 2 with a named reason. They already follow the doctrine. (The warning does apply to check_export_baseline.py and check_corpus_baseline.py, which skip when their single pinned replay is absent.)

The real gap is that the layer CI and developers actually trust — main() reading a verdict and choosing an exit code — had no tests at all. The helpers underneath (problems(), dead_counters(), reconcile(), invariants(), compare()) are well covered. The wiring between helper and exit code was not, and this repo has been burned there twice: a counter that vanished while the run printed WARNING and exited 0, and a run that decoded nothing while printing OK.

What this adds

22 tests across the three tools/tests/ files, following the existing convention for faking an executable. Each mutation reproduces a historical failure:

mutation what happens without the test
disable the invariant gate in check_metrics_baseline a run with rounds=0 and score={} is pinned into baseline.json, exit 0 — the exact shape of the 13.02 RoundResults regression
disable dead_counters in check_decode_errors_corpus prints "Decode errors: 0 ... over 0 decoded rows", exit 0 — the bug its own docstring cites
ignore problems() in validate_corpus prints "OK: 0/1 replays validated", exit 0

check_metrics_baseline's extract() also had no coverage of the nested valplay metrics shape it reads — a cross-repo seam that moves without warning. Five tests pin it.

Verified against the real 215-replay corpus and a locally built vrfkit before writing any of this, so the scripts' behaviour is measured rather than assumed.

Also: a pre-existing CI failure

check_docs.py --fast was already failing on main — three lines claim 117 ascii-checked Rust files where check_ascii.py reports 118. The docs step runs before the tools test step in python-checks, so without this fix the new tests would never have run in CI.

Not built, and why

  • A miniature corpus. The checksum_export fixture works because it is a structured JSON manifest, not raw replay bytes. These scripts feed real .vrf bytes to the real binary. crates/vrf-container/tests/corpus.rs already has a hand-built fixture and scopes itself to container smoke-testing — it never touches the replication payload inside a content block, and hand-authoring that is a project of its own. validate_corpus.py's claim is also a distribution claim across the corpus, which one hand-made replay cannot support.
  • A workflow_dispatch job with an out-of-band corpus. Cross-repo checkout needs a PAT, LFS bandwidth bills the other account, and above all a manual job gates nothing on an ordinary PR.
  • A "make the SKIP visible" job. There is no silent SKIP to surface, per the measurement above.

Verification

python -W error -m unittest discover -s tools/tests -p "test_*.py"   503 passed (was 481)
python tools/check_docs.py                                           exit 0, rust 563 / tools 503

No workflow change: python-checks already discovers tools/tests.

Handoff

metrics_builds.json shows drift on build 13.01 (kast_rounds 117→116, trade_kills 35→33), consistent with valplay's recent self-damage attribution fix. CONTRIBUTING.md requires explaining each changed line before --update, and that is a domain judgement — left alone.

check_docs.py --fast, which CI runs before the tools suite, was already
failing on main: three lines said 117 ascii-checked Rust files where
check_ascii.py reports 118. The docs job fails before the test job runs,
so this had to go first.
The three corpus scripts fail loudly without a corpus -- exit 2, not a
silent skip -- so the gap was never SKIP-reads-as-pass. It was that the
layer CI and developers actually trust, main() reading a verdict and
choosing an exit code, had no tests at all. Their helpers were well
covered; the wiring between helper and exit code was not, and this repo
has been burned there twice: a counter that vanished while the run
printed WARNING and exited 0, and a run that decoded nothing while
printing OK.

Each mutation reproduces one of those. Disabling the invariant gate in
check_metrics_baseline pins a run with rounds=0 and an empty score into
baseline.json and exits 0 -- the exact shape of the 13.02 regression.
Disabling the dead-counter check prints "Decode errors: 0 ... over 0
decoded rows" and exits 0, which is the bug its own docstring cites.
Ignoring problems() reports "0/1 replays validated" and exits 0.

check_metrics_baseline's extract() also had no coverage of the nested
valplay metrics shape it reads, which is a cross-repo seam that moves
without warning.

No workflow change: python-checks already discovers tools/tests.
@yakisoba0728
yakisoba0728 merged commit 1aafbc7 into main Aug 18, 2026
2 checks passed
@yakisoba0728
yakisoba0728 deleted the test/corpus-check-main-wiring branch August 18, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant