perf(ci): balance the Windows pytest shards by time, not test count (#88)

[KbWen]

perf(ci): balance the Windows pytest shards by time, not test count (#88)

--splits 3 --group N ran with no committed .test_durations, so pytest-split
fell back to an even COUNT split and clustered every subprocess-shelling deploy
test onto one shard. Real job times on PR #417: 21m57s / 3m19s / 4m14s -- the
worst shard carried 74% of all the work while two runners sat idle.

A 897-entry durations file is now committed. Predicted shards 19.6 / 19.5 /
18.2 min against a 19.1-min ideal: worst shard drops from 74% to 34% of total
work (33.3% is perfect). Scaled to the runner that is ~21m57s -> ~10.1 min, a
~2.2x CI wall-clock cut. No test, source, or workflow-behaviour change -- CI
already passes --splits/--group and pytest-split picks up the default path.

The row's own measurement was stale by ~3x and is corrected in place: #88
recorded "measured wall-clock 7:14"; the real numbers are above.

Verified at zero cost rather than by three real CI runs: each group was collected
with CI's exact invocation via --collect-only and its members' measured durations
summed. --splitting-algorithm least_duration was also measured and is better
(19.1 / 19.1 / 19.1, exactly ideal) -- and deliberately rejected, because it
reorders tests across groups and 2.6% is not worth that risk.

The ceiling is named rather than implied: two tests are 25.8% of the whole suite
(test_validator_count_parity_on_framework at 8.0 min,
test_170_underscore_meta_specs_no_status_warn at 6.8 min), and 87 of 897 tests
(9.7%) carry 90% of the runtime. A perfect split can never beat the heaviest
single test, so ~10 min/shard is the practical floor until those two are
addressed -- a separate unit with real fidelity risk. Two paths stay closed:
pytest-xdist was measured slower here, and deselecting slow in CI is explicitly
rejected in pytest.ini (subprocess fidelity is the design).

Honest provenance: the durations were generated on a Windows workstation
(57.3 min total), not on windows-latest (~29.5 min total). pytest-split consumes
RELATIVE weights and the dominant cost -- subprocess spawn -- is the same class on
both, but absolute times differ ~1.9x. The real verification is this PR's own job
times against the #417 baseline. Unknown test ids added later degrade to
count-splitting, so the file goes stale gracefully; no guard detects staleness and
none is claimed.

Also corrects two comments that asserted the file does not exist
(validate.yml:332, requirements-ci.txt:8).

Tests:

• validate.ps1 exit 0 - pass=118 warn=3 fail=0 skip=2 (unqualified pass)
• validate.sh exit 0 - pass=118 warn=3 fail=0 skip=2 (unqualified pass; twin WARN
  sets identical -- the run finished inside the 60-min lock window)
• full CI-equivalent suite, no -m filter: 896 passed, 1 skipped, exit 0 (47:19)
• test_ci_hardening.py (reads validate.yml) -> 13 passed
• scope check: .test_durations is not referenced by deploy.sh, so it is not
  shipped downstream

This improves CI wall-clock only. A local single-process run is unaffected by
shard configuration and no claim is made otherwise.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

🤖 Generated with Claude Code