Summary
leynos/wildside's scheduled mutation-testing run on 2026-07-16 (run 29494138566) ran for 383.7 minutes overall before failing. The mutants job's "Run mutation testing" step hit its 300-minute timeout-minutes ceiling and was killed:
##[error]The action 'Run mutation testing' has timed out after 300 minutes.
This is not the known embedded-PostgreSQL bootstrap failure (leynos/wildside#420); no pg-embed connection errors appear anywhere in the log, and wildside's caller workflow already pins POSTGRESQL_VERSION==16.10.0 as that issue's fix. The mutants job built and ran cleanly; it simply ran out of time.
Root cause
workflow_scripts/mutation_detect_changes.py's scoped_run_matrix always emits shard=0, shard_count=1 for scheduled (scoped) runs — see the docstring: "Scoped runs stay single-shard: each shard re-pays the baseline build-and-test cost, which is not worth it for a handful of mutants." Only workflow_dispatch (full/unscoped) runs fan out across shard-count (default 6).
That assumption breaks when a scheduled run's change-detection window (window-hours, default 25) catches a large batch of changes. In this run, the single scoped shard was handed 308 mutants across many files (mutation_cargo_command in the log lists dozens of --file arguments). Each mutant's build+test cycle took roughly 200–300 seconds, so completing the batch needed on the order of a day — far beyond the 300-minute step budget (and the job's own 360-minute ceiling). The job was destined to time out before it started; the 383-minute wall-clock is simply "ran until killed", not a hang on any single mutant.
By contrast, the following day's run (29575897246, 2026-07-17) picked up a much smaller change set (78 mutants) and completed in 50 minutes without issue — confirming the mechanism is batch size, not an environmental fault.
Proposed next step
Give scoped (scheduled) runs a size-aware fallback, for example:
- Estimate mutant count (or changed-file count) during
detect and switch to a multi-shard matrix once it exceeds a threshold, reusing the existing shard-count fan-out logic instead of forcing shard_count=1.
- Or cap per-run scope by processing at most N files per scheduled run, carrying the remainder into the next scheduled invocation.
- At minimum, fail fast with a clear diagnostic when the detected mutant/file count is unlikely to fit the timeout budget, rather than silently running until the step is killed.
Reference
Summary
leynos/wildside's scheduled mutation-testing run on 2026-07-16 (run 29494138566) ran for 383.7 minutes overall before failing. The
mutantsjob's "Run mutation testing" step hit its 300-minutetimeout-minutesceiling and was killed:This is not the known embedded-PostgreSQL bootstrap failure (leynos/wildside#420); no pg-embed connection errors appear anywhere in the log, and wildside's caller workflow already pins
POSTGRESQL_VERSION==16.10.0as that issue's fix. The mutants job built and ran cleanly; it simply ran out of time.Root cause
workflow_scripts/mutation_detect_changes.py'sscoped_run_matrixalways emitsshard=0, shard_count=1for scheduled (scoped) runs — see the docstring: "Scoped runs stay single-shard: each shard re-pays the baseline build-and-test cost, which is not worth it for a handful of mutants." Onlyworkflow_dispatch(full/unscoped) runs fan out acrossshard-count(default 6).That assumption breaks when a scheduled run's change-detection window (
window-hours, default 25) catches a large batch of changes. In this run, the single scoped shard was handed 308 mutants across many files (mutation_cargo_commandin the log lists dozens of--filearguments). Each mutant's build+test cycle took roughly 200–300 seconds, so completing the batch needed on the order of a day — far beyond the 300-minute step budget (and the job's own 360-minute ceiling). The job was destined to time out before it started; the 383-minute wall-clock is simply "ran until killed", not a hang on any single mutant.By contrast, the following day's run (29575897246, 2026-07-17) picked up a much smaller change set (78 mutants) and completed in 50 minutes without issue — confirming the mechanism is batch size, not an environmental fault.
Proposed next step
Give scoped (scheduled) runs a size-aware fallback, for example:
detectand switch to a multi-shard matrix once it exceeds a threshold, reusing the existingshard-countfan-out logic instead of forcingshard_count=1.Reference
scoped_run_matrix/full_run_matrix:workflow_scripts/mutation_detect_changes.pyleynos/wildside.github/workflows/mutation-testing.yml, usingleynos/shared-actions/.github/workflows/mutation-cargo.yml