diff --git a/skills/mutation-testing-rollout/SKILL.md b/skills/mutation-testing-rollout/SKILL.md new file mode 100644 index 0000000..db19467 --- /dev/null +++ b/skills/mutation-testing-rollout/SKILL.md @@ -0,0 +1,181 @@ +--- +description: Roll out scheduled, informational mutation testing across an estate of repositories, triage the results into issues and killing tests, and keep the callers documented and drift-free. +--- + +# Mutation-testing estate rollout and triage + +This skill captures the process used to roll mutation testing out across +github.com/leynos/\* — approximately forty Rust and Python repositories — +and to operate it afterwards: harvesting survivors into issues, fixing +broken baselines, writing killing tests, and documenting the machinery. +Apply it when adopting mutation testing in a new repository, when triaging +the output of scheduled runs, or when running an estate-wide sweep. + +## Architecture + +One pair of reusable workflows lives in `leynos/shared-actions`: + +- `mutation-cargo.yml` (Rust, [cargo-mutants](https://mutants.rs/)) and +- `mutation-mutmut.yml` (Python, [mutmut](https://mutmut.readthedocs.io/)). + +Each consumer repository carries only a thin caller, +`.github/workflows/mutation-testing.yml`, pinned to a shared-actions commit +SHA that Dependabot owns. The runs are **informational only**: they never +gate pull requests. A daily cron fires a change-scoped run (mutating only +files changed within a detection window, so quiet days are cheap no-ops); +manual dispatch runs the full mutation set, fanned out across shards. +Survivors surface in the job summary and in downloadable +`mutation-report-*` artefacts. + +The mutation run is a measurement of the test suite, not of the code. A +surviving mutant is a test gap; the goal is a falling survivor count, not +zero. + +## Per-repository adoption recipe + +1. **Verify eligibility.** The repository needs a healthy test suite that + passes in CI. No tests means every mutant survives as noise — defer + until tests exist. Python repositories must be uv-managed (a hard + requirement of the mutmut workflow). +2. **Verify the baseline under the mutation runner's conditions, not + CI's.** This is the single most common failure. Hazards found in + practice: + - Tests that pass under nextest's process isolation but fail under + plain `cargo test` (shared global state, log capture, a claimed + global tracing dispatcher). Either fix the test or pass + `--test-tool=nextest`. + - CI running `--all-targets` never runs doctests, but cargo-mutants + does. Broken doctests (including in vendored code) break the + baseline. Fix them, mark them `ignore`, or set `doctest = false`. + - mutmut copies sources into a `mutants/` sandbox that omits + `.github/`; any test reading workflow files needs a + `pytest.mark.skipif(not WORKFLOW_PATH.exists(), ...)` guard (or put + `.github/` in `also_copy`, or exclude the test from selection). + - Hygiene tests that walk the file system see the workflow-source + checkout; enumerate `git ls-files` instead. + - Stateful test dependencies (embedded PostgreSQL) break on re-run; + pin credentials/versions via the workflow's `setup-commands` input. +3. **Write the caller.** Copy a proven caller (wireframe for Rust, + cmd-mox for Python) and adapt: + - `paths`: change-detection globs matching the repository's source + layout. + - `exclude-globs`: example code, test scaffolding, fixture crates, and + `cfg(test)`-only companion files — anything whose survivors are + noise. Note that a module-root glob (`src/foo.rs`) does not match the + directory's submodules; add `src/foo/**` too. + - `extra-args`: match the CI baseline exactly (`--all-features`, + `--test-workspace=true` for workspaces). A mismatch reports + feature-gated code as untested. For workspaces where bare feature + names fail to resolve against the shard-scoped baseline, ensure every + mutated member defines the baseline feature set (forwarding features + if necessary). + - `setup-commands`: install anything `.cargo/config.toml` assumes + (clang/lld/mold), pin environmental state. + - `shard-count`: size from the first full run; a shard that brushes the + 360-minute job ceiling needs more shards. + - For mutmut: configure `[tool.mutmut]` in `pyproject.toml` + (`source_paths`, test selection, `do_not_mutate` for + subprocess-executed or generated code, `also_copy` for fixtures the + tests need). mutmut hard-codes importable layouts (`./`, `src/`, + `source/`); a `pkg/` layout needs a committed `src -> pkg` symlink + and `module-prefix-strip`. +4. **Pin and stagger.** Pin the `uses:` line to a full 40-character + shared-actions commit SHA. Give each repository a distinct cron slot; + mutation runs are heavy and synchronized schedules concentrate load. +5. **Add a contract test.** A PyYAML-based test + (`tests/workflow_contracts/` or `tests/test_workflow_contract.py`) + pins the caller's shape so drift fails the pull request: the `uses:` + path and that its ref is a 40-hex commit SHA (match the *shape* with a + regex — never hard-code the SHA value, or every Dependabot bump becomes + a manual chore), least-privilege permissions (`contents: read`, + `id-token: write`; empty workflow default), concurrency that queues per + ref without cancelling, the schedule-plus-plain-dispatch triggers, and + the `with:` block contents. +6. **Dispatch a full run** and calibrate: verify the summary populates, + record survivor counts and runtime, and adjust shards/timeouts. +7. **Document.** Add a "Mutation-testing workflow contract tests" section + to `docs/developers-guide.md`: why the workflow exists (informational + only), the two run modes, the rationale for each `with:` input, the + SHA-pin policy, the exact aspects the contract test validates, and the + local run command (a `make test-workflow-contracts` target where the + repository has a Makefile). Create the guide (and file an issue for a + fuller one) where it is missing. + +## Triage discipline + +Harvest survivors from `mutants.out/outcomes.json` (cargo; +`scenario.Mutant.file`, `span.start.line`, `name`) or the mutmut summary. +File **one issue per coherent area** (module or concern), not per mutant, +with counts, representative survivors (`file:line` plus the mutation), and +a proposed next step. Then work the issues as draft PRs: + +- **Real test gap** — write a killing test. Prefer exact assertions; + "equivalent" verdicts frequently fall to a sharper test. +- **Equivalent mutant** — suppress with justification: + `#[cfg_attr(test, mutants::skip)]` (Rust, via the `mutants` crate) or + `# pragma: no mutate` (Python). Keep skips rare and justified; prefer a + killing test wherever the mutation is observable. Beware two silent + no-op traps: mutmut honours the pragma only on single logical lines + (trailing pragmas on continuation lines of multi-line statements are + ignored — restructure to a single line), and cargo-mutants' skip + attribute does not cover plain `const` items or binary-expression + initializers (only functions, impls, and certain expression kinds) — a + skip placed there changes nothing and the mutant resurfaces next run. + Verify a suppression actually suppressed by re-running scoped + (`cargo mutants --file ` / `mutmut run `). +- **Dead code** — delete it; this is one of mutation testing's best + yields. +- **Untestable boundary** — document and move on. + +Link every kill site to its tracking issue (issue refs in PR titles and +doc comments at the test), so the provenance survives the merge. + +Red-green verification traps: stale `.pyc` bytecode can fake a red-green +cycle (`PYTHONDONTWRITEBYTECODE=1`); an ambient environment variable can +turn a gate into a no-op — check what the gate actually ran. + +## Operating the estate: run sweeps + +Periodically sweep all repositories' runs (`gh api +repos///actions/workflows`, then the runs endpoint). Do NOT +classify by wall-clock duration — runner-queue wait routinely inflates a +15-second no-op to an hour. The reliable signals are: +`mutation_detect_has_changes` in the detect step's log (the mutmut +workflow runs its gate *inside* the single mutants job, so a no-op still +reports a green job), and the `mutants` job's conclusion (the cargo +workflow skips it visibly). Remember success does not mean clean — a real +run's survivors live in the job summary — and conversely a "failure" can +mask a fully completed run (a teardown-step regression once failed +otherwise-green runs estate-wide). For failures, read +`gh run view --log-failed`, identify the root cause, and file one +issue per distinct cause (not per run); recurring identical quick failures +are one issue listing occurrences. Deduplicate against existing issues +first — comment on a persisting area rather than re-filing it (and when +concurrent sweep agents race, reconcile their duplicate filings onto one +canonical issue). If the cause lies in the reusable workflow, file it on +the shared-actions repository. + +Be aware the change-detection gate suppresses scheduled coverage on quiet +repositories: a repo can report weeks of green runs while executing zero +mutants, leaving a broken baseline undetected. After merging a +baseline fix, trigger a full `workflow_dispatch` to prove it end-to-end +and produce a survivor dataset — do not wait for the schedule. + +## Estate lessons (hard-won) + +- **Unpinned linters break baselines estate-wide.** `uv tool install ty`, + unpinned ruff preview rules, and rolling-release dylint each turned CI + red across multiple repositories at once. Pin linter versions. +- **Stale local checkouts lie.** Before concluding a file is absent, + verify against the remote default branch (`gh api + repos///contents/...`), and branch from a freshly fetched + `origin/`. +- **Companion and fixture crates produce false survivors.** Where a + crate's coverage lives in a sibling, scope the run or record that its + survivor table is advisory. +- **Keep a living rollout ledger** (per-repository status, hazards, + issue/PR numbers). The tracker outlives any one session and is the only + reliable memory across an estate this size. +- Spelling gates (typos, en-GB Oxford `-ize`) apply to generated prose + too; expect `summarising`/`serialises` to be rejected in favour of + `summarizing`/`serializes`.