From 229f061f8a3aff5ac3364a591c3d6199747375dd Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 7 Jul 2026 11:56:35 +0200 Subject: [PATCH 01/15] Add scheduled mutation testing to generated repos Add a mutation-testing caller workflow to the template so newly generated Rust projects run scheduled mutation testing by default. The workflow is a thin caller of the shared mutation-cargo reusable workflow in leynos/shared-actions, pinned to a full commit SHA in line with the template's action-pinning convention. The workflow runs daily at 09:15 UTC with a comment advising maintainers to stagger the slot per repository, and supports manual dispatch. It defaults the token to no scopes, with the job opting in to contents: read and id-token: write. The extra-args input mirrors the CI test baseline (--all-features); workspace inputs are omitted as the generated project is a single crate. The file is plain YAML rather than Jinja-templated, matching ci.yml and act-validation.yml, as it needs no template variables. The generated repository-layout document is updated to list the new workflow. --- .../.github/workflows/mutation-testing.yml | 37 +++++++++++++++++++ template/docs/repository-layout.md.jinja | 3 ++ 2 files changed, 40 insertions(+) create mode 100644 template/.github/workflows/mutation-testing.yml diff --git a/template/.github/workflows/mutation-testing.yml b/template/.github/workflows/mutation-testing.yml new file mode 100644 index 0000000..dcc51a6 --- /dev/null +++ b/template/.github/workflows/mutation-testing.yml @@ -0,0 +1,37 @@ +name: Mutation testing + +# Thin caller of the shared mutation-testing reusable workflow; caller +# guide: leynos/shared-actions docs/mutation-cargo-workflow.md. +# Scheduled runs mutate only files changed within the detection window; +# manual dispatch runs mutate everything, fanned out across shards. To +# test a branch, select it in the Actions "Run workflow" control. + +'on': + schedule: + # Daily, 09:15 UTC. Stagger this slot when adopting the workflow in a + # new repository: the estate already occupies 03:05-09:05 UTC, so pick + # an unclaimed slot to avoid concurrent runs across repositories. + - cron: "15 9 * * *" + workflow_dispatch: + +# Default the token to no scopes; the job opts in explicitly. +permissions: {} + +# Serialize runs per ref: a dispatch during a scheduled run (or vice +# versa) queues rather than racing. Runs are informational, so queued +# runs wait instead of cancelling. +concurrency: + group: mutation-testing-${{ github.ref }} + cancel-in-progress: false + +jobs: + mutation: + permissions: + contents: read + id-token: write # OIDC workflow-source resolution + uses: leynos/shared-actions/.github/workflows/mutation-cargo.yml@2b09d10192627fd6e1034e7c12625dd266b45503 + with: + # Match the CI baseline (`make test` runs --all-targets + # --all-features) so the test command exercised against mutants + # mirrors what gates pull requests. + extra-args: "--all-features" diff --git a/template/docs/repository-layout.md.jinja b/template/docs/repository-layout.md.jinja index 6ecb2e0..2cc430b 100644 --- a/template/docs/repository-layout.md.jinja +++ b/template/docs/repository-layout.md.jinja @@ -18,6 +18,7 @@ compact and omits build output such as `target/`. │ └── workflows/ │ ├── act-validation.yml │ ├── ci.yml +│ ├── mutation-testing.yml {% if flavour == 'app' %} │ └── release.yml {% endif %} @@ -55,6 +56,8 @@ compact and omits build output such as `target/`. validation through `act` separately from main CI. - `.github/workflows/ci.yml`: Runs the generated project's continuous integration checks. +- `.github/workflows/mutation-testing.yml`: Runs scheduled mutation testing + through the shared `mutation-cargo` reusable workflow. {% if flavour == 'app' %} - `.github/workflows/release.yml`: Builds and publishes binary release artefacts for the application flavour. From fd140299b868d6c0195464812e29a3ec5f267ff4 Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 7 Jul 2026 12:31:02 +0200 Subject: [PATCH 02/15] Install mold in mutation testing setup commands The generated .cargo/config.toml links Linux dev targets with clang and -fuse-ld=mold, and the generated ci.yml installs mold before any cargo command, but the mutation workflow's mutants jobs had no way to install it, so the cargo-mutants baseline would fail to link on a bare ubuntu-latest runner. Bump the shared workflow pin to 0c8bd605f8966a004decb0b54f717a745f51bc57, which introduces the setup-commands input, and use it to install clang, lld, and mold exactly as the generated ci.yml does. The install is unconditional, matching ci.yml's own convention: CI always runs on Linux runners, and ci.yml installs the packages regardless of the dev_target chosen at generation time, so the workflow stays a plain copied file rather than becoming Jinja-templated. --- template/.github/workflows/mutation-testing.yml | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/template/.github/workflows/mutation-testing.yml b/template/.github/workflows/mutation-testing.yml index dcc51a6..bc49243 100644 --- a/template/.github/workflows/mutation-testing.yml +++ b/template/.github/workflows/mutation-testing.yml @@ -29,9 +29,19 @@ jobs: permissions: contents: read id-token: write # OIDC workflow-source resolution - uses: leynos/shared-actions/.github/workflows/mutation-cargo.yml@2b09d10192627fd6e1034e7c12625dd266b45503 + uses: leynos/shared-actions/.github/workflows/mutation-cargo.yml@0c8bd605f8966a004decb0b54f717a745f51bc57 with: # Match the CI baseline (`make test` runs --all-targets # --all-features) so the test command exercised against mutants # mirrors what gates pull requests. extra-args: "--all-features" + # .cargo/config.toml links Linux dev targets with clang and + # -fuse-ld=mold, so the mutants jobs need the same packages that + # ci.yml installs before any cargo command runs on a bare + # ubuntu-latest runner. The reusable workflow executes these under + # bash with `set -euo pipefail`. + setup-commands: | + export DEBIAN_FRONTEND=noninteractive + # mold accelerates dev builds; coverage uses lld for llvm-tools compatibility. + sudo apt-get update \ + && sudo apt-get install --yes --no-install-recommends clang lld mold From 273a8ea1faf65999aa82885d1c147f8f67473742 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 16 Jul 2026 23:43:40 +0200 Subject: [PATCH 03/15] Assert generated mutation-testing workflow tooling contract The mutation-testing.yml caller shipped without any contract coverage, so a regression to the reusable-workflow reference, the setup-command package list, or the mutation job's permission scope could slip through review. Extend the rendered tooling-contract suite to parse mutation-testing.yml and assert that: root permissions grant no scopes; the mutation job calls the shared mutation-cargo reusable workflow; the job permissions stay scoped to contents: read and id-token: write; extra-args mirrors the CI --all-features baseline; and the setup-commands install clang, lld, and mold via apt-get. The pinned reusable-workflow SHA is deliberately not asserted, as Dependabot owns and bumps that reference independently of this contract. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../tooling_contracts/orchestration.py | 9 ++- tests/helpers/tooling_contracts/workflows.py | 56 +++++++++++++++++++ tests/test_template/test_tooling_contracts.py | 4 ++ 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/tests/helpers/tooling_contracts/orchestration.py b/tests/helpers/tooling_contracts/orchestration.py index 10b20c4..7ddbf5a 100644 --- a/tests/helpers/tooling_contracts/orchestration.py +++ b/tests/helpers/tooling_contracts/orchestration.py @@ -15,6 +15,7 @@ from tests.helpers.tooling_contracts.workflows import ( _assert_audit_workflow_contracts, _assert_ci_workflow_contracts, + _assert_mutation_workflow_contracts, _assert_release_workflow_contracts, ) @@ -32,6 +33,7 @@ def assert_generated_tooling_contracts( ci_workflow: str, audit_workflow: str, act_workflow: str, + mutation_workflow: str, docs_contents: str, repository_layout: str, readme: str, @@ -64,6 +66,8 @@ def assert_generated_tooling_contracts( Rendered generated scheduled dependency audit workflow text. act_workflow Rendered generated act-validation workflow text. + mutation_workflow + Rendered generated mutation-testing workflow text. docs_contents Rendered ``docs/contents.md`` text. repository_layout @@ -99,7 +103,10 @@ def assert_generated_tooling_contracts( parsed_ci_workflow, ci_workflow, act_workflow, test_stub ) _assert_audit_workflow_contracts(audit_workflow) - assert_documentation_navigation_contracts(docs_contents, repository_layout, flavour) + _assert_mutation_workflow_contracts(mutation_workflow) + assert_documentation_navigation_contracts( + docs_contents, repository_layout, flavour + ) assert "[Documentation contents](docs/contents.md)" in readme, ( "expected generated README to link to the documentation contents" ) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index cf04d73..fdacdf9 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -442,7 +442,63 @@ def _assert_ci_workflow_contracts( "expected generated test stub to explain when to delete it" ) +def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: + """Assert generated mutation-testing workflow contracts. + Parameters + ---------- + mutation_workflow + Rendered generated-project mutation-testing workflow text. + + Returns + ------- + None + The helper returns ``None`` when the mutation workflow contract passes. + + Raises + ------ + AssertionError + Raised when the reusable workflow reference, root or job permissions, + ``extra-args`` baseline, or setup-command package installation diverge + from the expected contract. + pytest.fail.Exception + Raised by YAML parsing helpers when the workflow cannot be parsed as + the expected mapping structure. + + Notes + ----- + The pinned reusable-workflow SHA is intentionally not asserted; Dependabot + owns that reference and bumps it independently of this contract. + """ + parsed = parse_yaml_mapping(mutation_workflow, "mutation-testing workflow") + assert parsed.get("permissions") == {}, ( + "expected mutation-testing workflow root permissions to grant no scopes" + ) + jobs = require_mapping(parsed, "jobs", "mutation-testing workflow") + mutation = require_mapping(jobs, "mutation", "mutation-testing workflow jobs") + assert str(mutation.get("uses", "")).startswith( + "leynos/shared-actions/.github/workflows/mutation-cargo.yml@" + ), "expected mutation job to call the shared mutation-cargo reusable workflow" + permissions = require_mapping(mutation, "permissions", "mutation job") + assert permissions == _MUTATION_JOB_PERMISSIONS, ( + "expected mutation job permissions to stay scoped to " + "contents: read and id-token: write" + ) + inputs = require_mapping(mutation, "with", "mutation job") + assert inputs.get("extra-args") == "--all-features", ( + "expected mutation job to mirror the CI --all-features test baseline" + ) + setup_commands = inputs.get("setup-commands") + assert isinstance(setup_commands, str), ( + "expected mutation job setup-commands to be a string" + ) + assert "apt-get install" in setup_commands, ( + "expected mutation job setup-commands to install packages via apt-get" + ) + for package in _MUTATION_SETUP_PACKAGES: + assert package in setup_commands, ( + f"expected mutation job setup-commands to install {package}" + ) def _assert_release_workflow_contracts(release_workflow: str) -> None: """Assert generated release workflow contracts.""" parsed_release_workflow = parse_yaml_mapping(release_workflow, "release workflow") diff --git a/tests/test_template/test_tooling_contracts.py b/tests/test_template/test_tooling_contracts.py index 25383ed..7bb4066 100644 --- a/tests/test_template/test_tooling_contracts.py +++ b/tests/test_template/test_tooling_contracts.py @@ -59,6 +59,9 @@ def test_generated_tooling_contracts( coverage_main_workflow = read_generated_text( project / ".github/workflows/coverage-main.yml" ) + mutation_workflow = read_generated_text( + project / ".github/workflows/mutation-testing.yml" + ) docs_contents = read_generated_text(project / "docs/contents.md") repository_layout = read_generated_text(project / "docs/repository-layout.md") readme = read_generated_text(project / "README.md") @@ -89,6 +92,7 @@ def test_generated_tooling_contracts( ci_workflow=ci_workflow, audit_workflow=audit_workflow, act_workflow=act_workflow, + mutation_workflow=mutation_workflow, docs_contents=docs_contents, repository_layout=repository_layout, readme=readme, From 24d9a9512750fafe603951017e4675c13d3d39a9 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 16 Jul 2026 23:47:37 +0200 Subject: [PATCH 04/15] Document scheduled mutation-testing workflow in the user guide repository-layout listed the generated mutation-testing.yml workflow, but neither user guide explained what it does or how it affects a generated project, leaving adopters without guidance. Add a "Scheduled Mutation Testing" section to both the template-rendered user guide (template/docs/users-guide.md.jinja) and the parent-repository user guide (docs/users-guide.md). The section explains that generated projects run cargo-mutants via the shared mutation-cargo reusable workflow on a daily schedule plus manual dispatch, what surviving mutants mean, that scheduled runs mutate only recently changed files while dispatch runs cover the whole crate, and that the runs are informational rather than gating pull requests. It also tells adopters to stagger the cron slot and notes the least-privilege token scope. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/users-guide.md | 24 ++++++++++++++++++++++++ template/docs/users-guide.md.jinja | 24 ++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/docs/users-guide.md b/docs/users-guide.md index 0e46957..3ed2173 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -69,3 +69,27 @@ The generated `Makefile` exposes these public targets: Install `clang`, `lld`, `mold`, `python3`, and `cargo-audit` before running the full generated workflow locally on Linux. + +## Scheduled Mutation Testing + +Generated projects include `.github/workflows/mutation-testing.yml`, a scheduled +GitHub Actions workflow that runs mutation testing with `cargo-mutants`. It is a +thin caller of the shared `leynos/shared-actions` `mutation-cargo` reusable +workflow. + +Mutation testing measures test-suite quality. It introduces small changes +(mutants) into the source and confirms the tests fail in response. A surviving +mutant marks a code path the tests do not meaningfully exercise, so promote it +into a new test rather than ignoring it. + +The workflow runs on a daily schedule (09:15 UTC by default) and can also be +started manually from the **Actions** tab with **Run workflow**. Scheduled runs +mutate only files changed within the detection window, so routine runs stay +fast; a manual dispatch mutates the whole crate, fanned out across shards. +Because the runs are scheduled rather than gating pull requests, they surface +coverage gaps without slowing day-to-day CI, and they do not block merges. + +When adopting the workflow in a new repository, stagger the cron slot: pick an +unclaimed daily time to avoid concurrent runs across related repositories. The +`mutation` job runs with a least-privilege token (`contents: read` plus +`id-token: write` for workflow-source resolution). diff --git a/template/docs/users-guide.md.jinja b/template/docs/users-guide.md.jinja index 914e7ac..a493fd2 100644 --- a/template/docs/users-guide.md.jinja +++ b/template/docs/users-guide.md.jinja @@ -44,3 +44,27 @@ The generated `Makefile` exposes these public targets: Install `clang`, `lld`, `mold`, `python3`, and `cargo-audit` before running the full generated workflow locally on Linux. + +## Scheduled Mutation Testing + +Generated projects include `.github/workflows/mutation-testing.yml`, a scheduled +GitHub Actions workflow that runs mutation testing with `cargo-mutants`. It is a +thin caller of the shared `leynos/shared-actions` `mutation-cargo` reusable +workflow. + +Mutation testing measures test-suite quality. It introduces small changes +(mutants) into the source and confirms the tests fail in response. A surviving +mutant marks a code path the tests do not meaningfully exercise, so promote it +into a new test rather than ignoring it. + +The workflow runs on a daily schedule (09:15 UTC by default) and can also be +started manually from the **Actions** tab with **Run workflow**. Scheduled runs +mutate only files changed within the detection window, so routine runs stay +fast; a manual dispatch mutates the whole crate, fanned out across shards. +Because the runs are scheduled rather than gating pull requests, they surface +coverage gaps without slowing day-to-day CI, and they do not block merges. + +When adopting the workflow in a new repository, stagger the cron slot: pick an +unclaimed daily time to avoid concurrent runs across related repositories. The +`mutation` job runs with a least-privilege token (`contents: read` plus +`id-token: write` for workflow-source resolution). From 5e026d81f9ad1a7f1741f28ac91e14182631f235 Mon Sep 17 00:00:00 2001 From: leynos Date: Sat, 18 Jul 2026 21:27:49 +0200 Subject: [PATCH 05/15] Harden mutation-testing workflow contract assertions The mutation-testing contract only checked root/job permissions, the reusable-workflow path prefix, extra-args, and substring package matches, so several workflow regressions could still pass: an un-pinned reusable ref, a dropped schedule or manual dispatch, a flipped cancel-in-progress, or a package name appearing only in a comment. Extend `_assert_mutation_workflow_contracts` to also require: - an `on.schedule` cron trigger and `workflow_dispatch` support; - `concurrency.group` present with `cancel-in-progress: false`, so queued runs wait rather than cancelling; - the mutation job's `uses` ref pinned to a full 40-character commit SHA (format only; the specific SHA is owned by Dependabot and not hard-coded, so the contract survives dependency bumps); - clang, lld, and mold present as complete `apt-get install` arguments, parsed from the extracted install command rather than substring-matched against the whole setup script. Collapse the helper's docstring to a single-line summary matching its sibling `_assert_*` workflow validators. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/workflows.py | 100 ++++++++++++++----- 1 file changed, 74 insertions(+), 26 deletions(-) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index fdacdf9..1f7a520 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -9,6 +9,7 @@ from __future__ import annotations import re +import shlex from typing import Any from tests.helpers.generated_files import ( @@ -442,43 +443,86 @@ def _assert_ci_workflow_contracts( "expected generated test stub to explain when to delete it" ) -def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: - """Assert generated mutation-testing workflow contracts. + +_MUTATION_JOB_PERMISSIONS = {"contents": "read", "id-token": "write"} +_MUTATION_SETUP_PACKAGES = ("clang", "lld", "mold") +_MUTATION_REUSABLE_WORKFLOW = ( + "leynos/shared-actions/.github/workflows/mutation-cargo.yml" +) + + +def _extract_apt_install_packages(setup_commands: str) -> list[str]: + """Return packages passed to the setup-commands ``apt-get install`` call. Parameters ---------- - mutation_workflow - Rendered generated-project mutation-testing workflow text. + setup_commands + Setup-commands shell script from the mutation job ``with`` inputs. Returns ------- - None - The helper returns ``None`` when the mutation workflow contract passes. + list[str] + Non-flag arguments supplied to ``apt-get install``, following any + backslash line continuations, or an empty list when no install command + is present. + """ + lines = setup_commands.splitlines() + marker = "apt-get install" + for index, line in enumerate(lines): + marker_at = line.find(marker) + if marker_at == -1: + continue + segment = line[marker_at + len(marker) :] + cursor = index + while cursor < len(lines) and lines[cursor].rstrip().endswith("\\"): + segment = segment.rstrip().removesuffix("\\") + cursor += 1 + if cursor < len(lines): + segment += " " + lines[cursor] + try: + tokens = shlex.split(segment) + except ValueError: + tokens = segment.split() + return [token for token in tokens if not token.startswith("-")] + return [] - Raises - ------ - AssertionError - Raised when the reusable workflow reference, root or job permissions, - ``extra-args`` baseline, or setup-command package installation diverge - from the expected contract. - pytest.fail.Exception - Raised by YAML parsing helpers when the workflow cannot be parsed as - the expected mapping structure. - Notes - ----- - The pinned reusable-workflow SHA is intentionally not asserted; Dependabot - owns that reference and bumps it independently of this contract. - """ +def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: + """Assert generated mutation-testing workflow contracts.""" parsed = parse_yaml_mapping(mutation_workflow, "mutation-testing workflow") assert parsed.get("permissions") == {}, ( "expected mutation-testing workflow root permissions to grant no scopes" ) + triggers = require_mapping(parsed, "on", "mutation-testing workflow") + schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") + assert any( + isinstance(entry, dict) and entry.get("cron") for entry in schedule + ), "expected mutation-testing workflow to define a scheduled cron trigger" + assert "workflow_dispatch" in triggers, ( + "expected mutation-testing workflow to support manual workflow_dispatch" + ) + concurrency = require_mapping(parsed, "concurrency", "mutation-testing workflow") + assert concurrency.get("group"), ( + "expected mutation-testing workflow to define a concurrency group" + ) + assert concurrency.get("cancel-in-progress") is False, ( + "expected mutation-testing workflow to queue runs " + "(concurrency.cancel-in-progress: false)" + ) jobs = require_mapping(parsed, "jobs", "mutation-testing workflow") mutation = require_mapping(jobs, "mutation", "mutation-testing workflow jobs") - assert str(mutation.get("uses", "")).startswith( - "leynos/shared-actions/.github/workflows/mutation-cargo.yml@" - ), "expected mutation job to call the shared mutation-cargo reusable workflow" + uses = str(mutation.get("uses", "")) + pinned_prefix = f"{_MUTATION_REUSABLE_WORKFLOW}@" + assert uses.startswith(pinned_prefix), ( + "expected mutation job to call the shared mutation-cargo reusable workflow" + ) + # Assert the ref is pinned to a full commit SHA rather than a mutable tag or + # branch; the specific SHA is owned by Dependabot and deliberately not + # hard-coded here, so the contract survives dependency bumps. + ref = uses[len(pinned_prefix) :] + assert re.fullmatch(r"[0-9a-f]{40}", ref), ( + "expected mutation job to pin the reusable workflow to a full commit SHA" + ) permissions = require_mapping(mutation, "permissions", "mutation job") assert permissions == _MUTATION_JOB_PERMISSIONS, ( "expected mutation job permissions to stay scoped to " @@ -492,13 +536,17 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: assert isinstance(setup_commands, str), ( "expected mutation job setup-commands to be a string" ) - assert "apt-get install" in setup_commands, ( + installed_packages = _extract_apt_install_packages(setup_commands) + assert installed_packages, ( "expected mutation job setup-commands to install packages via apt-get" ) for package in _MUTATION_SETUP_PACKAGES: - assert package in setup_commands, ( - f"expected mutation job setup-commands to install {package}" + assert package in installed_packages, ( + "expected mutation job setup-commands apt-get install to include " + f"{package}" ) + + def _assert_release_workflow_contracts(release_workflow: str) -> None: """Assert generated release workflow contracts.""" parsed_release_workflow = parse_yaml_mapping(release_workflow, "release workflow") From 8587477be99594ffa7a7e97f5294d86c9c156f7f Mon Sep 17 00:00:00 2001 From: leynos Date: Sat, 18 Jul 2026 21:28:00 +0200 Subject: [PATCH 06/15] Note the mutation-testing workflow in the developer guides The user guides describe the scheduled mutation-testing workflow, but the developer guides omitted it, leaving contributors without pointers to how it is tested or maintained. Add a note to the parent developer guide (docs/developers-guide.md) recording that the template renders the mutation-testing workflow into generated projects, that its rendered contract is asserted by `_assert_mutation_workflow_contracts`, and that Dependabot owns the pinned reusable-workflow SHA (so the test asserts the pin format, not a hard-coded value). Add a companion note to the generated-project developer guide (template/docs/developers-guide.md.jinja) explaining that the scheduled workflow runs cargo-mutants via the shared reusable workflow, is informational rather than a pull-request gate, has its pinned SHA kept current by Dependabot, and that surviving mutants should be promoted into new tests. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/developers-guide.md | 7 +++++++ template/docs/developers-guide.md.jinja | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index f5db883..123183d 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -79,6 +79,13 @@ The main `.github/workflows/ci.yml` workflow runs ordinary `make test` without `WITH_ACT=1` so the slower container-backed checks run in parallel instead of blocking the main test and coverage path. +The template also renders `.github/workflows/mutation-testing.yml` into +generated projects; its rendered contract is asserted by +`_assert_mutation_workflow_contracts` in +`tests/helpers/tooling_contracts/workflows.py`. Dependabot owns the pinned +reusable-workflow commit SHA, so the contract test asserts the pin format (a +full commit SHA) rather than a hard-coded value. + ## Required Tooling The parent tests expect these tools to be available when validating generated diff --git a/template/docs/developers-guide.md.jinja b/template/docs/developers-guide.md.jinja index bc0fb8b..2056c55 100644 --- a/template/docs/developers-guide.md.jinja +++ b/template/docs/developers-guide.md.jinja @@ -25,6 +25,13 @@ The main `.github/workflows/ci.yml` workflow deliberately does not run `make test WITH_ACT=1`; the separate Act workflow runs those slower container-backed checks in parallel. +A scheduled `.github/workflows/mutation-testing.yml` workflow also runs +`cargo-mutants` via the shared reusable workflow, daily and on manual +dispatch. It is informational and does not gate pull requests. Dependabot +keeps its pinned reusable-workflow SHA current. See the user guide's +"Scheduled Mutation Testing" section for behaviour, and promote surviving +mutants into new tests. + ## Tooling Development builds use Cranelift for debug code generation. On Linux targets, From 836d433c025ea7a8231c6c586c4d86b55a7f6a99 Mon Sep 17 00:00:00 2001 From: leynos Date: Sat, 18 Jul 2026 21:35:28 +0200 Subject: [PATCH 07/15] Stop hard-coding Dependabot-owned action SHAs in contract tests Dependabot bumps the pinned GitHub Actions commit SHAs in the template's workflow files, but it cannot update the contract tests, so hard-coding those SHAs in assertions made the suite drift and fail on every bump. Assert action pins by format instead of by value. Two helpers, `_assert_pinned_to_commit_sha` (for a parsed `uses` field) and `_assert_pinned_action_in_text` (for a whole-workflow substring search), require each action reference to target its expected path pinned to a full 40-character commit SHA, without naming the SHA. Apply them to the shared coverage, setup-rust, setup-python, upload-artifact, and mutation-cargo references, and route the mutation-job pin check through the shared helper. Replace the `_ci_workflow` fixture's real SHA with a 40-zero placeholder. The release workflow's `CROSS_REVISION` pin is left asserted verbatim: it is a `cargo install cross --rev` git revision, not a GitHub Actions `uses:` SHA, and Dependabot does not manage it. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/workflows.py | 29 ++++++++++---------- tests/test_helpers.py | 2 +- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index 1f7a520..172885c 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -27,6 +27,8 @@ _SETUP_RUST_USES_TEXT_RE = re.compile( r"leynos/shared-actions/\.github/actions/setup-rust@[0-9a-f]{40}" ) +_SETUP_PYTHON_USES_TEXT_RE = re.compile(r"actions/setup-python@[0-9a-f]{40}") +_UPLOAD_ARTIFACT_USES_TEXT_RE = re.compile(r"actions/upload-artifact@[0-9a-f]{40}") def _is_pinned_action(uses: str, action_path: str) -> bool: @@ -399,9 +401,10 @@ def _assert_ci_workflow_contracts( assert "Setup Python for audit manifest extraction" in ci_workflow, ( "expected generated CI workflow to install Python for audit metadata parsing" ) - assert "actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405" in ( - ci_workflow - ), "expected generated CI workflow to pin setup-python for audit parsing" + assert _SETUP_PYTHON_USES_TEXT_RE.search(ci_workflow), ( + "expected generated CI workflow to use the setup-python action " + "pinned to a full 40-hex commit SHA" + ) assert "make audit" in ci_workflow, ( "expected generated CI workflow to run the dependency audit gate" ) @@ -511,16 +514,14 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: ) jobs = require_mapping(parsed, "jobs", "mutation-testing workflow") mutation = require_mapping(jobs, "mutation", "mutation-testing workflow jobs") - uses = str(mutation.get("uses", "")) + mutation_uses = str(mutation.get("uses", "")) pinned_prefix = f"{_MUTATION_REUSABLE_WORKFLOW}@" - assert uses.startswith(pinned_prefix), ( + assert mutation_uses.startswith(pinned_prefix), ( "expected mutation job to call the shared mutation-cargo reusable workflow" ) - # Assert the ref is pinned to a full commit SHA rather than a mutable tag or - # branch; the specific SHA is owned by Dependabot and deliberately not - # hard-coded here, so the contract survives dependency bumps. - ref = uses[len(pinned_prefix) :] - assert re.fullmatch(r"[0-9a-f]{40}", ref), ( + # The specific SHA is owned by Dependabot and deliberately not hard-coded, so + # the contract survives dependency bumps while requiring an immutable pin. + assert re.fullmatch(r"[0-9a-f]{40}", mutation_uses[len(pinned_prefix) :]), ( "expected mutation job to pin the reusable workflow to a full commit SHA" ) permissions = require_mapping(mutation, "permissions", "mutation job") @@ -557,10 +558,10 @@ def _assert_release_workflow_contracts(release_workflow: str) -> None: step.get("with", {}).get("persist-credentials") is False for step in release_checkout_steps ), "expected release workflow checkout steps to disable credentials" - assert ( - "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" - in release_workflow - ), "expected app release workflow to use pinned upload-artifact action" + assert _UPLOAD_ARTIFACT_USES_TEXT_RE.search(release_workflow), ( + "expected app release workflow to use the upload-artifact action " + "pinned to a full 40-hex commit SHA" + ) cross_revision = "88f49ff79e777bef6d3564531636ee4d3cc2f8d2" assert f"CROSS_REVISION: {cross_revision}" in release_workflow, ( "expected app release workflow to pin cross to an immutable revision" diff --git a/tests/test_helpers.py b/tests/test_helpers.py index 9e666db..1fa4f73 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -578,6 +578,6 @@ def _ci_workflow(*, persist_credentials: str, coverage_inputs: str) -> str: with: persist-credentials: {persist_credentials} - name: Test and Measure Coverage - uses: leynos/shared-actions/.github/actions/generate-coverage@927edd45ae77be4251a8a18ca9eb5613a2e32cbd + uses: leynos/shared-actions/.github/actions/generate-coverage@0000000000000000000000000000000000000000 with: {coverage_inputs}""" From 658fe047dd711f3f1012ee53c07aca23b22fb5e4 Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 22 Jul 2026 16:17:53 +0200 Subject: [PATCH 08/15] Tighten workflow contract assertions against bypass The mutation-testing and pinned-action contract checks were satisfiable by non-runnable or mutable inputs: comment-only or malformed apt installs passed the package check, text-searched action pins could match a commented or otherwise non-step reference, and the schedule/concurrency checks only confirmed a cron and group existed rather than the exact daily per-ref behaviour. Parse the actual workflow steps and validate each `uses:` value with `re.fullmatch` via a new `_assert_pinned_step_uses` helper (plus `_iter_job_steps` for cross-job release steps), replacing the raw-text searches for the setup-rust, setup-python, and upload-artifact pins so a comment or mutable tag can no longer satisfy them. Make `_extract_apt_install_packages` skip commented lines and raise on `shlex` parse failures instead of masking malformed shell as a valid install. Assert the exact cron (`15 9 * * *`) and concurrency group (`mutation-testing-${{ github.ref }}`) for the mutation-testing workflow. Add focused unit tests for the tightened apt-install parser covering normal extraction, comment skipping, and malformed-shell rejection. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/workflows.py | 71 +++++++++++++++----- tests/test_helpers.py | 31 +++++++++ 2 files changed, 84 insertions(+), 18 deletions(-) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index 172885c..791d592 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -24,11 +24,11 @@ _UPLOAD_CODESCENE_COVERAGE_USES_RE = re.compile( r"^leynos/shared-actions/\.github/actions/upload-codescene-coverage@[0-9a-f]{40}$" ) -_SETUP_RUST_USES_TEXT_RE = re.compile( +_SETUP_RUST_USES_RE = re.compile( r"leynos/shared-actions/\.github/actions/setup-rust@[0-9a-f]{40}" ) -_SETUP_PYTHON_USES_TEXT_RE = re.compile(r"actions/setup-python@[0-9a-f]{40}") -_UPLOAD_ARTIFACT_USES_TEXT_RE = re.compile(r"actions/upload-artifact@[0-9a-f]{40}") +_SETUP_PYTHON_USES_RE = re.compile(r"actions/setup-python@[0-9a-f]{40}") +_UPLOAD_ARTIFACT_USES_RE = re.compile(r"actions/upload-artifact@[0-9a-f]{40}") def _is_pinned_action(uses: str, action_path: str) -> bool: @@ -359,9 +359,8 @@ def _assert_ci_workflow_contracts( assert "Cache Whitaker installation" in ci_workflow, ( "expected generated CI workflow to cache Whitaker installation" ) - assert _SETUP_RUST_USES_TEXT_RE.search(ci_workflow), ( - "expected generated CI workflow to use the shared setup-rust action " - "pinned to a full 40-hex commit SHA" + _assert_pinned_step_uses( + steps, _SETUP_RUST_USES_RE, "generated CI shared setup-rust action" ) build_test_checkout = [ step @@ -401,9 +400,8 @@ def _assert_ci_workflow_contracts( assert "Setup Python for audit manifest extraction" in ci_workflow, ( "expected generated CI workflow to install Python for audit metadata parsing" ) - assert _SETUP_PYTHON_USES_TEXT_RE.search(ci_workflow), ( - "expected generated CI workflow to use the setup-python action " - "pinned to a full 40-hex commit SHA" + _assert_pinned_step_uses( + steps, _SETUP_PYTHON_USES_RE, "generated CI setup-python action" ) assert "make audit" in ci_workflow, ( "expected generated CI workflow to run the dependency audit gate" @@ -452,6 +450,8 @@ def _assert_ci_workflow_contracts( _MUTATION_REUSABLE_WORKFLOW = ( "leynos/shared-actions/.github/workflows/mutation-cargo.yml" ) +_MUTATION_CRON = "15 9 * * *" +_MUTATION_CONCURRENCY_GROUP = "mutation-testing-${{ github.ref }}" def _extract_apt_install_packages(setup_commands: str) -> list[str]: @@ -472,6 +472,10 @@ def _extract_apt_install_packages(setup_commands: str) -> list[str]: lines = setup_commands.splitlines() marker = "apt-get install" for index, line in enumerate(lines): + # Ignore commented-out commands so a `# ... apt-get install ...` line + # can never be mistaken for a real install. + if line.strip().startswith("#"): + continue marker_at = line.find(marker) if marker_at == -1: continue @@ -482,10 +486,15 @@ def _extract_apt_install_packages(setup_commands: str) -> list[str]: cursor += 1 if cursor < len(lines): segment += " " + lines[cursor] + # Surface malformed shell instead of masking it as a valid install; a + # setup-commands script that cannot be parsed is not runnable. try: tokens = shlex.split(segment) - except ValueError: - tokens = segment.split() + except ValueError as error: + raise AssertionError( + "expected mutation job setup-commands apt-get install to be " + f"parseable shell, got {segment!r}" + ) from error return [token for token in tokens if not token.startswith("-")] return [] @@ -499,14 +508,16 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: triggers = require_mapping(parsed, "on", "mutation-testing workflow") schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") assert any( - isinstance(entry, dict) and entry.get("cron") for entry in schedule - ), "expected mutation-testing workflow to define a scheduled cron trigger" + isinstance(entry, dict) and entry.get("cron") == _MUTATION_CRON + for entry in schedule + ), f"expected mutation-testing workflow to schedule cron {_MUTATION_CRON!r}" assert "workflow_dispatch" in triggers, ( "expected mutation-testing workflow to support manual workflow_dispatch" ) concurrency = require_mapping(parsed, "concurrency", "mutation-testing workflow") - assert concurrency.get("group"), ( - "expected mutation-testing workflow to define a concurrency group" + assert concurrency.get("group") == _MUTATION_CONCURRENCY_GROUP, ( + "expected mutation-testing workflow concurrency group " + f"{_MUTATION_CONCURRENCY_GROUP!r}" ) assert concurrency.get("cancel-in-progress") is False, ( "expected mutation-testing workflow to queue runs " @@ -558,9 +569,10 @@ def _assert_release_workflow_contracts(release_workflow: str) -> None: step.get("with", {}).get("persist-credentials") is False for step in release_checkout_steps ), "expected release workflow checkout steps to disable credentials" - assert _UPLOAD_ARTIFACT_USES_TEXT_RE.search(release_workflow), ( - "expected app release workflow to use the upload-artifact action " - "pinned to a full 40-hex commit SHA" + _assert_pinned_step_uses( + _iter_job_steps(jobs), + _UPLOAD_ARTIFACT_USES_RE, + "app release upload-artifact action", ) cross_revision = "88f49ff79e777bef6d3564531636ee4d3cc2f8d2" assert f"CROSS_REVISION: {cross_revision}" in release_workflow, ( @@ -605,3 +617,26 @@ def extract_checkout_steps(jobs: dict[str, Any]) -> list[dict[str, Any]]: if isinstance(step, dict) and str(step.get("uses", "")).startswith("actions/checkout@") ] + +def _iter_job_steps(jobs: dict[str, Any]) -> list[Any]: + """Return every step across all jobs in a parsed jobs mapping.""" + return [ + step + for job in jobs.values() + if isinstance(job, dict) + for step in job.get("steps", []) + ] + +def _assert_pinned_step_uses( + steps: list[Any], uses_re: "re.Pattern[str]", label: str +) -> None: + """Assert one workflow step's ``uses:`` fully matches ``uses_re``. + + Parsing the actual ``uses:`` field, rather than scanning the raw workflow + text, ensures that a commented-out reference or a mutable tag or branch + cannot satisfy the pinned-action contract. + """ + assert any( + isinstance(step, dict) and uses_re.fullmatch(str(step.get("uses", ""))) + for step in steps + ), f"expected {label} pinned to a full 40-hex commit SHA in a workflow step" diff --git a/tests/test_helpers.py b/tests/test_helpers.py index 1fa4f73..f34bc7a 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -25,6 +25,7 @@ _disables_credential_persistence, _is_pinned_action, _step_mappings, + _extract_apt_install_packages, ) from tests.test_github_actions_integration import xfail_known_act_runtime_limitations from tests.utilities import ( @@ -35,6 +36,9 @@ ) +"""Validate direct helper-module error handling and edge cases.""" + + def _json_collections( children: st.SearchStrategy[Any], ) -> st.SearchStrategy[Any]: @@ -581,3 +585,30 @@ def _ci_workflow(*, persist_credentials: str, coverage_inputs: str) -> str: uses: leynos/shared-actions/.github/actions/generate-coverage@0000000000000000000000000000000000000000 with: {coverage_inputs}""" + +def test_extract_apt_install_packages_returns_non_flag_arguments() -> None: + """Return install arguments across backslash continuations, dropping flags.""" + setup_commands = ( + "export DEBIAN_FRONTEND=noninteractive\n" + "sudo apt-get update \\\n" + " && sudo apt-get install --yes --no-install-recommends clang lld mold\n" + ) + + assert _extract_apt_install_packages(setup_commands) == ["clang", "lld", "mold"], ( + "expected only the non-flag install arguments to be returned" + ) + +def test_extract_apt_install_packages_ignores_commented_installs() -> None: + """Skip commented-out install commands so they cannot satisfy the contract.""" + setup_commands = "# sudo apt-get install --yes clang lld mold\necho skip\n" + + assert _extract_apt_install_packages(setup_commands) == [], ( + "expected a commented-out apt-get install to be ignored" + ) + +def test_extract_apt_install_packages_rejects_malformed_shell() -> None: + """Surface malformed setup-commands shell instead of masking it as valid.""" + setup_commands = 'sudo apt-get install --yes "clang\n' + + with pytest.raises(AssertionError, match="parseable shell"): + _extract_apt_install_packages(setup_commands) From 52d1927c3b3d86bb9d7dc4c0e9f7dcf9894e6840 Mon Sep 17 00:00:00 2001 From: leynos Date: Fri, 24 Jul 2026 23:20:11 +0200 Subject: [PATCH 09/15] Reconcile mutation-testing contracts with audit workflow after rebase Rebasing onto main brought in the scheduled dependency-audit workflow contract (`_assert_audit_workflow_contracts` and its `_is_pinned_action` helper), which referenced the `_SETUP_RUST_USES_TEXT_RE` constant that this branch had renamed to `_SETUP_RUST_USES_RE` when switching the pinned-action checks to parse job steps and `re.fullmatch`. Point the audit contract at the renamed constant so both the audit and mutation-testing contracts share one setup-rust pattern. Also tidy merge artefacts: sort the workflows helper imports, drop a duplicate module docstring stranded in the test module by the three-way merge, and apply `ruff format` so `make check-fmt` passes. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/orchestration.py | 4 +--- tests/helpers/tooling_contracts/workflows.py | 7 ++++--- tests/test_helpers.py | 8 ++++---- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/tests/helpers/tooling_contracts/orchestration.py b/tests/helpers/tooling_contracts/orchestration.py index 7ddbf5a..1998ae3 100644 --- a/tests/helpers/tooling_contracts/orchestration.py +++ b/tests/helpers/tooling_contracts/orchestration.py @@ -104,9 +104,7 @@ def assert_generated_tooling_contracts( ) _assert_audit_workflow_contracts(audit_workflow) _assert_mutation_workflow_contracts(mutation_workflow) - assert_documentation_navigation_contracts( - docs_contents, repository_layout, flavour - ) + assert_documentation_navigation_contracts(docs_contents, repository_layout, flavour) assert "[Documentation contents](docs/contents.md)" in readme, ( "expected generated README to link to the documentation contents" ) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index 791d592..f1701ef 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -186,7 +186,7 @@ def _assert_audit_workflow_contracts(audit_workflow: str) -> None: "expected generated audit workflow to include one Setup Rust step" ) setup_rust_uses = str(setup_rust_steps[0].get("uses", "")) - assert _SETUP_RUST_USES_TEXT_RE.fullmatch(setup_rust_uses), ( + assert _SETUP_RUST_USES_RE.fullmatch(setup_rust_uses), ( "expected generated audit workflow to use setup-rust pinned to a full " f"40-hex commit SHA, got {setup_rust_uses!r}" ) @@ -554,8 +554,7 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: ) for package in _MUTATION_SETUP_PACKAGES: assert package in installed_packages, ( - "expected mutation job setup-commands apt-get install to include " - f"{package}" + f"expected mutation job setup-commands apt-get install to include {package}" ) @@ -618,6 +617,7 @@ def extract_checkout_steps(jobs: dict[str, Any]) -> list[dict[str, Any]]: and str(step.get("uses", "")).startswith("actions/checkout@") ] + def _iter_job_steps(jobs: dict[str, Any]) -> list[Any]: """Return every step across all jobs in a parsed jobs mapping.""" return [ @@ -627,6 +627,7 @@ def _iter_job_steps(jobs: dict[str, Any]) -> list[Any]: for step in job.get("steps", []) ] + def _assert_pinned_step_uses( steps: list[Any], uses_re: "re.Pattern[str]", label: str ) -> None: diff --git a/tests/test_helpers.py b/tests/test_helpers.py index f34bc7a..f76cc6e 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -23,9 +23,9 @@ from tests.helpers.tooling_contracts import assert_ci_coverage_action_contract from tests.helpers.tooling_contracts.workflows import ( _disables_credential_persistence, + _extract_apt_install_packages, _is_pinned_action, _step_mappings, - _extract_apt_install_packages, ) from tests.test_github_actions_integration import xfail_known_act_runtime_limitations from tests.utilities import ( @@ -36,9 +36,6 @@ ) -"""Validate direct helper-module error handling and edge cases.""" - - def _json_collections( children: st.SearchStrategy[Any], ) -> st.SearchStrategy[Any]: @@ -586,6 +583,7 @@ def _ci_workflow(*, persist_credentials: str, coverage_inputs: str) -> str: with: {coverage_inputs}""" + def test_extract_apt_install_packages_returns_non_flag_arguments() -> None: """Return install arguments across backslash continuations, dropping flags.""" setup_commands = ( @@ -598,6 +596,7 @@ def test_extract_apt_install_packages_returns_non_flag_arguments() -> None: "expected only the non-flag install arguments to be returned" ) + def test_extract_apt_install_packages_ignores_commented_installs() -> None: """Skip commented-out install commands so they cannot satisfy the contract.""" setup_commands = "# sudo apt-get install --yes clang lld mold\necho skip\n" @@ -606,6 +605,7 @@ def test_extract_apt_install_packages_ignores_commented_installs() -> None: "expected a commented-out apt-get install to be ignored" ) + def test_extract_apt_install_packages_rejects_malformed_shell() -> None: """Surface malformed setup-commands shell instead of masking it as valid.""" setup_commands = 'sudo apt-get install --yes "clang\n' From 193bf43c188aa1ed7ebbd094b7a8beb57f4abc9a Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 26 Jul 2026 04:15:47 +0200 Subject: [PATCH 10/15] Reject extra mutation triggers and decompose the non-SHA ref strategy Tighten the mutation-testing workflow contract to require the trigger mapping to be exactly {schedule, workflow_dispatch}, so an added push or pull-request trigger is rejected rather than silently accepted; the exact cron and manual-dispatch checks are preserved. This mirrors the audit workflow contract's trigger assertion. Refactor the `_non_sha_refs` Hypothesis strategy to drop its four-way `match kind` block. Extract focused strategies for too-short and too-long hex refs, branch names, and uppercased 40-character hex refs, then compose them with `st.one_of`, preserving the guarantee that generated refs are never a 40-character lowercase-hex commit SHA. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/workflows.py | 4 ++ tests/test_helpers.py | 56 ++++++++++---------- 2 files changed, 31 insertions(+), 29 deletions(-) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index f1701ef..0b3d10a 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -506,6 +506,10 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: "expected mutation-testing workflow root permissions to grant no scopes" ) triggers = require_mapping(parsed, "on", "mutation-testing workflow") + assert set(triggers) == {"schedule", "workflow_dispatch"}, ( + "expected mutation-testing workflow to trigger only on schedule and " + "workflow_dispatch, rejecting push or pull-request runs" + ) schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") assert any( isinstance(entry, dict) and entry.get("cron") == _MUTATION_CRON diff --git a/tests/test_helpers.py b/tests/test_helpers.py index f76cc6e..5ff969b 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -170,35 +170,33 @@ def test_require_sequence_property(key: str, value: object) -> None: _full_shas = st.text(alphabet=_HEX_ALPHABET, min_size=40, max_size=40) +# Focused strategies for refs that are never a 40-character lowercase-hex SHA, +# composed into ``_non_sha_refs`` below. +_too_short_hex_refs = st.text(alphabet=_HEX_ALPHABET, min_size=0, max_size=39) +_too_long_hex_refs = st.text(alphabet=_HEX_ALPHABET, min_size=41, max_size=64) +_branch_refs = st.sampled_from(("main", "master", "rolling", "HEAD", "v1.2.3")) + + @st.composite -def _non_sha_refs(draw: st.DrawFn) -> str: - """Draw refs that are never a 40-character lowercase-hex commit SHA.""" - kind = draw(st.sampled_from(("short", "long", "uppercased", "branch"))) - match kind: - case "short": - size = draw(st.integers(min_value=0, max_value=39)) - return draw(st.text(alphabet=_HEX_ALPHABET, min_size=size, max_size=size)) - case "long": - size = draw(st.integers(min_value=41, max_value=64)) - return draw(st.text(alphabet=_HEX_ALPHABET, min_size=size, max_size=size)) - case "uppercased": - hexes = draw(st.text(alphabet=_HEX_ALPHABET, min_size=40, max_size=40)) - position = draw(st.integers(min_value=0, max_value=39)) - existing = hexes[position] - # Uppercasing a hex digit is a no-op, so fall back to a hex letter - # to keep the ref out of the lowercase-hex SHA shape. - flipped = ( - existing.upper() - if existing.isalpha() - else draw(st.sampled_from("ABCDEF")) - ) - return f"{hexes[:position]}{flipped}{hexes[position + 1 :]}" - case "branch": - return draw( - st.sampled_from(("main", "master", "rolling", "HEAD", "v1.2.3")) - ) - case _: # pragma: no cover - kind is drawn from the sampled set above - raise AssertionError(f"unexpected ref kind: {kind}") +def _uppercased_hex_refs(draw: st.DrawFn) -> str: + """Draw a 40-character hex ref carrying at least one uppercase digit.""" + hexes = draw(st.text(alphabet=_HEX_ALPHABET, min_size=40, max_size=40)) + position = draw(st.integers(min_value=0, max_value=39)) + existing = hexes[position] + # Uppercasing a hex digit is a no-op, so fall back to a hex letter to keep + # the ref out of the lowercase-hex SHA shape. + flipped = ( + existing.upper() if existing.isalpha() else draw(st.sampled_from("ABCDEF")) + ) + return f"{hexes[:position]}{flipped}{hexes[position + 1 :]}" + + +_non_sha_refs = st.one_of( + _too_short_hex_refs, + _too_long_hex_refs, + _uppercased_hex_refs(), + _branch_refs, +) @given(path=_action_paths, sha=_full_shas) @@ -212,7 +210,7 @@ def test_is_pinned_action_accepts_full_sha(path: str, sha: str) -> None: ) -@given(path=_action_paths, ref=_non_sha_refs()) +@given(path=_action_paths, ref=_non_sha_refs) def test_is_pinned_action_rejects_non_sha_refs(path: str, ref: str) -> None: """Refs that are not a full 40-hex commit SHA are never treated as pinned.""" assert not _is_pinned_action(f"{path}@{ref}", path), ( From 39978b4a2d0feb4902f7571ee401b2c2efec3928 Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 26 Jul 2026 04:21:25 +0200 Subject: [PATCH 11/15] Add property tests for the apt-get install parser The `_extract_apt_install_packages` shell parser handled many input shapes (flags in any position, comment lines, backslash continuations) but was only exercised by example tests, leaving its invariant implicit. Add Hypothesis strategies for shell-safe package and flag tokens and three property tests: one asserts the parser returns exactly the non-flag arguments in order for an install line of interleaved flags and packages (with varied indentation and optional sudo); one asserts a commented install line never contributes packages; and one asserts packages split onto a backslash-continued line are still recovered. The prior example tests, including the exact-template form and the malformed-shell rejection, remain as concrete anchors. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_helpers.py | 67 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/tests/test_helpers.py b/tests/test_helpers.py index 5ff969b..e520d6a 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -610,3 +610,70 @@ def test_extract_apt_install_packages_rejects_malformed_shell() -> None: with pytest.raises(AssertionError, match="parseable shell"): _extract_apt_install_packages(setup_commands) + + +# Shell-safe package tokens: a leading alphanumeric (so they never look like a +# flag) followed by characters valid in a Debian package name. Flag tokens +# always begin with ``--`` so the parser drops them. +_apt_package_tokens = st.builds( + lambda head, tail: head + tail, + st.sampled_from("abcdefghijklmnopqrstuvwxyz0123456789"), + st.text(alphabet="abcdefghijklmnopqrstuvwxyz0123456789+.-", max_size=11), +) +_apt_flag_tokens = st.builds( + lambda word: "--" + word, + st.text(alphabet="abcdefghijklmnopqrstuvwxyz-", min_size=1, max_size=12), +) +_apt_install_tokens = st.lists( + st.one_of( + st.tuples(_apt_package_tokens, st.just(False)), + st.tuples(_apt_flag_tokens, st.just(True)), + ), + min_size=1, + max_size=6, +) + + +@given( + tagged=_apt_install_tokens, + use_sudo=st.booleans(), + indent=st.text(alphabet=" \t", max_size=3), +) +def test_extract_apt_install_packages_recovers_non_flag_arguments( + tagged: list[tuple[str, bool]], use_sudo: bool, indent: str +) -> None: + """Return exactly the non-flag install arguments, in order, dropping flags.""" + tokens = [token for token, _ in tagged] + expected = [token for token, is_flag in tagged if not is_flag] + prefix = "sudo " if use_sudo else "" + setup_commands = ( + "export DEBIAN_FRONTEND=noninteractive\n" + f"{indent}{prefix}apt-get install {' '.join(tokens)}\n" + ) + + assert _extract_apt_install_packages(setup_commands) == expected + + +@given( + packages=st.lists(_apt_package_tokens, min_size=1, max_size=5), + indent=st.text(alphabet=" \t", max_size=3), +) +def test_extract_apt_install_packages_skips_commented_installs( + packages: list[str], indent: str +) -> None: + """A commented apt-get install line never contributes packages.""" + setup_commands = ( + f"{indent}# sudo apt-get install --yes {' '.join(packages)}\necho noop\n" + ) + + assert _extract_apt_install_packages(setup_commands) == [] + + +@given(packages=st.lists(_apt_package_tokens, min_size=1, max_size=4)) +def test_extract_apt_install_packages_follows_line_continuations( + packages: list[str], +) -> None: + """Recover packages split onto a backslash-continued line.""" + setup_commands = f"sudo apt-get install --yes \\\n {' '.join(packages)}\n" + + assert _extract_apt_install_packages(setup_commands) == packages From b7c06951cb9f802e364938b400e8cc30fa65da1c Mon Sep 17 00:00:00 2001 From: leynos Date: Mon, 27 Jul 2026 14:52:21 +0200 Subject: [PATCH 12/15] Parse setup-commands per shell command and tighten contract assertions Rework `_extract_apt_install_packages` so it recognises a real install command rather than any line containing the substring "apt-get install". Each logical line is tokenised with shlex, split on shell control operators (`&&`, `||`, `;`, `|`, `&`), and a command counts only when its tokens are an optional `sudo` followed by exactly `apt-get install`. Embedded text such as an `echo "... apt-get install ..."` argument is now ignored. Add a negative test covering that case. Also address adjacent review points: - Assert the mutation-testing `on.schedule` equals exactly `[{"cron": _MUTATION_CRON}]`, rejecting extra schedule entries or fields, instead of an existential any() check. - Reduce the `_extract_apt_install_packages` and `_assert_pinned_step_uses` docstrings to single-line summaries, matching the sibling helpers. - Annotate the module-level Hypothesis strategy aliases with explicit `SearchStrategy` element types so they stay statically checked. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/helpers/tooling_contracts/workflows.py | 86 ++++++++++---------- tests/test_helpers.py | 30 +++++-- 2 files changed, 65 insertions(+), 51 deletions(-) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index 0b3d10a..a04b99a 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -454,48 +454,51 @@ def _assert_ci_workflow_contracts( _MUTATION_CONCURRENCY_GROUP = "mutation-testing-${{ github.ref }}" -def _extract_apt_install_packages(setup_commands: str) -> list[str]: - """Return packages passed to the setup-commands ``apt-get install`` call. +_SHELL_COMMAND_SEPARATORS = frozenset({"&&", "||", ";", "|", "&"}) - Parameters - ---------- - setup_commands - Setup-commands shell script from the mutation job ``with`` inputs. - Returns - ------- - list[str] - Non-flag arguments supplied to ``apt-get install``, following any - backslash line continuations, or an empty list when no install command - is present. - """ - lines = setup_commands.splitlines() - marker = "apt-get install" - for index, line in enumerate(lines): - # Ignore commented-out commands so a `# ... apt-get install ...` line - # can never be mistaken for a real install. - if line.strip().startswith("#"): - continue - marker_at = line.find(marker) - if marker_at == -1: - continue - segment = line[marker_at + len(marker) :] - cursor = index - while cursor < len(lines) and lines[cursor].rstrip().endswith("\\"): - segment = segment.rstrip().removesuffix("\\") - cursor += 1 - if cursor < len(lines): - segment += " " + lines[cursor] +def _split_shell_commands(tokens: list[str]) -> list[list[str]]: + """Split a shell token stream into commands on shell control operators.""" + commands: list[list[str]] = [] + current: list[str] = [] + for token in tokens: + if token in _SHELL_COMMAND_SEPARATORS: + commands.append(current) + current = [] + else: + current.append(token) + commands.append(current) + return commands + + +def _apt_install_packages(command: list[str]) -> list[str] | None: + """Return non-flag args of ``[sudo] apt-get install ...``, else ``None``.""" + words = command[1:] if command[:1] == ["sudo"] else command + if words[:2] != ["apt-get", "install"]: + return None + return [arg for arg in words[2:] if not arg.startswith("-")] + + +def _extract_apt_install_packages(setup_commands: str) -> list[str]: + """Return the packages installed by a setup-commands ``apt-get install``.""" + # Join backslash continuations so each command sits on one logical line. + joined = setup_commands.replace("\\\n", " ") + for line in joined.splitlines(): # Surface malformed shell instead of masking it as a valid install; a # setup-commands script that cannot be parsed is not runnable. try: - tokens = shlex.split(segment) + tokens = shlex.split(line, comments=True) except ValueError as error: raise AssertionError( - "expected mutation job setup-commands apt-get install to be " - f"parseable shell, got {segment!r}" + "expected mutation job setup-commands to be parseable shell, " + f"got {line!r}" ) from error - return [token for token in tokens if not token.startswith("-")] + # Only a real `[sudo] apt-get install` command counts; embedded text + # such as an echo argument must never be treated as an install. + for command in _split_shell_commands(tokens): + packages = _apt_install_packages(command) + if packages is not None: + return packages return [] @@ -511,10 +514,10 @@ def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: "workflow_dispatch, rejecting push or pull-request runs" ) schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") - assert any( - isinstance(entry, dict) and entry.get("cron") == _MUTATION_CRON - for entry in schedule - ), f"expected mutation-testing workflow to schedule cron {_MUTATION_CRON!r}" + assert schedule == [{"cron": _MUTATION_CRON}], ( + "expected mutation-testing workflow on.schedule to be exactly one cron " + f"entry {_MUTATION_CRON!r}" + ) assert "workflow_dispatch" in triggers, ( "expected mutation-testing workflow to support manual workflow_dispatch" ) @@ -635,12 +638,7 @@ def _iter_job_steps(jobs: dict[str, Any]) -> list[Any]: def _assert_pinned_step_uses( steps: list[Any], uses_re: "re.Pattern[str]", label: str ) -> None: - """Assert one workflow step's ``uses:`` fully matches ``uses_re``. - - Parsing the actual ``uses:`` field, rather than scanning the raw workflow - text, ensures that a commented-out reference or a mutable tag or branch - cannot satisfy the pinned-action contract. - """ + """Assert one workflow step's ``uses:`` fully matches ``uses_re``.""" assert any( isinstance(step, dict) and uses_re.fullmatch(str(step.get("uses", ""))) for step in steps diff --git a/tests/test_helpers.py b/tests/test_helpers.py index e520d6a..8f5b3b7 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -172,9 +172,15 @@ def test_require_sequence_property(key: str, value: object) -> None: # Focused strategies for refs that are never a 40-character lowercase-hex SHA, # composed into ``_non_sha_refs`` below. -_too_short_hex_refs = st.text(alphabet=_HEX_ALPHABET, min_size=0, max_size=39) -_too_long_hex_refs = st.text(alphabet=_HEX_ALPHABET, min_size=41, max_size=64) -_branch_refs = st.sampled_from(("main", "master", "rolling", "HEAD", "v1.2.3")) +_too_short_hex_refs: st.SearchStrategy[str] = st.text( + alphabet=_HEX_ALPHABET, min_size=0, max_size=39 +) +_too_long_hex_refs: st.SearchStrategy[str] = st.text( + alphabet=_HEX_ALPHABET, min_size=41, max_size=64 +) +_branch_refs: st.SearchStrategy[str] = st.sampled_from( + ("main", "master", "rolling", "HEAD", "v1.2.3") +) @st.composite @@ -191,7 +197,7 @@ def _uppercased_hex_refs(draw: st.DrawFn) -> str: return f"{hexes[:position]}{flipped}{hexes[position + 1 :]}" -_non_sha_refs = st.one_of( +_non_sha_refs: st.SearchStrategy[str] = st.one_of( _too_short_hex_refs, _too_long_hex_refs, _uppercased_hex_refs(), @@ -612,19 +618,29 @@ def test_extract_apt_install_packages_rejects_malformed_shell() -> None: _extract_apt_install_packages(setup_commands) +def test_extract_apt_install_packages_ignores_echoed_marker() -> None: + """Ignore an echo whose argument merely contains the install marker text.""" + setup_commands = 'echo "run apt-get install clang lld mold first"\n' + + assert _extract_apt_install_packages(setup_commands) == [], ( + "expected embedded marker text in an echo argument not to count as an " + "install command" + ) + + # Shell-safe package tokens: a leading alphanumeric (so they never look like a # flag) followed by characters valid in a Debian package name. Flag tokens # always begin with ``--`` so the parser drops them. -_apt_package_tokens = st.builds( +_apt_package_tokens: st.SearchStrategy[str] = st.builds( lambda head, tail: head + tail, st.sampled_from("abcdefghijklmnopqrstuvwxyz0123456789"), st.text(alphabet="abcdefghijklmnopqrstuvwxyz0123456789+.-", max_size=11), ) -_apt_flag_tokens = st.builds( +_apt_flag_tokens: st.SearchStrategy[str] = st.builds( lambda word: "--" + word, st.text(alphabet="abcdefghijklmnopqrstuvwxyz-", min_size=1, max_size=12), ) -_apt_install_tokens = st.lists( +_apt_install_tokens: st.SearchStrategy[list[tuple[str, bool]]] = st.lists( st.one_of( st.tuples(_apt_package_tokens, st.just(False)), st.tuples(_apt_flag_tokens, st.just(True)), From 2b4c15fbd9f1b2563ac08e23909601259af967af Mon Sep 17 00:00:00 2001 From: leynos Date: Mon, 27 Jul 2026 14:56:33 +0200 Subject: [PATCH 13/15] Add negative tests for the mutation-testing schedule contract The mutation-testing schedule assertion already requires `schedule == [{"cron": _MUTATION_CRON}]`, but nothing proved the contract actually rejects a non-conforming schedule, leaving the invariant asserted only against the canonical rendered workflow. Add two focused unit tests that drive `_assert_mutation_workflow_contracts` with hand-built workflows and confirm it raises: one with a second schedule entry beyond the sanctioned cron, and one with an extra key smuggled into the single schedule entry. A small fixture builder populates just the fields inspected before the schedule check so the assertion is exercised in isolation. Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_helpers.py | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/tests/test_helpers.py b/tests/test_helpers.py index 8f5b3b7..dfd1ef1 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -22,6 +22,8 @@ from tests.helpers.rendering import read_generated_file from tests.helpers.tooling_contracts import assert_ci_coverage_action_contract from tests.helpers.tooling_contracts.workflows import ( + _MUTATION_CRON, + _assert_mutation_workflow_contracts, _disables_credential_persistence, _extract_apt_install_packages, _is_pinned_action, @@ -693,3 +695,32 @@ def test_extract_apt_install_packages_follows_line_continuations( setup_commands = f"sudo apt-get install --yes \\\n {' '.join(packages)}\n" assert _extract_apt_install_packages(setup_commands) == packages + + +def _mutation_workflow_with_schedule(schedule_block: str) -> str: + """Build a mutation-testing workflow whose ``on`` reaches the schedule check. + + Only the fields inspected before the schedule assertion are populated, so + the returned document exercises the schedule contract in isolation. + """ + return f'permissions: {{}}\n"on":\n{schedule_block} workflow_dispatch:\n' + + +def test_assert_mutation_workflow_contracts_rejects_extra_schedule_entry() -> None: + """Reject a second schedule entry beyond the single sanctioned cron.""" + workflow = _mutation_workflow_with_schedule( + f' schedule:\n - cron: "{_MUTATION_CRON}"\n - cron: "0 0 * * *"\n' + ) + + with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): + _assert_mutation_workflow_contracts(workflow) + + +def test_assert_mutation_workflow_contracts_rejects_extra_schedule_key() -> None: + """Reject an extra key smuggled into the single schedule entry.""" + workflow = _mutation_workflow_with_schedule( + f' schedule:\n - cron: "{_MUTATION_CRON}"\n branches: ["main"]\n' + ) + + with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): + _assert_mutation_workflow_contracts(workflow) From 78a7112b915112c542181f104992276929917498 Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 29 Jul 2026 02:06:40 +0200 Subject: [PATCH 14/15] Split the mutation-testing contract into its own module The mutation-testing contract had accumulated inside the general workflows helper, leaving `workflows.py` and `test_helpers.py` as the two largest files in the test tree and mixing an unrelated shell parser in with the GitHub Actions assertions. Move the mutation contract into `tooling_contracts/mutation.py`: the `_MUTATION_*` constants, the setup-commands shell parser (`_split_shell_commands`, `_apt_install_packages`, `_extract_apt_install_packages`), and `_assert_mutation_workflow_contracts`. Point the orchestration entrypoint at the new module and drop the now-unused `shlex` import from `workflows.py`. Move the matching apt-install and mutation-schedule tests into `tests/test_mutation_contracts.py`, alongside the strategies they use. The move is behaviour-preserving: the suite still reports 77 passed and 1 skipped. Every touched file now sits well under 400 logical lines (workflows 175, mutation 70, test_helpers 247, test_mutation_contracts 57). Also trim the `_mutation_workflow_with_schedule` docstring to a single-line summary, matching the sibling private helpers. Co-Authored-By: Claude Opus 5 (1M context) --- tests/helpers/tooling_contracts/mutation.py | 135 ++++++++++++++++ .../tooling_contracts/orchestration.py | 4 +- tests/helpers/tooling_contracts/workflows.py | 121 --------------- tests/test_helpers.py | 139 ----------------- tests/test_mutation_contracts.py | 145 ++++++++++++++++++ 5 files changed, 283 insertions(+), 261 deletions(-) create mode 100644 tests/helpers/tooling_contracts/mutation.py create mode 100644 tests/test_mutation_contracts.py diff --git a/tests/helpers/tooling_contracts/mutation.py b/tests/helpers/tooling_contracts/mutation.py new file mode 100644 index 0000000..c5c604d --- /dev/null +++ b/tests/helpers/tooling_contracts/mutation.py @@ -0,0 +1,135 @@ +"""Assert the rendered mutation-testing workflow contract. + +The mutation job's pinned reusable-workflow SHA is asserted by shape, not by +exact value: Dependabot owns bumping it, so hard-coding a value here would make +the suite fail on every routine bump. +""" + +from __future__ import annotations + +import re +import shlex + +from tests.helpers.generated_files import ( + parse_yaml_mapping, + require_mapping, + require_sequence, +) + +_MUTATION_JOB_PERMISSIONS = {"contents": "read", "id-token": "write"} +_MUTATION_SETUP_PACKAGES = ("clang", "lld", "mold") +_MUTATION_REUSABLE_WORKFLOW = ( + "leynos/shared-actions/.github/workflows/mutation-cargo.yml" +) +_MUTATION_CRON = "15 9 * * *" +_MUTATION_CONCURRENCY_GROUP = "mutation-testing-${{ github.ref }}" + +_SHELL_COMMAND_SEPARATORS = frozenset({"&&", "||", ";", "|", "&"}) + + +def _split_shell_commands(tokens: list[str]) -> list[list[str]]: + """Split a shell token stream into commands on shell control operators.""" + commands: list[list[str]] = [] + current: list[str] = [] + for token in tokens: + if token in _SHELL_COMMAND_SEPARATORS: + commands.append(current) + current = [] + else: + current.append(token) + commands.append(current) + return commands + + +def _apt_install_packages(command: list[str]) -> list[str] | None: + """Return non-flag args of ``[sudo] apt-get install ...``, else ``None``.""" + words = command[1:] if command[:1] == ["sudo"] else command + if words[:2] != ["apt-get", "install"]: + return None + return [arg for arg in words[2:] if not arg.startswith("-")] + + +def _extract_apt_install_packages(setup_commands: str) -> list[str]: + """Return the packages installed by a setup-commands ``apt-get install``.""" + # Join backslash continuations so each command sits on one logical line. + joined = setup_commands.replace("\\\n", " ") + for line in joined.splitlines(): + # Surface malformed shell instead of masking it as a valid install; a + # setup-commands script that cannot be parsed is not runnable. + try: + tokens = shlex.split(line, comments=True) + except ValueError as error: + raise AssertionError( + "expected mutation job setup-commands to be parseable shell, " + f"got {line!r}" + ) from error + # Only a real `[sudo] apt-get install` command counts; embedded text + # such as an echo argument must never be treated as an install. + for command in _split_shell_commands(tokens): + packages = _apt_install_packages(command) + if packages is not None: + return packages + return [] + + +def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: + """Assert generated mutation-testing workflow contracts.""" + parsed = parse_yaml_mapping(mutation_workflow, "mutation-testing workflow") + assert parsed.get("permissions") == {}, ( + "expected mutation-testing workflow root permissions to grant no scopes" + ) + triggers = require_mapping(parsed, "on", "mutation-testing workflow") + assert set(triggers) == {"schedule", "workflow_dispatch"}, ( + "expected mutation-testing workflow to trigger only on schedule and " + "workflow_dispatch, rejecting push or pull-request runs" + ) + schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") + assert schedule == [{"cron": _MUTATION_CRON}], ( + "expected mutation-testing workflow on.schedule to be exactly one cron " + f"entry {_MUTATION_CRON!r}" + ) + assert "workflow_dispatch" in triggers, ( + "expected mutation-testing workflow to support manual workflow_dispatch" + ) + concurrency = require_mapping(parsed, "concurrency", "mutation-testing workflow") + assert concurrency.get("group") == _MUTATION_CONCURRENCY_GROUP, ( + "expected mutation-testing workflow concurrency group " + f"{_MUTATION_CONCURRENCY_GROUP!r}" + ) + assert concurrency.get("cancel-in-progress") is False, ( + "expected mutation-testing workflow to queue runs " + "(concurrency.cancel-in-progress: false)" + ) + jobs = require_mapping(parsed, "jobs", "mutation-testing workflow") + mutation = require_mapping(jobs, "mutation", "mutation-testing workflow jobs") + mutation_uses = str(mutation.get("uses", "")) + pinned_prefix = f"{_MUTATION_REUSABLE_WORKFLOW}@" + assert mutation_uses.startswith(pinned_prefix), ( + "expected mutation job to call the shared mutation-cargo reusable workflow" + ) + # The specific SHA is owned by Dependabot and deliberately not hard-coded, so + # the contract survives dependency bumps while requiring an immutable pin. + assert re.fullmatch(r"[0-9a-f]{40}", mutation_uses[len(pinned_prefix) :]), ( + "expected mutation job to pin the reusable workflow to a full commit SHA" + ) + permissions = require_mapping(mutation, "permissions", "mutation job") + assert permissions == _MUTATION_JOB_PERMISSIONS, ( + "expected mutation job permissions to stay scoped to " + "contents: read and id-token: write" + ) + inputs = require_mapping(mutation, "with", "mutation job") + assert inputs.get("extra-args") == "--all-features", ( + "expected mutation job to mirror the CI --all-features test baseline" + ) + setup_commands = inputs.get("setup-commands") + assert isinstance(setup_commands, str), ( + "expected mutation job setup-commands to be a string" + ) + installed_packages = _extract_apt_install_packages(setup_commands) + assert installed_packages, ( + "expected mutation job setup-commands to install packages via apt-get" + ) + for package in _MUTATION_SETUP_PACKAGES: + assert package in installed_packages, ( + f"expected mutation job setup-commands apt-get install to include {package}" + ) diff --git a/tests/helpers/tooling_contracts/orchestration.py b/tests/helpers/tooling_contracts/orchestration.py index 1998ae3..e42a4fa 100644 --- a/tests/helpers/tooling_contracts/orchestration.py +++ b/tests/helpers/tooling_contracts/orchestration.py @@ -12,10 +12,12 @@ assert_documentation_navigation_contracts, ) from tests.helpers.tooling_contracts.makefile import _assert_makefile_contracts +from tests.helpers.tooling_contracts.mutation import ( + _assert_mutation_workflow_contracts, +) from tests.helpers.tooling_contracts.workflows import ( _assert_audit_workflow_contracts, _assert_ci_workflow_contracts, - _assert_mutation_workflow_contracts, _assert_release_workflow_contracts, ) diff --git a/tests/helpers/tooling_contracts/workflows.py b/tests/helpers/tooling_contracts/workflows.py index a04b99a..8ee5cdf 100644 --- a/tests/helpers/tooling_contracts/workflows.py +++ b/tests/helpers/tooling_contracts/workflows.py @@ -9,7 +9,6 @@ from __future__ import annotations import re -import shlex from typing import Any from tests.helpers.generated_files import ( @@ -445,126 +444,6 @@ def _assert_ci_workflow_contracts( ) -_MUTATION_JOB_PERMISSIONS = {"contents": "read", "id-token": "write"} -_MUTATION_SETUP_PACKAGES = ("clang", "lld", "mold") -_MUTATION_REUSABLE_WORKFLOW = ( - "leynos/shared-actions/.github/workflows/mutation-cargo.yml" -) -_MUTATION_CRON = "15 9 * * *" -_MUTATION_CONCURRENCY_GROUP = "mutation-testing-${{ github.ref }}" - - -_SHELL_COMMAND_SEPARATORS = frozenset({"&&", "||", ";", "|", "&"}) - - -def _split_shell_commands(tokens: list[str]) -> list[list[str]]: - """Split a shell token stream into commands on shell control operators.""" - commands: list[list[str]] = [] - current: list[str] = [] - for token in tokens: - if token in _SHELL_COMMAND_SEPARATORS: - commands.append(current) - current = [] - else: - current.append(token) - commands.append(current) - return commands - - -def _apt_install_packages(command: list[str]) -> list[str] | None: - """Return non-flag args of ``[sudo] apt-get install ...``, else ``None``.""" - words = command[1:] if command[:1] == ["sudo"] else command - if words[:2] != ["apt-get", "install"]: - return None - return [arg for arg in words[2:] if not arg.startswith("-")] - - -def _extract_apt_install_packages(setup_commands: str) -> list[str]: - """Return the packages installed by a setup-commands ``apt-get install``.""" - # Join backslash continuations so each command sits on one logical line. - joined = setup_commands.replace("\\\n", " ") - for line in joined.splitlines(): - # Surface malformed shell instead of masking it as a valid install; a - # setup-commands script that cannot be parsed is not runnable. - try: - tokens = shlex.split(line, comments=True) - except ValueError as error: - raise AssertionError( - "expected mutation job setup-commands to be parseable shell, " - f"got {line!r}" - ) from error - # Only a real `[sudo] apt-get install` command counts; embedded text - # such as an echo argument must never be treated as an install. - for command in _split_shell_commands(tokens): - packages = _apt_install_packages(command) - if packages is not None: - return packages - return [] - - -def _assert_mutation_workflow_contracts(mutation_workflow: str) -> None: - """Assert generated mutation-testing workflow contracts.""" - parsed = parse_yaml_mapping(mutation_workflow, "mutation-testing workflow") - assert parsed.get("permissions") == {}, ( - "expected mutation-testing workflow root permissions to grant no scopes" - ) - triggers = require_mapping(parsed, "on", "mutation-testing workflow") - assert set(triggers) == {"schedule", "workflow_dispatch"}, ( - "expected mutation-testing workflow to trigger only on schedule and " - "workflow_dispatch, rejecting push or pull-request runs" - ) - schedule = require_sequence(triggers, "schedule", "mutation-testing workflow on") - assert schedule == [{"cron": _MUTATION_CRON}], ( - "expected mutation-testing workflow on.schedule to be exactly one cron " - f"entry {_MUTATION_CRON!r}" - ) - assert "workflow_dispatch" in triggers, ( - "expected mutation-testing workflow to support manual workflow_dispatch" - ) - concurrency = require_mapping(parsed, "concurrency", "mutation-testing workflow") - assert concurrency.get("group") == _MUTATION_CONCURRENCY_GROUP, ( - "expected mutation-testing workflow concurrency group " - f"{_MUTATION_CONCURRENCY_GROUP!r}" - ) - assert concurrency.get("cancel-in-progress") is False, ( - "expected mutation-testing workflow to queue runs " - "(concurrency.cancel-in-progress: false)" - ) - jobs = require_mapping(parsed, "jobs", "mutation-testing workflow") - mutation = require_mapping(jobs, "mutation", "mutation-testing workflow jobs") - mutation_uses = str(mutation.get("uses", "")) - pinned_prefix = f"{_MUTATION_REUSABLE_WORKFLOW}@" - assert mutation_uses.startswith(pinned_prefix), ( - "expected mutation job to call the shared mutation-cargo reusable workflow" - ) - # The specific SHA is owned by Dependabot and deliberately not hard-coded, so - # the contract survives dependency bumps while requiring an immutable pin. - assert re.fullmatch(r"[0-9a-f]{40}", mutation_uses[len(pinned_prefix) :]), ( - "expected mutation job to pin the reusable workflow to a full commit SHA" - ) - permissions = require_mapping(mutation, "permissions", "mutation job") - assert permissions == _MUTATION_JOB_PERMISSIONS, ( - "expected mutation job permissions to stay scoped to " - "contents: read and id-token: write" - ) - inputs = require_mapping(mutation, "with", "mutation job") - assert inputs.get("extra-args") == "--all-features", ( - "expected mutation job to mirror the CI --all-features test baseline" - ) - setup_commands = inputs.get("setup-commands") - assert isinstance(setup_commands, str), ( - "expected mutation job setup-commands to be a string" - ) - installed_packages = _extract_apt_install_packages(setup_commands) - assert installed_packages, ( - "expected mutation job setup-commands to install packages via apt-get" - ) - for package in _MUTATION_SETUP_PACKAGES: - assert package in installed_packages, ( - f"expected mutation job setup-commands apt-get install to include {package}" - ) - - def _assert_release_workflow_contracts(release_workflow: str) -> None: """Assert generated release workflow contracts.""" parsed_release_workflow = parse_yaml_mapping(release_workflow, "release workflow") diff --git a/tests/test_helpers.py b/tests/test_helpers.py index dfd1ef1..c9afe83 100644 --- a/tests/test_helpers.py +++ b/tests/test_helpers.py @@ -22,10 +22,7 @@ from tests.helpers.rendering import read_generated_file from tests.helpers.tooling_contracts import assert_ci_coverage_action_contract from tests.helpers.tooling_contracts.workflows import ( - _MUTATION_CRON, - _assert_mutation_workflow_contracts, _disables_credential_persistence, - _extract_apt_install_packages, _is_pinned_action, _step_mappings, ) @@ -588,139 +585,3 @@ def _ci_workflow(*, persist_credentials: str, coverage_inputs: str) -> str: uses: leynos/shared-actions/.github/actions/generate-coverage@0000000000000000000000000000000000000000 with: {coverage_inputs}""" - - -def test_extract_apt_install_packages_returns_non_flag_arguments() -> None: - """Return install arguments across backslash continuations, dropping flags.""" - setup_commands = ( - "export DEBIAN_FRONTEND=noninteractive\n" - "sudo apt-get update \\\n" - " && sudo apt-get install --yes --no-install-recommends clang lld mold\n" - ) - - assert _extract_apt_install_packages(setup_commands) == ["clang", "lld", "mold"], ( - "expected only the non-flag install arguments to be returned" - ) - - -def test_extract_apt_install_packages_ignores_commented_installs() -> None: - """Skip commented-out install commands so they cannot satisfy the contract.""" - setup_commands = "# sudo apt-get install --yes clang lld mold\necho skip\n" - - assert _extract_apt_install_packages(setup_commands) == [], ( - "expected a commented-out apt-get install to be ignored" - ) - - -def test_extract_apt_install_packages_rejects_malformed_shell() -> None: - """Surface malformed setup-commands shell instead of masking it as valid.""" - setup_commands = 'sudo apt-get install --yes "clang\n' - - with pytest.raises(AssertionError, match="parseable shell"): - _extract_apt_install_packages(setup_commands) - - -def test_extract_apt_install_packages_ignores_echoed_marker() -> None: - """Ignore an echo whose argument merely contains the install marker text.""" - setup_commands = 'echo "run apt-get install clang lld mold first"\n' - - assert _extract_apt_install_packages(setup_commands) == [], ( - "expected embedded marker text in an echo argument not to count as an " - "install command" - ) - - -# Shell-safe package tokens: a leading alphanumeric (so they never look like a -# flag) followed by characters valid in a Debian package name. Flag tokens -# always begin with ``--`` so the parser drops them. -_apt_package_tokens: st.SearchStrategy[str] = st.builds( - lambda head, tail: head + tail, - st.sampled_from("abcdefghijklmnopqrstuvwxyz0123456789"), - st.text(alphabet="abcdefghijklmnopqrstuvwxyz0123456789+.-", max_size=11), -) -_apt_flag_tokens: st.SearchStrategy[str] = st.builds( - lambda word: "--" + word, - st.text(alphabet="abcdefghijklmnopqrstuvwxyz-", min_size=1, max_size=12), -) -_apt_install_tokens: st.SearchStrategy[list[tuple[str, bool]]] = st.lists( - st.one_of( - st.tuples(_apt_package_tokens, st.just(False)), - st.tuples(_apt_flag_tokens, st.just(True)), - ), - min_size=1, - max_size=6, -) - - -@given( - tagged=_apt_install_tokens, - use_sudo=st.booleans(), - indent=st.text(alphabet=" \t", max_size=3), -) -def test_extract_apt_install_packages_recovers_non_flag_arguments( - tagged: list[tuple[str, bool]], use_sudo: bool, indent: str -) -> None: - """Return exactly the non-flag install arguments, in order, dropping flags.""" - tokens = [token for token, _ in tagged] - expected = [token for token, is_flag in tagged if not is_flag] - prefix = "sudo " if use_sudo else "" - setup_commands = ( - "export DEBIAN_FRONTEND=noninteractive\n" - f"{indent}{prefix}apt-get install {' '.join(tokens)}\n" - ) - - assert _extract_apt_install_packages(setup_commands) == expected - - -@given( - packages=st.lists(_apt_package_tokens, min_size=1, max_size=5), - indent=st.text(alphabet=" \t", max_size=3), -) -def test_extract_apt_install_packages_skips_commented_installs( - packages: list[str], indent: str -) -> None: - """A commented apt-get install line never contributes packages.""" - setup_commands = ( - f"{indent}# sudo apt-get install --yes {' '.join(packages)}\necho noop\n" - ) - - assert _extract_apt_install_packages(setup_commands) == [] - - -@given(packages=st.lists(_apt_package_tokens, min_size=1, max_size=4)) -def test_extract_apt_install_packages_follows_line_continuations( - packages: list[str], -) -> None: - """Recover packages split onto a backslash-continued line.""" - setup_commands = f"sudo apt-get install --yes \\\n {' '.join(packages)}\n" - - assert _extract_apt_install_packages(setup_commands) == packages - - -def _mutation_workflow_with_schedule(schedule_block: str) -> str: - """Build a mutation-testing workflow whose ``on`` reaches the schedule check. - - Only the fields inspected before the schedule assertion are populated, so - the returned document exercises the schedule contract in isolation. - """ - return f'permissions: {{}}\n"on":\n{schedule_block} workflow_dispatch:\n' - - -def test_assert_mutation_workflow_contracts_rejects_extra_schedule_entry() -> None: - """Reject a second schedule entry beyond the single sanctioned cron.""" - workflow = _mutation_workflow_with_schedule( - f' schedule:\n - cron: "{_MUTATION_CRON}"\n - cron: "0 0 * * *"\n' - ) - - with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): - _assert_mutation_workflow_contracts(workflow) - - -def test_assert_mutation_workflow_contracts_rejects_extra_schedule_key() -> None: - """Reject an extra key smuggled into the single schedule entry.""" - workflow = _mutation_workflow_with_schedule( - f' schedule:\n - cron: "{_MUTATION_CRON}"\n branches: ["main"]\n' - ) - - with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): - _assert_mutation_workflow_contracts(workflow) diff --git a/tests/test_mutation_contracts.py b/tests/test_mutation_contracts.py new file mode 100644 index 0000000..6c48d28 --- /dev/null +++ b/tests/test_mutation_contracts.py @@ -0,0 +1,145 @@ +"""Validate the mutation-testing workflow contract helpers.""" + +from __future__ import annotations + +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from tests.helpers.tooling_contracts.mutation import ( + _MUTATION_CRON, + _assert_mutation_workflow_contracts, + _extract_apt_install_packages, +) + + +def test_extract_apt_install_packages_returns_non_flag_arguments() -> None: + """Return install arguments across backslash continuations, dropping flags.""" + setup_commands = ( + "export DEBIAN_FRONTEND=noninteractive\n" + "sudo apt-get update \\\n" + " && sudo apt-get install --yes --no-install-recommends clang lld mold\n" + ) + + assert _extract_apt_install_packages(setup_commands) == ["clang", "lld", "mold"], ( + "expected only the non-flag install arguments to be returned" + ) + + +def test_extract_apt_install_packages_ignores_commented_installs() -> None: + """Skip commented-out install commands so they cannot satisfy the contract.""" + setup_commands = "# sudo apt-get install --yes clang lld mold\necho skip\n" + + assert _extract_apt_install_packages(setup_commands) == [], ( + "expected a commented-out apt-get install to be ignored" + ) + + +def test_extract_apt_install_packages_rejects_malformed_shell() -> None: + """Surface malformed setup-commands shell instead of masking it as valid.""" + setup_commands = 'sudo apt-get install --yes "clang\n' + + with pytest.raises(AssertionError, match="parseable shell"): + _extract_apt_install_packages(setup_commands) + + +def test_extract_apt_install_packages_ignores_echoed_marker() -> None: + """Ignore an echo whose argument merely contains the install marker text.""" + setup_commands = 'echo "run apt-get install clang lld mold first"\n' + + assert _extract_apt_install_packages(setup_commands) == [], ( + "expected embedded marker text in an echo argument not to count as an " + "install command" + ) + + +# Shell-safe package tokens: a leading alphanumeric (so they never look like a +# flag) followed by characters valid in a Debian package name. Flag tokens +# always begin with ``--`` so the parser drops them. +_apt_package_tokens: st.SearchStrategy[str] = st.builds( + lambda head, tail: head + tail, + st.sampled_from("abcdefghijklmnopqrstuvwxyz0123456789"), + st.text(alphabet="abcdefghijklmnopqrstuvwxyz0123456789+.-", max_size=11), +) +_apt_flag_tokens: st.SearchStrategy[str] = st.builds( + lambda word: "--" + word, + st.text(alphabet="abcdefghijklmnopqrstuvwxyz-", min_size=1, max_size=12), +) +_apt_install_tokens: st.SearchStrategy[list[tuple[str, bool]]] = st.lists( + st.one_of( + st.tuples(_apt_package_tokens, st.just(False)), + st.tuples(_apt_flag_tokens, st.just(True)), + ), + min_size=1, + max_size=6, +) + + +@given( + tagged=_apt_install_tokens, + use_sudo=st.booleans(), + indent=st.text(alphabet=" \t", max_size=3), +) +def test_extract_apt_install_packages_recovers_non_flag_arguments( + tagged: list[tuple[str, bool]], use_sudo: bool, indent: str +) -> None: + """Return exactly the non-flag install arguments, in order, dropping flags.""" + tokens = [token for token, _ in tagged] + expected = [token for token, is_flag in tagged if not is_flag] + prefix = "sudo " if use_sudo else "" + setup_commands = ( + "export DEBIAN_FRONTEND=noninteractive\n" + f"{indent}{prefix}apt-get install {' '.join(tokens)}\n" + ) + + assert _extract_apt_install_packages(setup_commands) == expected + + +@given( + packages=st.lists(_apt_package_tokens, min_size=1, max_size=5), + indent=st.text(alphabet=" \t", max_size=3), +) +def test_extract_apt_install_packages_skips_commented_installs( + packages: list[str], indent: str +) -> None: + """A commented apt-get install line never contributes packages.""" + setup_commands = ( + f"{indent}# sudo apt-get install --yes {' '.join(packages)}\necho noop\n" + ) + + assert _extract_apt_install_packages(setup_commands) == [] + + +@given(packages=st.lists(_apt_package_tokens, min_size=1, max_size=4)) +def test_extract_apt_install_packages_follows_line_continuations( + packages: list[str], +) -> None: + """Recover packages split onto a backslash-continued line.""" + setup_commands = f"sudo apt-get install --yes \\\n {' '.join(packages)}\n" + + assert _extract_apt_install_packages(setup_commands) == packages + + +def _mutation_workflow_with_schedule(schedule_block: str) -> str: + """Build a mutation-testing workflow whose ``on`` reaches the schedule check.""" + return f'permissions: {{}}\n"on":\n{schedule_block} workflow_dispatch:\n' + + +def test_assert_mutation_workflow_contracts_rejects_extra_schedule_entry() -> None: + """Reject a second schedule entry beyond the single sanctioned cron.""" + workflow = _mutation_workflow_with_schedule( + f' schedule:\n - cron: "{_MUTATION_CRON}"\n - cron: "0 0 * * *"\n' + ) + + with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): + _assert_mutation_workflow_contracts(workflow) + + +def test_assert_mutation_workflow_contracts_rejects_extra_schedule_key() -> None: + """Reject an extra key smuggled into the single schedule entry.""" + workflow = _mutation_workflow_with_schedule( + f' schedule:\n - cron: "{_MUTATION_CRON}"\n branches: ["main"]\n' + ) + + with pytest.raises(AssertionError, match="schedule to be exactly one cron entry"): + _assert_mutation_workflow_contracts(workflow) From c0f532d5f1abd6483da313650889a939ff966b08 Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 29 Jul 2026 02:10:33 +0200 Subject: [PATCH 15/15] Document where mutation-testing diagnostics land MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The user guides explained what the scheduled mutation-testing workflow does but not where to look at its output, so a maintainer had no pointer to the reporting the shared reusable workflow already produces. Record the observability story: the run's job summary carries per-target outcome counts and a surviving-mutants table, each shard uploads its `mutants.out/` directory as a `mutation-report-*` artefact, and a quiet day writes a skip message instead. Note the failure semantics too — survivors and timeouts are informational and leave the run green, so a red run means a usage error, an already-failing test baseline, or an internal error, and GitHub's failed-scheduled-run notifications are what surface it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/users-guide.md | 9 +++++++++ template/docs/users-guide.md.jinja | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/docs/users-guide.md b/docs/users-guide.md index 3ed2173..ef38aa3 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -93,3 +93,12 @@ When adopting the workflow in a new repository, stagger the cron slot: pick an unclaimed daily time to avoid concurrent runs across related repositories. The `mutation` job runs with a least-privilege token (`contents: read` plus `id-token: write` for workflow-source resolution). + +Results land in the run's job summary: a final job posts per-target outcome +counts and a table of surviving mutants, and each shard uploads its +`mutants.out/` directory as a `mutation-report-*` artefact. When nothing +relevant changed, the run writes a skip message and finishes in seconds. +Surviving mutants and timeouts are informational and leave the run green, so a +red run means something actually broke — a usage error, an already-failing test +baseline, or an internal error. Watch for those through GitHub's notifications +for failed scheduled runs. diff --git a/template/docs/users-guide.md.jinja b/template/docs/users-guide.md.jinja index a493fd2..5e06f5f 100644 --- a/template/docs/users-guide.md.jinja +++ b/template/docs/users-guide.md.jinja @@ -68,3 +68,12 @@ When adopting the workflow in a new repository, stagger the cron slot: pick an unclaimed daily time to avoid concurrent runs across related repositories. The `mutation` job runs with a least-privilege token (`contents: read` plus `id-token: write` for workflow-source resolution). + +Results land in the run's job summary: a final job posts per-target outcome +counts and a table of surviving mutants, and each shard uploads its +`mutants.out/` directory as a `mutation-report-*` artefact. When nothing +relevant changed, the run writes a skip message and finishes in seconds. +Surviving mutants and timeouts are informational and leave the run green, so a +red run means something actually broke — a usage error, an already-failing test +baseline, or an internal error. Watch for those through GitHub's notifications +for failed scheduled runs.