Document mutation-testing workflow contract tests - #297
Conversation
Add a "Mutation-testing workflow contract tests" section to the developer's guide describing the thin caller workflow, its configuration, the pin-only adoption rationale (dylint crates needing per-target RUSTFLAGS that the shared workflow cannot reproduce), and the local command for running the contract test suite.
WalkthroughDocumented the mutation-testing workflow’s scheduled and manual modes, configuration inputs, pinned reusable-workflow reference, pin-only limitation, ADR, and contract-test coverage. ChangesMutation-testing workflow documentation
Possibly related PRs
Poem
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (18 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![codescene-access[bot]](https://avatars.githubusercontent.com/in/3755875?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 087be29a97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| adoption is **pin-only**: the caller declares the best approximation of the CI | ||
| scope it safely can, rather than a `--test-workspace` run that mirrors | ||
| `make test` crate-for-crate, and the contract test asserts only that this | ||
| declared configuration holds — not that a full workspace mutation baseline | ||
| passes. |
There was a problem hiding this comment.
This establishes a substantive architecture decision—accepting a mutation scope that cannot reproduce the CI baseline and deliberately testing only its declared configuration—but records the rationale solely in the developer operating guide. Move the decision and its consequences into the relevant design document or an ADR, then link to it here, so the repository's normative design record preserves this constraint as required.
AGENTS.md reference: AGENTS.md:L48-L51
Useful? React with 👍 / 👎.
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/developers-guide.md`:
- Line 279: Update the documentation text near “job summary” to use the required
British spelling “artefacts” instead of “artifacts,” preserving the surrounding
wording.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: df12017f-a585-422e-935a-6f503314fad3
📒 Files selected for processing (1)
docs/developers-guide.md
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
leynos/rstest-bdd(auto-detected) → reviewed against open PR#598docs/mutation-contract-testsinstead of the default branch
| lifting — running `cargo-mutants`, sharding, and summarizing survivors — lives | ||
| in `shared-actions`; this repository carries only declarative configuration. | ||
| The run is **informational only**: it never gates a pull request. Survivors | ||
| are reported through the job summary and downloadable artefacts so they can be |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the required British spelling.
Change artifacts to artefacts.
Triage: [type:spelling]
As per path instructions, Markdown uses en-GB-oxendist spelling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/developers-guide.md` at line 279, Update the documentation text near
“job summary” to use the required British spelling “artefacts” instead of
“artifacts,” preserving the surrounding wording.
Source: Path instructions
|
@coderabbitai resume |
✅ Action performedReviews resumed. |
Preserve the architectural constraint that the shared mutation workflow cannot reproduce Whitaker's per-crate test flags. Document the accepted declared-configuration contract and link it from the maintainer guide.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/adr-004-pin-only-mutation-testing-contract.md`:
- Around line 5-11: Update the acceptance date in the ADR heading and the
corresponding Date section to the actual non-future acceptance date, replacing
both occurrences of 2026-07-23 consistently.
- Around line 38-56: Replace the prose-only “Options considered” list in the ADR
with a comparison table covering Options A, B, and C across feasibility,
mutation coverage, operational risk, and other relevant dimensions. Preserve
each option’s existing rationale, and add a caption directly below the table.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: eb1e1c64-d562-48c7-af27-e5c3c8a5ff0e
📒 Files selected for processing (3)
docs/adr-004-pin-only-mutation-testing-contract.mddocs/contents.mddocs/developers-guide.md
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
leynos/rstest-bdd(auto-detected) → reviewed against open PR#598docs/mutation-contract-testsinstead of the default branch
| Accepted (2026-07-23): Keep Whitaker's mutation-testing adoption pin-only and | ||
| contract-test its declared configuration rather than claiming parity with the | ||
| continuous integration (CI) test baseline. | ||
|
|
||
| ## Date | ||
|
|
||
| 2026-07-23. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use a non-future ADR date.
Align Line [5] and Line [11] with the actual acceptance date. 2026-07-23 is tomorrow relative to the review date, Wednesday, July 22, 2026, so the ADR currently records a decision before it occurred.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/adr-004-pin-only-mutation-testing-contract.md` around lines 5 - 11,
Update the acceptance date in the ADR heading and the corresponding Date section
to the actual non-future acceptance date, replacing both occurrences of
2026-07-23 consistently.
| ## Options considered | ||
|
|
||
| ### Option A: require full CI-baseline parity | ||
|
|
||
| Pass a workspace-testing argument and treat the mutation run as equivalent to | ||
| `make test`. This is not implementable through the current shared workflow | ||
| because it cannot inject the required flags per crate. | ||
|
|
||
| ### Option B: exclude every crate that needs dynamic-linking flags | ||
|
|
||
| Remove the affected crates from mutation scope. This would make the workflow | ||
| run reliably, but would deliberately omit important production code and make | ||
| the reported scope less useful. | ||
|
|
||
| ### Option C: adopt a pin-only declared-configuration contract | ||
|
|
||
| Run the closest safe approximation supported by the shared workflow and test | ||
| the caller's security and configuration shape without asserting that a full | ||
| workspace mutation baseline passes. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add the required options comparison table.
Replace the prose-only options list with a table comparing the alternatives across multiple dimensions, such as feasibility, mutation coverage, and operational risk. Add a caption below the table.
Triage: [type:docstyle]
As per coding guidelines, every ADR must compare options across multiple dimensions using a table with a caption below it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/adr-004-pin-only-mutation-testing-contract.md` around lines 38 - 56,
Replace the prose-only “Options considered” list in the ADR with a comparison
table covering Options A, B, and C across feasibility, mutation coverage,
operational risk, and other relevant dimensions. Preserve each option’s existing
rationale, and add a caption directly below the table.
Source: Coding guidelines
2 participants
Summary
docs/developers-guide.md, inserted between "Running Tests" and "Proofworkflows" (the guide has no existing mutation-testing section and no
table of contents/index to cross-link, so this is a new top-level
##section with no TOC entry to add).
.github/workflows/mutation-testing.yml,which delegates to
leynos/shared-actions/.github/workflows/mutation-cargo.ymlpinned at
927edd45ae77be4251a8a18ca9eb5613a2e32cbd, its realwith:inputs (
paths,exclude-globs,extra-args: --all-features), and thecontract test at
tests/workflow_contracts/mutation_testing_test.py.the mutation caller's
exclude-globsdoes not (and cannot) fully mirrormake test'sTEST_EXCLUDES, because the excluded crates(
whitaker,function_attrs_follow_docs,module_max_lines,no_expect_outside_tests) usefeature(rustc_private)under thedylint-driverfeature and need the dynamic-linkingRUSTFLAGS(
-C prefer-dynamic -Z force-unstable-if-unmarked) injected only by thetest/typecheckMakefile targets — see the note in.cargo/config.toml. The shared mutation workflow has no equivalentper-crate RUSTFLAGS step, so this repository does not (and cannot) run a
full
--test-workspacemutation baseline matching the CI test scopewholesale. The section states this honestly rather than claiming parity.
Details
USES_REregex asserting a full40-hex commit SHA pin, not a hard-coded value), so Dependabot bumps the
pin without an accompanying test edit. No
pytestmark/skipifguard ispresent or needed — this is a Rust workspace, not a Python package whose
tests run inside a mutmut sandbox.
make test-workflow-contracts, which wrapsuv run --with 'pytest>=8' --with 'pyyaml>=6' pytest tests/workflow_contracts -q.docs/roadmap.mdanddocs/execplans/*— no task references mutation testing or this docwork, so no roadmap/execplan tracking applies and none was
annotated.
make markdownlint(spelling viatypos, thenmarkdownlint-cli2across all 69 Markdown files in the repository).Both passed with zero errors after fixing two
-ise/-izespellingissues (
summarising→summarizing,serialises→serializes) thatthe new section introduced.
Deviations from the template
extra-argswould carry a--test-workspace/--workspaceflag; whitaker's caller instead usesextra-args: --all-features(matching the Makefile'sCARGO_FLAGSfeature baseline), with workspace-wide coverage coming from the
pathsinput rather than a mutants flag. The section documents the real value
and explains the pin-only reasoning instead.
Edited files:
docs/developers-guide.md,docs/contents.md, anddocs/adr-004-pin-only-mutation-testing-contract.md.References