Skip to content

Mutation testing: mutation_detect_changes survivors #340

Description

@leynos

Context

Local mutation-testing run against main at 3c1fa01bcf4c26dac79975c18b74c01d577dcf95 (mutmut 3.6.0, committed [tool.mutmut] configuration: source_paths = ["workflow_scripts/"], test selection workflow_scripts/tests/, act-gated workflow tests excluded). Run totals: 1463 mutants — 979 killed, 478 survived, 5 no-tests, 1 timeout.

These scripts power the estate's CI, so mutation gaps here matter most. This issue covers one coherent area; sibling issues cover the rest. Equivalent mutants are recorded in the triage worklist and deliberately omitted here. Inspect any survivor with uv run --with mutmut==3.6.0 mutmut show <name>; names below are workflow_scripts.<module>.x_<function>__mutmut_<n> with the numbers listed per line.

Analysis

mutation_detect_changes.py builds the change-detection matrix for the mutation workflows. The git invocation in changed_files is not integration-tested (dropping --format=, the base ref, or the -- separator survives), the exact matrix JSON contract is unasserted (sort_keys mutants survive), full_run_matrix extra-crate entries and the skip summary are unchecked, and the step-output writer's formatting escapes. 19 survivors are real test gaps; 4 are accepted (below). Mutants marked 2-equivalent in the listing are annotated with their rationale and need no test.

Suggested tests

  1. An integration test for changed_files against a temporary git repository (two commits inside and outside the window, a deleted file, and a non-matching pathspec), asserting the exact returned tuple. Killing requires the real git log --since ... --format= <ref> -- <pathspec> semantics, so the argument-dropping mutants die.
  2. A golden test of matrix_json asserting the exact serialised string for a two-entry matrix (kills the sort_keys mutants and pins the step-output contract).
  3. full_run_matrix with extra_crate_dirs populated: assert each extra-crate entry is files="", shard=0, shard_count=1 alongside the sharded root entries.
  4. scoped_run_matrix with adversarial bucket names (for example a and a-b, plus the root): assert root-first ordering and each entry's dir (kills the dir=None mutant and the item[1] sort-key mutant, which can reorder when a file path and a sibling directory name compare differently).
  5. _write_output and _write_skip_summary: assert the exact name=value\n line appended to GITHUB_OUTPUT and the exact rendered skip message (base ref and window hours interpolated) in GITHUB_STEP_SUMMARY.
  6. _slug_for: assert crates/foo maps to exactly crates-foo.

Accepted survivors (no test to write)

  • _write_output mutants 2 and 4, and _write_skip_summary mutants 7 and 9 (workflow_scripts/mutation_detect_changes.py:305,314): encoding=None/omitted encoding falls back to the locale encoding and is equivalent on UTF-8 CI runners. Candidate # pragma: no mutate.

Surviving mutants

  • workflow_scripts/mutation_detect_changes.py:193 in changed_files (mutants 13, 20; 1-test-gap)
    • original: --format=",
    • 13: None,
    • 20: ``
  • workflow_scripts/mutation_detect_changes.py:194 in changed_files (mutants 14, 21; 1-test-gap)
    • original: config.base_ref,
    • 14: None,
    • 21: ``
  • workflow_scripts/mutation_detect_changes.py:195 in changed_files (mutants 15, 22; 1-test-gap)
    • original: --",
    • 15: None,
    • 22: ``
  • workflow_scripts/mutation_detect_changes.py:205 in _is_under (mutants 5, 6, 7; 2-equivalent)
    • original: base_path = PurePosixPath(base.rstrip("/"))
    • 5: base_path = PurePosixPath(base.rstrip(None)) — PurePosixPath ignores trailing '/'; rstrip(None) leaves it, comparison unchanged
    • 6: base_path = PurePosixPath(base.lstrip("/")) — lstrip('/') is a no-op for the relative dirs this receives
    • 7: base_path = PurePosixPath(base.rstrip("XX/XX")) — 'XX/XX' never present; rstrip a no-op, PurePosixPath normalises anyway
  • workflow_scripts/mutation_detect_changes.py:243 in _slug_for (mutants 9, 10; 1-test-gap)
    • original: return "root" if target_dir == "." else target_dir.replace("/", "-")
    • 9: return "root" if target_dir == "." else target_dir.replace("XX/XX", "-")
    • 10: return "root" if target_dir == "." else target_dir.replace("/", "XX-XX")
  • workflow_scripts/mutation_detect_changes.py:248 in _relative_files (mutants 2; 2-equivalent)
    • original: if target_dir == ".":
    • 2: if target_dir == "XX.XX": — relative_to(PurePosixPath('.')) returns the path unchanged, same join
  • workflow_scripts/mutation_detect_changes.py:271 in full_run_matrix (mutants 20, 21, 29, 30; 1-test-gap)
    • original: MatrixEntry(dir=d, slug=_slug_for(d), files="", shard=0, shard_count=1)
    • 20: MatrixEntry(dir=d, slug=_slug_for(d), files=None, shard=0, shard_count=1)
    • 21: MatrixEntry(dir=d, slug=_slug_for(d), files="", shard=None, shard_count=1)
    • 29: MatrixEntry(dir=d, slug=_slug_for(d), files="XXXX", shard=0, shard_count=1)
    • 30: MatrixEntry(dir=d, slug=_slug_for(d), files="", shard=1, shard_count=1)
  • workflow_scripts/mutation_detect_changes.py:281 in scoped_run_matrix (mutants 3, 5, 7, 9, 10; 1-test-gap/2-equivalent)
    • original: ordered = sorted(buckets.items(), key=lambda item: (item[0] != ".", item[0]))
    • 3: ordered = sorted(buckets.items(), key=None) — '.' sorts before all dir names alphabetically; key=None gives same order
    • 5: ordered = sorted(buckets.items(), ) — default tuple sort preserves root-first order ('.' < letters)
    • 7: ordered = sorted(buckets.items(), key=lambda item: (item[1] != ".", item[0])) — item[1] (list) != '.' always True; falls back to alphabetical, '.' still first
    • 9: ordered = sorted(buckets.items(), key=lambda item: (item[0] != "XX.XX", item[0])) — root sorts first alphabetically regardless of the sentinel
    • 10: ordered = sorted(buckets.items(), key=lambda item: (item[0] != ".", item[1]))
  • workflow_scripts/mutation_detect_changes.py:285 in scoped_run_matrix (mutants 11; 1-test-gap)
    • original: dir=target_dir,
    • 11: dir=None,
  • workflow_scripts/mutation_detect_changes.py:299 in matrix_json (mutants 2, 8; 1-test-gap)
    • original: sort_keys=True,
    • 2: sort_keys=None,
    • 8: sort_keys=False,
  • workflow_scripts/mutation_detect_changes.py:299 in matrix_json (mutants 4; 1-test-gap)
    • original: sort_keys=True, / )
    • 4: )
  • workflow_scripts/mutation_detect_changes.py:305 in _write_output (mutants 2, 4, 8; 2-equivalent/4-untestable)
    • original: with output_path.open("a", encoding="utf-8") as handle:
    • 2: with output_path.open("a", encoding=None) as handle: — encoding=None locale-dependent
    • 4: with output_path.open("a", ) as handle: — encoding omitted, locale-dependent
    • 8: with output_path.open("a", encoding="UTF-8") as handle: — UTF-8 codec alias
  • workflow_scripts/mutation_detect_changes.py:314 in _write_skip_summary (mutants 7, 9, 14; 2-equivalent/4-untestable)
    • original: with Path(summary_path).open("a", encoding="utf-8") as handle:
    • 7: with Path(summary_path).open("a", encoding=None) as handle: — encoding=None locale-dependent
    • 9: with Path(summary_path).open("a", ) as handle: — encoding omitted, locale-dependent
    • 14: with Path(summary_path).open("a", encoding="UTF-8") as handle: — UTF-8 codec alias
  • workflow_scripts/mutation_detect_changes.py:317 in _write_skip_summary (mutants 16, 17; 1-test-gap)
    • original: base_ref=config.base_ref, window_hours=config.window_hours
    • 16: base_ref=None, window_hours=config.window_hours
    • 17: base_ref=config.base_ref, window_hours=None

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions