mutation.yml

name: Mutation testing

# Repo-wide, sharded, incrementally-cached mutation testing via Stryker. Each package is gated by its own stryker.config.ts break threshold (derived from a completed run's measured baseline -- see stryker.shared.ts's breakThreshold), and the mutation-result job at the bottom aggregates the shards into the one stable check context a branch ruleset can require: a required status check must report on every pull request, and the shard matrix's width varies with the affected set, so no individual shard context can be required without stranding a pull request that produced fewer shards than the ruleset names. Requiredness itself lives in the repository ruleset ("main required checks"), never in this file -- a pull request can only prepare the job the ruleset points at, and the job is only ready to be required once every package completes a green run behind its threshold.
on:
  pull_request:
  push:
    branches: [main]

# One fixed group across every ref, not per-ref: a per-ref group only stops the SAME branch from running two overlapping mutation jobs, which does nothing to stop main and several PR branches each spinning up their own full 8-shard run at once -- exactly the shape that starved the shared runner pool of capacity every OTHER workflow (including required CI checks and main's own post-merge run) also needs, confirmed directly (ExaDev/documents.js, 2026-09-06: four simultaneous mutation runs across main/fix/epub-codec-silent-content-drops/fix/markdown-codec-soft-break-newline/feat/doc-codec-numbering-and-right-indent-write left every PR's required checks queued with zero progress for over an hour, and main's own post-merge CI queued behind them too). GitHub queues a superseding run behind whichever one is already active rather than running it at all, which is exactly what a non-required, tolerant-of-delay job like this one should do. Never cancels: a cancelled shard mid-run would leave that shard's incremental cache save skipped for no benefit, and this is a QUEUE, not a per-ref supersession, so there is nothing stale to cancel in the first place -- every queued run still runs eventually, just serialised.
concurrency:
  group: mutation-testing
  cancel-in-progress: false

permissions:
  contents: read

env:
  # Mirrors ci.yml's own TURBO_FLAGS exactly: a PR scopes to the packages it actually touches (and their dependents, via turbo's own dependency-aware --affected), main runs the whole workspace.
  TURBO_FLAGS: ${{ github.event_name == 'pull_request' && '--affected' || '' }}

jobs:
  plan:
    name: Plan mutation shards
    runs-on: ubuntu-latest
    timeout-minutes: 10
    outputs:
      matrix: ${{ steps.plan.outputs.matrix }}
      has-packages: ${{ steps.plan.outputs.has-packages }}
    steps:
      - uses: actions/checkout@v7
        with:
          # --affected compares against the base branch; the plan script's own turbo dry-run needs that history present, same reason ci.yml's own affected-scoped jobs fetch it.
          fetch-depth: 0
      # Not the setup-workspace composite action here: it declares no outputs of its own, so a step's $GITHUB_OUTPUT writes inside it never reach this job's `outputs:` block above -- confirmed the hard way in audit-autofix.ts's own GITHUB_OUTPUT comment. commitlint's own affected-range step has the identical shape (needs both install and a step output) and solves it the same way: plain checkout/setup/install, then a separate `id:`-bearing step.
      - uses: pnpm/action-setup@v6
      - uses: actions/setup-node@v7
        with:
          node-version-file: .tool-versions
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - name: Compute weighted mutation shards
        id: plan
        run: node --experimental-strip-types .github/scripts/compute-mutation-shards.ts $TURBO_FLAGS

  mutation-test:
    name: Mutation test (shard ${{ matrix.index }})
    needs: plan
    if: needs.plan.outputs.has-packages == 'true'
    runs-on: ubuntu-latest
    # Generous, deliberately: a shard's incremental cache can only ever help (see the caching step below), never hurt, so a cold run -- no prior cache to restore, e.g. this workflow's first ever run, or a shard whose package assignment shifted since the last one that covered it -- pays the full mutation-test cost for whichever packages landed in it. documents.js alone (the single largest package, ~44k mutatable source lines) is sharded onto its own shard for exactly this reason; the timeout has to fit its cold-run cost, not a warm one.
    timeout-minutes: 180
    strategy:
      fail-fast: false
      matrix: ${{ fromJson(needs.plan.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      # Stryker's own incremental report: reuses a prior run's per-mutant kill/survive result for anything unchanged since it was written, instead of re-testing every mutant every run -- the difference between a full run and a genuinely incremental one on a workspace this size.
      #
      # matrix.index in the SAVE key, not just run_id: every shard within one workflow run shares an identical run_id, so without the index every shard racing to save its own results at the end of the same run would collide on one identical key -- actions/cache silently DROPS a save whose key already exists rather than erroring, so that collision would mean only the first shard to finish ever actually persisted anything.
      #
      # restore-keys deliberately stays the broad "mutation-incremental-" prefix, not further scoped by shard index: which packages land in which shard index is not stable run to run (it depends on which packages are affected and their currently-measured weights each time), so a shard-index-scoped restore-keys would often miss a package's own prior incremental history the moment its shard assignment shifts to a different index on a later run. The broad prefix can only help, never hurt: it restores whichever cache was saved most recently, from any shard of any prior run; a package whose data isn't in that blob just starts cold, exactly as if this caching didn't exist.
      - uses: actions/cache@v6
        with:
          path: packages/*/reports/stryker-incremental.json
          key: mutation-incremental-${{ github.run_id }}-${{ matrix.index }}
          restore-keys: |
            mutation-incremental-
      - uses: ./.github/actions/setup-workspace
        with:
          # matrix.index in the turbo cache's own task name too, for the identical reason as the Stryker cache key above: every shard shares one run_id, and the composite action's own cache key is task-scoped, so an unqualified shared task name would let every shard's save collide on the same key.
          task: mutation-shard-${{ matrix.index }}
          # Within a shard, packages run strictly sequentially (--concurrency=1), not turbo's normal per-task parallelism: a standard GitHub-hosted runner has few cores, and Stryker layers its own subprocess pool on top of each package's own Vitest pool -- running several packages' pools concurrently on one runner causes real, measurable thread contention. Parallelism instead comes from separate shard jobs, which genuinely run on separate runners.
          #
          # --continue=dependencies-successful, not turbo's own default (never, which stops ALL further scheduling the moment one task in the run fails): confirmed directly elsewhere in this family of CI designs that the default leaves a multi-package run reporting only the first package's own failure and never even attempting the rest. dependencies-successful still correctly skips a package whose own _build failed (mutating code that doesn't build is meaningless) -- it just stops one broken sibling from blocking every unrelated package after it in the same shard.
          #
          # matrix.packages is a space-separated package-name list this workflow's own plan job produced (compute-mutation-shards.ts's shardMatrix), never attacker-influenced PR content -- the same trust level ci.yml already extends to other workflow-computed values interpolated directly into a run: command (e.g. steps.lint-range.outputs.from).
          command: |
            filters=()
            for pkg in ${{ matrix.packages }}; do
              filters+=("--filter=$pkg")
            done
            pnpm exec turbo run _test:mutation --concurrency=1 --continue=dependencies-successful "${filters[@]}"
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: mutation-report-shard-${{ matrix.index }}
          path: packages/*/reports/mutation/mutation.html
          if-no-files-found: ignore

  # The single check context to list as required in the repository ruleset. The mutation-test job above is a matrix whose width the plan job computes per run from the affected package set (three affected packages produce three shards, not eight), so requiring any shard context by name would leave a pull request that produced fewer shards forever waiting on a check that never reports. This job always runs -- `if: always()` keeps it reporting even when planning fails or produces no packages, where the matrix job is skipped -- and derives one conclusion from the outcomes it aggregates, so it is stable across every shard count.
  mutation-result:
    name: Mutation testing result
    needs: [plan, mutation-test]
    if: always()
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Aggregate the shard outcomes into one conclusion
        # A plan failure is a real failure (nothing can be said about the mutation state); a plan that found no affected packages is a pass (nothing to mutate); anything else gates on the matrix's own aggregate result, which is failure when any shard failed (a package fell below its break threshold or its run errored).
        run: |
          if [ "${{ needs.plan.result }}" != "success" ]; then
            echo "::error::Mutation shard planning failed; see the Plan mutation shards job."
            exit 1
          fi
          if [ "${{ needs.plan.outputs.has-packages }}" != "true" ]; then
            echo "No mutation-affected packages in this change; nothing to gate."
            exit 0
          fi
          if [ "${{ needs.mutation-test.result }}" != "success" ]; then
            echo "::error::One or more mutation shards failed: a package's score fell below its break threshold, or its run failed."
            exit 1
          fi
          echo "Every mutation shard succeeded."