feat: report OOM kills from the scenario's memory cgroup - #653
Open
ykhrustalev wants to merge 7 commits into
Open
ykhrustalev wants to merge 7 commits into
ykhrustalev wants to merge 7 commits into
Conversation
On a timeout, terminate_child() sent one SIGTERM and then blocked in wait() with no bound. A cargo process that ignores the signal, or that has been stopped and so never receives it, hung the whole run at that point. Wait only for a short grace period, then SIGKILL the process group. This needs a second signal, so the errno handling moves into a signal_group() helper rather than being duplicated.
Tests can leave processes running after they exit: a helper a test spawned and forgot, or a test binary that was never reaped. Until now cargo-mutants only signalled the child's process group on a timeout, so on a normal exit those processes survived and kept allocating while later mutants were tested, in a window where no cargo phase is running at all. Sweep the group in the process layer, so every phase gets it: once the direct child exits with any status, SIGTERM the group, wait a bounded grace period, then SIGKILL whatever is left. What was reaped goes into the scenario log, and the pids into the debug log, enumerated from /proc on Linux. Windows has no process groups and the job object equivalent is out of scope, so the sweep is a no-op there.
A mutant can turn a bounded loop into an unbounded allocator, and a test process growing at hundreds of MB/s exhausts the machine well before a 5x-baseline test timeout arrives. On a CI runner the VM is then torn down with no log, and the shard's mutants are never recorded. --max-memory SIZE, and the max_memory config key, put a ceiling on each scenario's cargo process tree instead. It is enforced with setrlimit (RLIMIT_AS), which limits address space rather than resident memory and so has to be set generously; which mechanism is in use is logged at startup. macOS accepts RLIMIT_AS and ignores it, so there the option warns and does nothing, as documented. Where it cannot be applied at all, giving the option is an error raised before any mutant runs, rather than a long run that silently had no limit. Zero is rejected too: unlike -t 0 it would mean 'stop everything', not 'no limit'.
A mutant caught because its tests were killed is indistinguishable, in the counts, from one caught by a failing assertion -- which is exactly what you want to know when diagnosing a run. Carry the process group sweep's findings on each phase result, and render them, with the name of any signal that killed the phase, in parentheses on the outcome line, in the scenario log, and in outcomes.json. Classification is deliberately untouched: this only makes the reason visible.
RLIMIT_AS limits address space, not resident memory, which is a poor proxy: rustc reserves far more than it makes resident, so a limit tight enough to stop a runaway test can fail the build instead. A cgroup v2 memory.max per scenario limits what we actually care about. Finding somewhere to create those cgroups is the fiddly part, because the kernel won't let a cgroup that holds processes delegate the memory controller to its children. We try our own cgroup, then our parent -- which already delegates memory whenever something has fenced us in with a memory.max, the case this is meant for -- and only as a last resort move ourselves into a leaf. Using the parent puts scenario cgroups outside that outer fence, so that path warns. The cgroup is removed when the scenario's limit is dropped, so no failure part-way through a phase can leak it.
# Conflicts: # NEWS.md # book/src/timeouts.md # src/process.rs # tests/main.rs
An OOM-killed mutant is also a caught mutant, so without this it is indistinguishable from one caught by a failing assertion -- which is exactly what you want to know when a sweep keeps hitting the memory fence. Read memory.events after each phase, carry the oom_kill count alongside the sweep on the phase result, and name it on the outcome line and in outcomes.json. Count the scenarios it stopped in the run summary too, since caught mutants are not printed by default. Classification is unchanged.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An OOM-killed mutant is also a caught mutant, so without this it is indistinguishable from one caught by a failing assertion — exactly what you want to know when a sweep keeps hitting the memory fence.
Solution
memory.eventsafter each phase and carries theoom_killcount alongside the sweep on the phase result.outcomes.json, and counts the scenarios it stopped in the run summary, since caught mutants aren't printed by default.Testing
Stacked on #651 and #652, merged; only the last commit is new here. Its tree is byte-identical to #647 plus the two integration tests added in #649, so nothing was lost in the split.