feat: bound runaway mutants with --max-memory and a process group sweep - #647
Open
ykhrustalev wants to merge 7 commits into
Open
ykhrustalev wants to merge 7 commits into
ykhrustalev wants to merge 7 commits into
Conversation
Processes left running by a scenario's tests survived a normal exit and kept allocating while later mutants were tested. Sweep the group after every phase, not only on timeout: SIGTERM, a bounded grace period, then SIGKILL. The timeout path now shares this sweep instead of its own kill.
A mutant can turn a bounded loop into an unbounded allocator that exhausts the machine well before the test timeout fires. Prefer a cgroup v2 memory.max per scenario, falling back to setrlimit(RLIMIT_AS), log which is in use, and fail before any mutant runs if neither can apply.
Surface the killing signal, the cgroup oom_kill count, and anything the process group sweep reaped on the outcome line, in the scenario log, and in outcomes.json, so an OOM-caught mutant is distinguishable from one caught by a failing assertion. Classification rules are unchanged.
Owner
|
Thanks, this sounds potentially good but please split it into smaller individual PRs. |
Author
|
@sourcefrog thanks for the feedback |
Creation was not atomic: a failure writing memory.max left the directory behind, as did any early return between creating the cgroup and finishing with it. Move removal into Drop so every path is covered, and stop treating a missing memory.swap.max as fatal -- it only exists where the kernel accounts for swap, and memory.max alone still bounds resident memory. Also tolerate a name left over by a run that died with this pid, and say so in the log when memory.events can't be read.
Carrying a MemoryMechanism beside an Option<CgroupTree> let the two disagree, and the code paid for it with an expect(), two unreachable!()s and five clippy allows. Give each variant the state its mechanism needs, and cfg out the RlimitAs variants where nix has no RLIMIT_AS, so the impossible combinations cannot be written rather than merely never happening.
terminate() sent one SIGTERM and then waited indefinitely, so a cargo process that ignored or could not receive it -- stopped, say -- hung the whole run, and the process group sweep that would have killed it never ran. Wait only for the shared grace period, then kill the group. Also probe the process group before listing it: nearly every phase leaves nothing behind, and listing meant reading every /proc/<pid>/stat on the machine, once per phase.
--max-memory=0 was accepted and meant 'stop every scenario immediately', which is the opposite of what -t 0 means elsewhere in this tool; require at least 1M. An OOM-killed mutant is also a caught mutant, so it was invisible without -v: count those separately in the run summary. The Linux test needed a writable cgroup and failed rather than skipped without one, which would be red on most CI runners; it now detects the mechanism and skips.
This was referenced Sep 16, 2026
Author
|
@sourcefrog it is split now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A mutant can turn a bounded loop into an unbounded allocator, and a test process growing at hundreds of MB/s exhausts the machine long before a 5x-baseline test timeout fires — on CI the VM is torn down with no log and the shard's mutants are never recorded. Separately, tests that leave processes running survive a normal exit, because the child's process group is only signalled on timeout, so they keep allocating between scenarios.
Solution
SIGTERM, a bounded grace period, thenSIGKILL. The timeout path shares this sweep instead of doing its own kill.--max-memory SIZEand themax_memoryconfig key to bound each scenario's cargo process tree.memory.maxper scenario where a writable cgroup is available, falling back tosetrlimit(RLIMIT_AS), and logs which mechanism is in use.--max-memoryis given and neither mechanism can be applied, rather than running with no limit. macOS accepts but ignoresRLIMIT_AS, so the option warns and is a no-op there.oom_killcount, and anything the sweep reaped on the outcome line, in the scenario log, and inoutcomes.json, so an OOM-caught mutant is distinguishable from one caught by a failing assertion. Caught / missed / unviable / timeout classification is unchanged.Testing
testdatatrees:spawns_background_child(test leavessleep 300running; the integration test probes the recorded pids afterwards) andunbounded_allocation(one mutant allocates a MiB at a time forever).max_memory_catches_a_mutant_that_allocates_without_boundis gated to Linux and asserts the mutant is caught in under a second with a 60s timeout, and that the outcome line names the OOM kill.--max-memorywith no usable mechanism is a hard error.RLIMIT_ASfallback under a read-only cgroupfs, and the hard error with theRLIMIT_AShard limit lowered below the request.