feat: prefer cgroup v2 memory.max for --max-memory - #651
Open
ykhrustalev wants to merge 2 commits into
Open
ykhrustalev wants to merge 2 commits into
ykhrustalev wants to merge 2 commits into
Conversation
A mutant can turn a bounded loop into an unbounded allocator, and a test process growing at hundreds of MB/s exhausts the machine well before a 5x-baseline test timeout arrives. On a CI runner the VM is then torn down with no log, and the shard's mutants are never recorded. --max-memory SIZE, and the max_memory config key, put a ceiling on each scenario's cargo process tree instead. It is enforced with setrlimit (RLIMIT_AS), which limits address space rather than resident memory and so has to be set generously; which mechanism is in use is logged at startup. macOS accepts RLIMIT_AS and ignores it, so there the option warns and does nothing, as documented. Where it cannot be applied at all, giving the option is an error raised before any mutant runs, rather than a long run that silently had no limit. Zero is rejected too: unlike -t 0 it would mean 'stop everything', not 'no limit'.
RLIMIT_AS limits address space, not resident memory, which is a poor proxy: rustc reserves far more than it makes resident, so a limit tight enough to stop a runaway test can fail the build instead. A cgroup v2 memory.max per scenario limits what we actually care about. Finding somewhere to create those cgroups is the fiddly part, because the kernel won't let a cgroup that holds processes delegate the memory controller to its children. We try our own cgroup, then our parent -- which already delegates memory whenever something has fenced us in with a memory.max, the case this is meant for -- and only as a last resort move ourselves into a leaf. Using the parent puts scenario cgroups outside that outer fence, so that path warns. The cgroup is removed when the scenario's limit is dropped, so no failure part-way through a phase can leak it.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RLIMIT_AS(#649) limits address space, not resident memory. rustc reserves far more than it makes resident, so a limit tight enough to stop a runaway test can fail the build instead: on a small crate, a 256 MiBRLIMIT_ASfailed the build in 54 ms, while the same build peaked at 140 MB resident under a 256 MiB cgroup.Solution
memory.maxper scenario (plusmemory.swap.max=0where swap is accounted), falling back toRLIMIT_AS, and logs which is in use.memory.max— and as a last resort moves itself into acargo-mutants-supervisorleaf. Warns when using the parent, since scenario cgroups then sit outside that outer fence.Testing
unbounded_allocationtree: one mutant allocates a MiB at a time forever. The Linux test asserts it's caught in under a second against a 60s timeout, and skips where no writable cgroup is available (theRLIMIT_ASfallback can't build at 256M).RLIMIT_ASfallback under a read-only cgroupfs, and zero leftover cgroup directories after a run.Stacked on #649; only the last commit is new here. Split out of #647 as requested.